Seems to me you don't have to "speak" to have a town hall meeting - especially if your office has a projector, laptop, a screen, and a staffer who types well!

Tell the crowd the rules first (one speaker at a time, no applause, no interruptions, etc), and if if factions in the crowd drown you or someone else out, fire up the laptop!

Take the questions by hand - pass out pencils and paper, and have the questioners pass the questions up front. Shuffle the questions.

Re-type the question.
Type the answer.

As long as the crowd plants don't get violent, you've just gained control back over your town hall...