Judges 3:19

‘ But he himself turned back from the graven images which were by Gilgal, and said, “I have a secret errand to you, Oh king.” And he said, “keep silence.” And all who stood by him went out from him.'

He returned to where the king's party were. If anything went wrong he wanted it to be seen as an individual act, not bringing retribution on his people. Then he indicated that he had a message to convey that required utmost secrecy. In their eyes he was clearly unarmed.

“And he said, ‘keep silence.” And all who stood by him went out from him.' The king's words clearly indicated that they leave him alone with Ehud, and were probably a standard signal. Total silence was only possible in an emptied room. Then they all left the room. Alternately it may be that his words were to Ehud, telling him to say nothing until they were alone. Then by a signal he would dismiss his servants.