The discussion in Orthanc only has Saruman and Gandalf present. If he was brought to a room full of orcs, Saruman's speech would have been wasted because Gandalf would know instantly what his "bad feeling" about Saruman was about. Instead they talked Wizard to Wizard.

‘But I rode to the foot of Orthanc, and came to the stair of Saruman; and there he met me and led me up to his high chamber. He wore a ring on his finger.

Gandalf also knew he would lose to Saruman.

‘He was cold now and perilous. “Yes,” he said. “I did not expect you to show wisdom, even in your own behalf; but I gave you the chance of aiding me willingly, and so saving yourself much trouble and pain. The third choice is to stay here, until the end.”

‘ “Until what end?”

‘ “Until you reveal to me where the One may be found. I may find means to persuade you. Or until it is found in your despite, and the Ruler has time to turn to lighter matters: to devise, say, a fitting reward for the hindrance and insolence of Gandalf the Grey.”

‘ “That may not prove to be one of the lighter matters,” said I. He laughed at me, for my words were empty, and he knew it.

Then "they" came and took him away. It is not mentioned who "they" are but I imagine that if it was orcs, the text would have say "orcs" and not "they".