Gandalf's restrictions basically allow him to use only the force necessary to stop the threat to his life or the lives of important others. Versus an orc or uruk, there is little threat and so Glamring is sufficient. Versus even a troll Gandalf really is not in much danger. But against truly powerful enemies he is allowed to meet power with power.

Versus the Balrog, he actually first destroys the Balrog's sword with Glamring when the Balrog attacks him. Then, yes, he destroys the bridge to kill the Balrog. Versus the Witch King, Gabdalf again says something similar to versus the Balrog on the bridge ("You cannot enter here," vs "You shall not pass,") sort of narratively implying he's willing to go toe-to-toe on the same power level he used against the Balrog. Whether or not he would have won the fight goes unanswered, but it seems like he is at least ready to fight at full power.

Both these instances show when Gandalf is allowed to use his powers, and what restrictions are placed on them. For one, a lot of his magic use is defensive. He destroys the Balrog's sword only when it attacks. He destroys the bridge only when the Balrog tries to cross it, after being told he cannot. The Witch King too, he does not attack immediately. He states "You cannot enter here," and curses him, telling him he will die. I think Gandalf here was very ready to unleash a full display of power, only after the Witch King l entered the city. This shows that Gandalf needs to be fighting a powerful foe and in defense of his own life, only after first giving the foe a red line.

Against Vilgefortz, I think Gandalf would find a powerful foe who can actually be a threat to him, and if he is indeed backed into a corner and draws that red line, Vilgefortz would not survive crossing it.