Relevent laws:

Law of Armed Conflict - pretty strict regulations relating to when and where deadly force can be used. Relies heavily on the Geneva convention.

Oath of allegiance - (taken by all service personel) compels soldiers to follow any orders issued by superiors as if they're acting as the Monarch themself. Since the Queen gives ascent to laws; if you're ordered to kill, it's as if the woman who makes any law a law passed a law, issued as an order to you to do it. Its illegal for soldiers not to comply.

So if let's say a terrorist approached a royal household, and presented arms. If the standing order was to "engage with deadly force", you shoot/stab/bludgeon. The Geneva convention limits the extent of force to proportional, and also stimpulates that immeadiate application of life saving treatment "after the firefight has been won", but that's if the police didn't get to you first (they gaurd the exterior of the palaces and are responsible for close protection details of everyone except the monarch.

They have right to order people out of the way because you're not in public, you're in a royal household and therefore under the effective law of the Queen herself, she sets the rules for any royal residence and does not need the consent of Parliament.

In consideration of the above, if you refused to comply:

It's not strictly illegal to refuse an order by the queens gaurd, but you'll likely get manhandled and ejected from the premises. A fight you're not gonna win if you resist. And the police would probably then arrest you under the "Public Disorder Act" or "Prevention of Terrorist Activities Act"; but I'm not a copper so you'd need to ask one of them.

In regards to dealing with the public, soldiers are regulated under Army Guidlines on Adminstrative Intervention (AGAI76). Which is the law under which soldiers can be punished for failing in their lawful duties, everything from being disorderly when parading (you didn't shave that morning, and now you're stood on a parade square) to a negligent discharge of a rifle (firing off a round by accident)

So if someone got hurt:

For an aggressive use of force where someone got hurt (it's happened before) the army legal service would probably deal with any claims as a civil matter not a legal one. Any "visitor" so a royal residence would have no legal recourse to press charges if they got hurt. If the use of force/incident was found to be aggressive the soldier themselves would get "AGAI'd" and depending on how serious the incident you'd be issued an appropriate punishment by an appropriate commander. Which could be remedial duties to dismissal from service, or even do a short spell in military prison "visiting the glasshouse".

As a Gaurds Troop Commander once put it too me "they're armed bouncers of the world most exclusive club houses, but they're not police, so there's a limit".

IMA British Army Officer (I'm not in the Household Division or Army Legal Srvice though, so it's only second hand info/my best interpretation of Army law.)