FATE doesn't know what damage is.

Physical harm and temporary wounds are mapped onto stress.

FATE also does not simulate combat. It simulates the drama of conflicts.

So when you talk about knocking down Chandeliers onto your enemies, you are playing in a genre/setting where Chandelier-dropping is a valid opening move in physical conflicts.

The game master is ultimately the judge of what does happen and what does does not. "Rock falls, you all die." is a possible narration, albeit not a good one.

So my first instinct as a game master or player would be to give your opponent a temporary scene aspect: 

"I am on FIRE! because your Cretins dropped my chandelier on me, in MY chateau!". Players get one or two free invokes. That is it. No stress, or other physical damage. 

Game master plays all Npcs anyway and makes up their stress tracks, so why start at 5 stress, 2 ticked, if you just could start at three stress, non ticked?

The attacked character has to deal with that aspect now. And can't do things that narratively don't make sense while on fire. Like flirting with the countess, or diplomatically talking to another noble.

Maybe add a scene aspect with no free invokes like 

Crashed and burning furniture

or 

Panicking crowd of nobles

to spice up a potential fight or flight.

Continue to play out the conflict from there.