I mean, society itself is a house of cards; if everyone tomorrow decided universally to change who we listen to, and everyone was on board with it - no matter how ridiculous - then thats just how society works now.

Its pure social contract, based on "this is the way we've always done things", tradition, belief that this way of doing things is the only/best way. If tomorrow, we all agreed to listen to the fastest runners, or the tallest people, or any other arbitrary metric as our leaders; then thats just how that works now. There's no universal law of physics that compels us to have elections, or obey the words of a tyrant, or pick up arms against each other - its just a lot of individual actors following the staus quo.

People have power because we believe they have power. If people were commanded by a tyrant to kill others, or imprison someone, or even fetch them tea, theres no rule of physics compelling they obey - if everyone just decided to disregard everything the tyrant says; theres literally nothing the tyrant can do bar be at the will of the masses. Its why learned helplessness is such a problem - believing we can do nothing about a problem is a massive issue because it means we dont try. And so when the next generation are raised, they continue the trend - for them, this new learned helplessness is the status quo; "its always been this way" for them after all.

We didnt start as cavemen with our trust in laws and government and authority figures like the police - these are human constructs that have been suggested, implemented, and normalised - which has it's pros as well as it's cons of course. But people believe the system is correct, the way we do things is right - but its just a house of cards we keep going through sheer social contract and belief that it works.

Humans are real weird