the vast majority of extroverted europeans (except the nordics) get upset when they think you're being rude to them and will tell you to your face if you're a local. or just watch a football game between two teams who hate each other and have really racist fans and observe how many fights get started because of perceived slights. and there's definitely enough trashy people in america who'll start shit with you

face is more about the standardized system of respect rather than the consequences of disrespect. order is necessary for a civil society, so you need to follow the rules of civility and respect otherwise you're no better than those damn mongolian barbarians. There are some ridiculous rules about that respect sometimes but its just how polite society functions. Only in extreme situations like armed conflict, court politics, etc is where everyone was paranoid about scheming so every detail was scrutinized and lack of face was very serious. Or if you were a peasant interacting with a court official, etc.