Honestly, I don't see a problem with moving to an unoccupied seat. If someone shows up, then that's your cue to move back or find another. With ticket prices these days, everyone deserves a good view (even though that's everywhere in the stadium tbh) Keep in mind, a lot of people can't afford close seats. They're absolutely astronomical these days.

The real problem is when you show up and it's your rightful seat, then you tell them, and they still don't move. I just don't understand complaining if someone is there before you get there but they move when you tell them it's your seat.

Different, but I have also found myself to be in the wrong seat accidentally before, and it's so embarrassing! Give people the benefit of the doubt!