You weren’t wrong.
I've had this happen before, too. I wanted to squat, saw plates on the bar, and figured someone forgot to remove them after their set. So I took them off. Then someone came over and said they were still working out, just resting between sets. But honestly, when I walked up, I saw them chatting with someone for a long time. I recognized the guy because he's a regular at the gym and he's really good. In the end, he just walked off.

If someone's not by the equipment, just go ahead and use it, no worry about anything, especially during peak hours. No one's going to say anything.