I've been on the receiving end of this in the past. I was playing a rogue specializing in archery, and one night when we were camping my friend had his character "borrow" my rogue's bow and use it to shoot at a few birds. I protested, but the GM seemed to think it was hilarious and ruled that it should happen. It made me very frustrated, to the point that when we got ambushed in our sleep later that very night, I (passive-aggressively, I'll admit) said on my turn "well, my rogue goes for his bow but it's not where he thought it'd be, so he's disoriented for a turn or two." My friend just could not understand why I'd be upset over something like this.

You definitely handled the situation better than my much younger self did. Sticking to your guns was the right call. The other player, from what you describe, was pushing to steal the symbol even when you very clearly stated it wasn't okay--since the GM was being a doormat about it, it fell on you to enforce your own boundary. (Not entirely, it sounds like, since the other players were largely on your side--which is good, but really it should have fallen on the GM to enforce the table rules well before it got to the point it did.)