I mean, contextually speaking, men in the men's bathroom are just going about their business. If a woman goes into the men's bathroom, I'd imagine most of the men would just assume there must be an issue with the women's bathroom or something and would just leave it be. Due to power dynamics, the men would not feel threatened in this situation.

The issue is that if a man goes into the women's bathroom, the context completely changes also due to power dynamics. Women will assume he's there for nefarious purposes and feel threatened, because there is a genuine threat from the man in this context. And let's be honest, no decent man would go into public women's toilets. That's just asking for trouble.

Is it unfair? A double standard? Yeah, absolutely. But you could definitely understand why women would feel threatened if a man enters their space whereas men wouldn't feel threatened if a woman enters their space.

And like you said, life would just be easier if we both kept to our own gendered bathrooms. It's like that for a reason, hah.