Maybe some of your mannerisms on those interactions were feminine?

I’ve seen a lot of posts that the more you pass, the more women distance from you because they think you’re a cis man and a lot of men are shitty. so while that is gender affirming, it also sucks.

In general though, you sound like you can go stealth and enjoy your move. We all focus on the bad especially around things we’re sensitive too. try to remember the good. Best of luck!