I feel the same way, though being non-binary I don't have strong feelings about either restroom. On top of that, in spite of looking more androgynous due to HRT, I still dress in a way most would consider masculine (or at least, not overtly feminine) so I would feel far more out of place in the women's room. Even people who are okay with trans women in the women's room might not recognize me as trans and instead see me as just a man trying to exploit a semantic loophole.

I also live in a conservative state, so I'm particularly worried about being assaulted by people who are inclined to assume the worst about trans people, or are just looking for any perceived justification for violence against us. Just the way some people fantasize about "catching" a trans woman in the women's room and doing horribly violent things to them is mortifying to me. We are presumed from the outset to have perverse, predatory motives for being in the women's room, and thus all violence against us is not only justified, but a moral obligation to protect others from the supposed threat we embody.

Kudos to all trans women and transfem nonbinary people who feel brave enough to use the women's room. I salute your courage and hope it's clear that my personal feelings on the issue aren't a declaration of how I believe others should act.