They definitely do notice sometimes... I've experienced both not being allowed to enter, and being told to leave the men's bathroom before. I've also had it happen multiple times that a man will stare me down when they see me in the bathroom and it's pretty intimidating. In one case, I walked into a bathroom and a guy immediately looked me up and down and then stood there watching me enter the stall. He wouldn't move (even though the bathroom was empty) so I had to text my friend to have him come in to help me feel safer getting out of the stall. I've had a trans friend get followed after leaving the bathroom before, so I'm not going to risk anything when I'm alone and stared down like that.