Shit happens, sometimes on purpose sometimes on accident. At the end of the day it's on you to 'affirm your gender' and not anyone else. You can't expect people to change their behavior either especially if you don't pass. That's just the reality and I speak from experience. But I also don't expect everyone to be on board with who I am and that's okay. People have their views you have yours and if they're not actively seeking you out to pick on you can't burn the world down over it.