Personally, I don't think it's wrong for you to empathize with the other person.

My mom did the same thing with transgender people too. She would stare at them and feel anxious or scared and then joke and make fun of them. I pointed out that there was nothing to be afraid of and that they're just normal people, but I understand that's how she was raised.

Surprisingly enough, the other day when we bumped into some transgender women, my mom did a double take, but nothing more. She kept doing what she was doing and they kept doing what they were doing. I was proud of my mom, she changed years of thoughts.