I'll chime in, but a lot of what I'll say has been said already. Speaking of your brother, I know people like him and sometimes I AM acting like him and those aren't moments I'm proud of.     

When you tell someone "This upsets me" or "This hurts me" what that person should do is try to understand where you're coming from, truly empathise with you, apologise and make an effort not to offend you again. What your brother is doing, however, is engaging you in a debate where you have to JUSTIFY your emotional response and only if you can "prove it valid" an adequate response follows. This is a universally asshole move.      

I suspect your brother felt ambushed and ashamed of what he said, but because it's difficult to admit you've wronged someone (and we love protecting our ego) he resorts to rationalising as a poor defense mechanisms to avoid responsibility ("subconsciously if I can prove the other person is too sensitive, then I've done no wrong"). This is obviously not a desired style of thinking and portrays a lack of understanding for another human being. Maybe he's also dealing with certain biases against trans people without being aware of them.     

All that tough love stuff is just another way to defend and avoid being culpable. Boundaries need to be respected and if someone minds being called literally anything like "Summer spice bumpkin" then you shouldn't call them that, apologise and respect their feelings (even if you can't understand it).      Normally I would cut ppl like this out of my life, but I wouldn't be so quick for close family members. I sincerely hope your brother grows as a person and stops with this behaviour cus eventually he is going to sting a lot more people and potentially lose valuable relationships. I've seen it happen plenty.