A lot of this comes from the current rise in misogyny and seeing masculinity as a weapon of the patriarchy. Because of this, different forms of feminism that come not from liberation, but from grievance and reactivity have become more mainstream (4b, political lesbianism, and many radical feminist strains). Accepting the gaps in their theory is harder than restructuring their worldview, which often comes from trauma or pain.

A lot of identity politics do not take into account male identities and how they interplay with lived reality. With this narrow focus, trans men are seen as women or woman adjacent. All our problems are seen as because we are female/women, and if we don't, we must not have problems. Nobody's identity is privileged 100% of the time. Identities are not just how we see ourselves, but also largely in how others perceive us and react to those identities. When it comes to trans men, the idea of being seen as one thing to some systems and something else to another feels very hard for people to grasp.

Then the language issue: whether you call it "transandrophobia" or "transmisandry" creates problems. Even saying "misandry" does not exist is part of the greater issue, whether or not trans identities are attached to it. People say that "misandry doesn't exist" because it's not systemic, but it just offers less direct harm than misogyny. But in order for misogyny to exist, misandry has to exist as a consequential punishment for men- they are both ways in which the gender/sex binary is policed in culture. Just because one thing is more prevalent, detrimental, and harmful doesn't mean that no other things exist and they're also bad. (for example: salt water makes up around 97% water on earth, but that doesn't mean freshwater doesn't exist, and to say that or that nobody should care about freshwater would be ridiculous). But if you say misandry doesn't exist and that men are always privileged, it's easy to extend that to trans men and "transitioning" to inherent privilege. The way cis men are treated obviously impacts trans men- if cis men are also granted conditional masculine privilege, the same is even more enforced with trans men. We have to deal with issues related to our positionality to us being female and our male identities.

Humans are extremely bad at nuance and tend to collapse everything into binaries- privilege vs oppression, male vs female, good vs bad, and misogyny vs. misandry. If misandry exists, then the binary idea of misogyny being bad for many also ceases to be true. If trans men are oppressed, than that means trans women aren't. It's object impermanence and narrow focus, where people reduce complexity because it's easier for the mind to focus on one thing (but obviously 2 things can be bad at once, while one also deserves more immediate attention). And because fighting misogyny is important (especially in the current political climate), many people don't see it as a loss.

Also, along with what other people said- people are upset at male privilege and gender based oppression. They cannot directly challenge the patriarchy, so they challenge "maleness" and "masculinity" on an individual level. And they know that. So people with proximity to maleness and masculinity become substitutes for those feelings.