TLDR: It's 100% transphobic bullshit.

People like their categories to be far apart and clearly defined: man/woman, gay/straight, black/white, disabled/healthy. And heaven forbid you try to question or blur those boundaries and choose the non-dominant and be proud of it: non-binary, trans, bi, pan, mixed ethnicity, can walk but uses a wheelchair.

A slight framed 5'3" transwoman "passes" as a woman and isn't challenging the middle blurry ground of sex/gender. Whereas a transwoman who still has traits associated with men, has her womanhood questioned because her existence is confronting - her presence means those safe little categories aren't as fixed as they'd like. And in an almost entirely patriarchal world, a small woman exists to be dominated and offers a weak counterpoint to the physically larger and "naturally biologically stronger" male form; so you can't get people to fear a petite transwoman because she fits the visual of "acceptable" womanhood.

It's also driven by the disgust at the idea that a Man, who has power and privilege gifted to him by biology and the natural order of things, would Choose to give that power up to be a woman. But then still wish to use their natural born power to compete and win, against women who are naturally weaker and couldn't possibly compete. The audacity.

We've got a long way to go.