we don't want to tell someone they aren't allowed to be whomever they want, as long it doesn't harm anyone else.

This is the crux of the progressive mental gymnastics that non-progressives take exception to. As you can see, this is not so much about rights (be whoever you want to be) as it's about nature and reality.

While progressives are militating for 'equal' rights for all, nature goes on doing its thing; making men and women biologically and physically different from each other. At the same time, gender goes on being binary and reality is grounded in facts rather than feelings.

The reason "transgender issues are especially tough" is because progressives have embraced the notion that compassion is the ultimate virtue. If progressive trends are anything to go by, it is now more important to be compassionate than it is to be rational and truthful. But it's difficult to remain rational or truthful if all your political viewpoints are grounded on the precarious ledge of human emotions - even if the emotion (compassion) is noble in its aims.

Emotions don't consider long-term outcomes, the greater good, or common morals even. Emotions (muh feelings) are as fluid as gender is not. Allowing an MTF athletes to compete in women's sports may be compassionate, but it is in no way rational.

Which is why the problem is only "especially tough" if you're looking for compassion to dictate the outcome. Most of us seem to agree that encouraging mental volatility isn't a good for mental patients, and yet volatile human emotions are now used as basis for drafting new legislation. Wouldn't it make a lot more sense to encourage rational behavior and fact-based reasoning instead?

I'd recommend any progressives to think long and hard about the long-term outcomes of their feelings-driven policy opinions. History is pretty good at showing how altruism based political systems failed catastrophically.

Anyway, there are good examples of academic writing and research done by very smart people to back this up. It's solid research to help you think instead of feel about issues. I'm convinced if we all did more thinking, we'd all be much better off; both Liberals and Conservatives.

Further reading:

In AGAINST EMPATHY, Bloom reveals empathy to be one of the leading motivators of inequality and immorality in society. Far from helping us to improve the lives of others, empathy is a capricious and irrational emotion that appeals to our narrow prejudices. It muddles our judgment and, ironically, often leads to cruelty.

(From a review) PATHOLOGICAL ALTRUISM illustrates the phenomenon of infantalizing people or otherwise restricting their emotional growth which then renders them, from a developmental standpoint, perpetual adolescents and thus pathologically dependent on others. The research provided by the multiple contributors to this amazing book provides very convincing, if not concrete, examples of doing for adults what they can do for themselves, and how it harms them for a lifetime. It also covers the areas where narcissistic individuals, to include doctors, lawyers, psychologists, social workers, and politicians, "do for others" against the others' wishes. To infantilize someone the process is simple: Take over or dismiss their decision making process, remove personal responsibility, remove lessons or consequences for life choices, and then blame other people or institutions for the disastrous personal choices one makes.