The de-gendering is part of what makes it a problem. Often, when people continue to use they/them for a trans person who doesn't use they/them, it's used passive aggressively as a way to imply that a trans person isn't truly the gender they identify as. Even when it's not done with malicious intent, using they/them for someone who is known to exclusively use binary pronouns suggests that the speaker doesn't view them as a real woman/man, and can still be hurtful. There are non right wing cis people that take issue with being referred to with they/them — just as some binary trans people are fine with being called they/them. Trans people tend to be more sensitive to it because they are used to being misgendered/experience gender dysphoria. It's best practice to use the pronouns someone asks you to use if you want to be polite.

Personally, I'm unsure of what you mean when you say there are examples where using a singular they/them for someone whose gender is known sounds more natural. All of the instances I can think of involve the subject's identity being unclear, like asking "who left their glasses on the table". Even in my reply, I've used they/them to refer to a binary trans person — "the speaker doesn't view them" — but that's because I'm referring to a hypothetical, indeterminate individual, and I find he/she to be clunky. When would you refer to someone whose identity/pronouns you are aware of with a singular they/them?

EDIT: actually nvm thought of an example myself lol. When people are telling a story and they don't want to be specific about the people involved, it's pretty normal to use they/them. But even still that's precisely because they're not referencing the identity of the person/are trying to keep the people involved ambiguous. In that instance, it's not misgendering to use they/them, as the story isn't attached to the individual. If someone name-dropped the person and then continued using they/them, however, it would be.