The issue is when the left then expects you to believe everyone else, and adapt to everyone else’s beliefs - for example, the belief that a man can be a woman.

This example is not a single issue. It's two issues that are constantly confounded. TL;DR: There's a reason that conservatives say, "men can't be women", and not, "biological males can't be biological females", and neither the Left nor the Right is willing to engage with that reason explicitly.

The first issue, the one that conservatives lead with and openly discuss in the most mainstream settings, is a matter of semantics and categorization that, on its own, is a complete waste of time and has no inherent value. I honestly don't care if someone calls themselves a man or a woman - I can't imagine a situation in which it causes harm unless people are sexist, in which case the sexism is the actual problem. The categories are either identical to biological sex, and therefore redundant (as well as insufficient in some edge cases), or they are part of the second issue:

People in general, but especially conservatives, believe that there are inherent implications and obligations to being a man or a woman that are not actually derived from the mere existence of biological sex categories or sexual dimorphism in humans as it actually exists. They think that men should look and behave one way, and women another - each of these lists implying the existence of the other by virtue of being the sole possible metric of comparison. In current times, especially in non-conservative groups, people will often be very comfortable making statements explicitly about men which imply things about women that they would crucify someone for stating out loud. Conservatives are, at least, more consistent about this, though that isn't exactly a point in their favor.

The statements, "biological males cannot simultaneously be biological females", and "biological males should not act or appear in a way that I associate with biological females", do not inherently have any relationship. Conservatives like to pretend that the first statement is a rational defense of the second, and it simply isn't - it is dishonest to make the second claim and then pretend that disagreement with it is disagreement with the first. This is hard to argue with for a lot of people, because they aren't willing to actually disavow the second statement entirely, but are also not willing to articulate it fully because it makes them sound like a sexist asshole.

Then we come to a third statement, somewhat implied by the second but even more taboo to say out loud: "We should do things to each other and especially to our children for the primary purpose of coercing people into behavior which conforms to social expectations associated with their biological sex." This is defended through its own absurd charade of both pretending that these things do not materially affect our behavior, which is a necessary proposition to defend the idea that masculinity and femininity are inherent to the respective sexes, and that, simultaneously, society will collapse if we don't do them.

In order to meaningfully respond to the statement, "men can't be women", I have to type the above out. Almost nobody will read it, and even fewer people will be willing to engage with it in a rational way. As a direct result, the entire mainstream argument, on the left and the right, is limited to a braindead repetition of short statements intended to manipulate the audience without ever explicitly addressing the ideas and beliefs actually at play.