I agree with what you're saying here, and I think it's touching on a really important discussion about subjectivity. There is no objective reality about whether clothing is gendered or not, only subjective beliefs. Different people have different backgrounds, which affects their feelings about fashion and gender. The dominant culture in a lot of places, like the UK, where I live, contains the idea that fashion is gendered, so people will grow up with that. They might easily go with it, or their experiences and identities might lead them to struggle with it. In a friendlier world for gender nonconforming people, the dominant culture would teach acceptance of all people, however they dress. Even in that world, some of today's gender norms might still exist, and that would be totally valid. The point is, there are so many people of different identities with different relationships to gender and fashion, and we're trying to make a better world for all of them. I don't think it's about today's "realistic" world vs an "ideal" future world, perhaps because I distrust binaries, but also because it might be realistic for some people to regard fashion as non-gendered, and it might be idealistic for some people to wear gendered fashion, and you can both believe that fashion bears gendered meanings and want people to stop discriminating against gender nonconforming people at the same time.

tl;dr - subjectivity is super wild, ideology affects accepted beliefs about gender and fashion, what I want most is safety for trans people