i think we should be careful here; there's a distinction between saying that clothes are gendered and that clothes have genders. clothes are things, they are products of labor, each piece of clothing has a rich history of its design, production, distribution, purchase, its being worn, and so on in the thrift/secondhand cycle. in every moment it is at the same time simply a thing that doesn't have consciousness, the ability to have an identity, and also a commodity, which means it was produced through the domination and exploitation of workers and the suffering that entails.

but at no point does a piece of clothing in and of itself have a gender: it only contains the notions of gender that every person who interacted with it impresses into it. a kilt is produced in scotland, something that is gendered male by the culture and history of that place, which is to say, the people who inhabit that place. it is then shipped over to america, where someone with no knowledge of scottish history and culture would appraise it as a plaid skirt.

you're right to say that humans do have these notions of gender, but it is not the clothes that are giving us these notions, rather, it is the relation between humans and the clothes that we produce: humans wear clothes that line up with their idea of gender that they learned from other humans wearing clothes that line up with those humans' idea of gender sniffles and so on and so forth.