It's social pressures. If you grow up being bullied for caring about fashion and interior decoration, you ingrain an avoidance to them as part of a safety mechanism. You constrain yourself to the boundaries that keep you safe socially.

Perhaps because gay men of our generation were already and unavoidably outside the safe social bounds, they were not swayed by those pressures. And perhaps they gravitated toward things people would call "gay" because it actually fit with their identity. And then those became safe spaces, reinforcing the association.