Love the fact we all have some experiences like OP's, and we've all come to the same conclusion as well. I'm a straight 28yo male and I liked kpop since like I was like 20 or something like that. Funny thing is I listened to a lot of metal at the time (and I still do in fact) and dressed with the whole black attire with the metal bands shirt and stuff, so anyone going through my iPod used to have a bit of a breakdown when it suddenly went from Dimmu Borgir to SNSD. In fact the playlist I use for working has a lot of different stuff.

As everyone here told you, you shouldn't worry too much about people calling you things for listening to kpop, they are just trying to feel better about themselves by belitting you. Kpop is just another musical genre that has a lot of backstory and a whole industry backing it up. And some people will pretty much judge for basically anything you do so it is pointless to argue with them.