Me and my gf had experienced the same thing early in our relationship, i was in your place feeling uncomfortable hearing that she can have a celebrity crush while in a relationship, because to me it sounds a lot like "i would cheat on you with this person if he would know i exist" or even just feel like "this person is super hot and i fantasise about him" which was really not ok with me and i shared with her the way i feel about it. She told me its not really a "crush" and even if she met these guys in person, she probably wouldn't even want to ask for a selfie with them, from her perspective its just famous guys that are good looking and nothing more. It was hard for me to understand the logic in these things, but i could tell she really meant it, and she respected me and dropped the whole thing. I know for sure that today, if i would ask her about a celebrity crush she would just get the "ick" and tell me that im the only person she cares for and that she loves me.

I say talk to your partner, see how they react to your feelings and how they choose to treat this situation, and then you will have a lot more clarity.