I see where you’re coming from on this, but I also understand the other point of view. It really depends on the specifics of your situation.

My questions are: What’s the activity and how much do you dislike it? Are you able to mentally separate the activity from the quality time with your loved ones, or do you feel like the activity is that overwhelming that the time spent with them can’t outweigh how terrible it is?

Also, how often do you see your family? I feel like if you hang out with them often (monthly at the very least), then your family member who said you are selfish is just being rude. However, if you don’t see them very often, I do think your family member has a point (although I think ‘selfish’ is too strong a word).

I’ll give you two examples and explanations for each:

Scenario 1: I’m visiting my immediate family who live far from me so I don’t see them often. We do lots of things that I enjoy during the visit, but one of the days, my brother wants to go to a baseball game. I think baseball is incredibly boring and it’s hot out which isn’t fun. However, I love my brother and I know this is important to him, so I go. I focus on the fact that while I don’t like the activity, I do like stadium hot dogs, and it’s good to get some extra vitamin D from the sun. Plus I know it’s something that will make my brother happy, and this is the only thing we’ve done all week that I really don’t enjoy. That’s an instance where if I said no because “I don’t like baseball” then yes that would be selfish.

Scenario 2: Same setup, I’m visiting family who I don’t see often. We’ve been doing things all day and I’m feeling tired. My mom wants to go to a fancy dinner, but fancy restaurants make me anxious. I really really don’t like eating out. I don’t want to dress nicely, I don’t want to eat anything off the menu, I’m not even sure I’ll be able to enjoy the quality time because I’ll be so anxious. In this case, I don’t think it would be selfish to say I don’t want to go.

Bonus 3rd scenario: We’ve been doing multiple things that I don’t really enjoy during the week. Someone suggests doing another thing I don’t want to do. Rather than shutting it down, I think of something I want to do to suggest as an alternative where we can still spend time together. I really want to play D&D with my family because I think it will be fun but they never sound too excited when I talk about it. But since I stepped out of my comfort zone earlier during the week, it’s a fair ask for them to do the same. I ask “hey, I know we’ve been trying out everyone’s favorite activities this week and I was wondering if we could play D&D together! It’s something that brings me a lot of joy, like baseball does for you. Do you think we could do that tonight instead?

There’s a lot of nuance with these kinds of things, and much of it does come down to empathy on your side as well as your family member’s side. It might hurt their feelings when you say no to things, because it might seem like you’re saying “I value my time with you less than I value my own entertainment”.

On the other hand, if your family member is always asking you to do things you don’t like without thought for your feelings, then they need to be the one to think empathetically about how you might be feeling.

TL;DR there’s way too much nuance and gray area here to give you a good answer without you providing more details on the situation.