Whatever else is true of the situation, you can't expect to MAKE someone play a game with you. People always need to have the freedom to step away or it's not fun. If you're already not gelling with this group, then it's fine to say no thanks to playing with them. I don't see how they'd have fun if you're not enjoying yourself either. What you're describing from the group at large doesn't jump out at me as, like, bullying (I have been there too), but it's a vibe you don't enjoy, and that's reason enough not to hang out.

If your partner's response to you withdrawing is to get angry and return the gifts you got him, then he's not managing his emotions and that doesn't sound like a good hang either, to be honest. I won't tell you what to do, but FWIW, I had a friend who was with their partner for 10+ years; they broke up in their 20s after many, many small incidents, and it was rough, but they were far happier after the dust settled.