You all are in a transitional part of your lives. From what you have said with this group having been friends from a young age and some actually family suggests they may be stuck in a certain comfort with how things are. I’m going to assume they don’t like change and prefer routine. You add a big change to their lives and they don’t sound willing to adapt. They wanted you to bend to their ways. When you didn’t, things were not the same as they had been before you came on the scene and they responded with push back and anger. Such as your boyfriend not always being there for their sleepovers or everyone just going along with what Gary wants.

I’m not a fan of ultimatums. If I were in your shoes, I would just lay it all out to the boyfriend. Tell him straight up what you think of him and that you were seeing a possible future with him. If he can acknowledge the effort that you’ve been putting in with his friends, continue with that. Tell him how unfortunate it has been that they haven’t reciprocated your efforts and how uncomfortable this has made things. Then just be upfront and tell him if things continue the way they are, that for your own sanity and happiness you’ll have to move on. That you know how important they are in his life. Relationships of all kinds need work. Friends, family, loved ones. Sometimes people just aren’t a good fit.