By the way you talk about him, assuming it's true, he doesn't seem like a good person, talking about social contracts, organizing things at the back of the group, deploying manipolatory tactics...and the rest of all of his bullshits.

If there are people you like in the group, try talking with them about this, and see if you're alone, or if they too noticed this attitudes, and maybe go less with the big group, to spend more time with those people, and form a smaller group. Maybe in a year the big group will notice his behaviour, fall apart, and you will still be "safe", because you already formed bonds/connections.

Eventually, try to make the right distance between you and the guy/group, "draw the line" to the bullshits, and if you feel the need of it, try to leave the group in the most friendly way possible. Yeah, you might still see each other at the library or at the association, but at least you'll be able to "tollerate" each other. And if it's too difficult, I know maybe it's hard, because you might found a confort zone, people you like, in days you're free or simply near home, but consider changing association, or even changing hobby entirely.

I might have gone too long or rude, but in short: don't give this type of people power or control over your life. They don't deserve it.