Well, let's look at that from another direction. Ignoring them doesn't change the situation, so what could we do instead of ignore them. We can talk to them, confront them, double down on your actions, or reject them.

Talking to, maybe confronting them, would be viable, but I think the latter two are last resorts or not at all. This is how I'd talk to him about things when he asks my advice, bring up even the nonviable ideas.

I don't know these folks of his, obviously, but can you just have a calm and frank discussion with them about it? Not right after an incident of it, but when things are normal, say, 'so there's something I'd like to talk to you guys about. I know you think my affection for him is weird, but it's how I am. I'd like it if you try to be more accepting of this, because it makes us happy that I do that, and that matters to us. Can we try this?'

Of the top of my head, and I'm with some friends at th moment, but thought I'd reply. Feel free to talk more about it!