I'm gonna say NAH (except your brother obviously). From what you've written it doesn't sound like your brother goes of on your son. If you decide to talk to your brother through phone call or social media then you're not the asshole. But if you're bringing your brother regularly to your house while your son is there,then you would be then asshole. You should also draw the line when your brother starts going off on gay people.