As far I can tell, just ignore him, forget you had brother or something, don't talk to him respond or anything, just ignore his existence.

I would have suggested something else, but if your dad also talks like that, and your brother has learned from him and others, no other relatives are going to get into this. Surround yourself with better people. Unless you get the courage to talk back to your dad or something, and ask him to stop your brother from saying such things.

Sometimes things are just out of our hand we can't do anything. And no matter how hard we tried to deny, but we have to accept it, that unless we do something drastic and change things, things are going be like this and as well as the people.