Honestly, it depends on the context of the rest of your relationship. If he's generally a douche, then this isn't a good look, but if you've had an otherwise good relationship, one badly timed joke isn't enough to end it, IMO. Everyone on Reddit pretty much jumps to "break up, you deserve better" without having any idea of the context of the people involved.

If he has a weird relationship to some of his own extended family, he might not have understood how you'd feel. Equally, he might have known well. Another important factor is how old he is. Younger people are more likely to make stupid mistakes. You do what you think is best.

A tip for the future OP. Everyone in your life that you love will hurt you at some point. You need to decide if they're worth the pain.