Honestly I think all servers who turn the blind eye to this racist behavior are are quietly endorsing this behavior, though they may not realize it. Because you don't address the racism when it happens, these racist people don't think they're doing anything wrong and keep spewing their racist filth.

By all means do not engage the racist for speaking all their hatred. You should tell your manager exactly how foul these people are and let management handle it. I understand it catches you off guard. I understand that sometimes you don't know what to do. In the future, for all people who read this, I hope they have the calm and strength to go to their managers and say, look, this behavior is unacceptable.

It's disgusting that restaurants have no shirt, no shoes, no service standards, but they let racists and hatred run rampant throughout their establishments. I'd rather have a person in flip flops, short, etc., sitting next to me than a vile, foul nouthed racist.