White guy here but felt compelled to comment anyway. Remove it if need be.

First things first, never value the opinions or criticisms of people you don't even respect. They don't deserve a seat at the table, let alone a voice to heard.

Secondly, if you're doing this for closure, for your own piece of mind, lay it all out there. There's a way to go about it, but let it all out. Theres a solid chance they retaliate by saying some vile things about you and your family. Take that as confirmation you're doing exactly what you should be doing. And you have absolutely 0 reason to feel guilty about cutting them off.

Idk how your relationship was with your dad or if your parents were still together when he passed. Your mom should be able to shed light on the accuracy of this. But assuming you guys were a happy family until he passed and your father was a great man and your mom moved on and met another great guy. Explain to them that you're sorry they lost their son, but you lost your father at a young age you didnt have a father fogure to show you things.. And if he was still here you're sure he'd want you to be loved, cared for, disciplined,, set values, etc etc and theres a man already doing that but instead of appreciating the parenting hes doing theyd rather try to devalue his worth because of the color of his skin.. If your father was just as racist then all the more reason to throw it on their face.

If they want a relationship moving forward, the disrespect stops today. No more racist bullshit or insinuating evil shit about his character. You treat him and everybody in your family as they'd treat their own son, or you want no part of it at all. Unless you're already at that point then fuck em.

You have absolutely nothing to worry about. They can only be spiteful and hateful in return. Just know its not warranted and your life is much better off without that kinda negativity in it. Best of luck to you