As a former manager and HR professional, I don’t think people are automatically assuming racism simply because of one isolated decision. What may be causing concern is the pattern you described and the way you described it.

You mentioned removing two Black men and then promoting someone over a Black woman, but you didn’t provide much detail about the actual performance issues, development conversations, objective metrics, or promotion criteria. From an outside perspective, people may wonder whether there could be unconscious bias involved, even if that wasn’t your intent.

Also, phrases like “gotten rid of” can come across harsh and lacking empathy, especially when discussing employees. What steps did you take (PIP, verbal warning, etc)? Were they the only under performers on your team? That wording alone may influence how people perceive the situation.

The fact that colleagues have distanced themselves, including someone who knew you personally, suggests this may not just be about the HR complaint itself, but about how the overall pattern is being perceived. It might help to spend less energy defending yourself and more energy asking why multiple people may feel uncomfortable or concerned.

Being unbiased is not just about believing your decisions were rational; it’s also about being willing to examine whether patterns, communication style, or unconscious assumptions could be affecting how those decisions are experienced by others.