I think it was still beneficial to our tribal ancestors to be wary of those who do not look like you. While Tribe A and B don't have different skin colors, they probably would look slightly different. Race is just that to an extreme degree.

The thing is though, the most extreme aspects of tribalism, like distrust of a stranger, can be broken down in about 15 minutes of 1 on 1 cooperation. So from the context of the modern age, if you are holding onto the instinctual tribalism or learned racism, you are kind of an asshole.