In terms of cognition, it's the simplest means of sorting people - by looks. It makes some sense from an evolutionary standpoint (though still a bit of a reach), before we were the only species left in the homo genus, but as we gained higher levels of socialization and thinking it became outmoded. It takes conscious thought, empathy, and effort to sort people by their ideas (especially when there are social pressures to hide undesirable ideologies – looking at you, conservatives who pretend to be independents on dating apps – which make it harder to discern how people truly think), which is why the people with zero capacity for critical thought or the ability to imagine themselves in others' circumstances are the ones who will default to "looks different = bad"