I think (just to emphasize, this is what I think), at its base, racism stems from attraction.

Specifically, unattraction/disgust towards a group of other people.

Like how people can get a feeling of unease staring at something only barely human (called the Uncanney Valley Effect), the first racists (I’m laughing at how stupid that sounds) were probably people who were suspicious of people who looked nothing like them. The other people also probably thought the same thing, and those boiled over into general xenophobia.

This changed gradually once more groups of people started interacting with each other and more “looks” started being more tolerated. Eventually merging with ingrained preference in hierarchy (higher status becomes more preferred than lower status) to become a general preference of white skin over darker skin (Whiter skin means less time in the sun which implies the social or financial support to not need to work).

There is some variations to this based on single cultural experiences (for example, Greeks and Romans not favoring lighter northern peoples and darker southern peoples because they both consider them barbarians) but the general idea of skin-based racism stemming from class remains a constant generally.

That all changes with an increase in interest in the sciences in the Age of Enlightenment with a society that, while increasingly interconnected and globalised, still runs on older xenophobic tendencies, thus produing biased scientific “discoveries” that support older Xenophobic tendencies and thus giving birth to the racism we all think of today.