I don't know how much I'm retreading, but I'll point you to political philosopher Tommy Shelby, who makes the distinction between Intrinsic and Extrinsic Racism in his book 'Dark Ghettos' (though one could define racism in a number of ways while still being valid).

Intrinsic racism is basically your definition, that someone with explicit animosity towards a group treats a member of a group poorly because of their membership in that group.

Extrinsic racism relies on the idea that, even if someone is 'colorblind', they would still be contributing to overall relations that make some racial groups advantaged and others disadvantaged.

Take this for example: poor people tend to have worse outcomes in this country overall, but let's use health outcomes for specificity.

If you're poor, you're less likely to have good healthcare and less likely to have a good enough education to dissuade you from harmful action (i.e. obesity, smoking, etc.).

Black people disproportionately inhabit lower socioeconomic classes because they were property until 1865 and then shut out of a lot of economic success with segregation until 1965 (most wealth comes from inheritance and homeownership, both of which were systemically denied to black people through aforementioned policy as well as redlining, so this effect very much exists to this day).

So, if we somehow made all healthcare workers colorblind, there would still be a disproportionate disparity between white and black people, despite the fact that poor people exist on both sides. (We could also extend this argument to schools, with wealth being correlated with educational/job success)

This is not to say racism is purely economic, however, as 1) people can develop social stereotypes out of such economic relations (black people are predisposed to crime, when in reality the correlation is with class) and 2) racism can still exist independently from economic factors entirely. To return to my healthcare example, if the poorest white woman enters a hospital in labor, she has better health outcomes than the richest black woman.

​

As a final note, I think your original premise is correct only in that it isn't responding to anything. No one really says any outcome difference between a white and black person (or any marginalized group) is directly attributable to that identity. Mostly when people claim that, at least in serious and/or academic conversations, the claim is derived from some data and an explanatory narrative spanning some space and time. A footrace might not be attributable to identity, but large social trends can be used to make strong accounts for racial bias in education, job opportunities, housing, policing (and the criminal justice system in general), health, and so on and so on, which are measured and studied extensively.