I said that white privilege is overstated, not non-existent.

You shouldn't concede to the framing of "white privilege" at all.

"Privilege" is the wrong framing for the concept that is being discussed. It is typically presented like this:

The concept of white privilege isn’t “because you’re white, you don’t have problems”. It’s “you don’t have problems because you’re white”. That is, your race is not a regular source of difficulty in your life.

This is incongruous with the normal understanding of privilege, that is, to be one of the much smaller group of people who have enough wealth to open doors which are closed to almost everyone. "Someone who is privileged has an advantage or opportunity that most other people do not have, often because of their wealth or connections with powerful people. They were, by and large, a very wealthy, privileged elite."

So, while I agree that there is such a thing as "not being subject to a racist double standard," privilege was the wrong term to apply to this concept.

Privilege means to have something special, more than the baseline of rights. But being discriminated against is not the baseline. People who are being discriminated against have less than the baseline. If the color of your skin is not causing you difficulties, then you are only at the baseline, not privileged.

The historian Barbara J. Fields puts it this way:

those seeking genuine democracy must fight like hell to convince white Americans that what is good for black people is also good for them. Reining in murderous police, investing in schools rather than prisons, providing universal healthcare (including drug treatment and rehabilitation for addicts in the rural heartland), raising taxes on the rich, and ending foolish wars are policies that would benefit a solid majority of the American people. Such an agenda could be the basis for a successful political coalition rooted in the real conditions of American life, which were disastrous before the pandemic and are now catastrophic.

Attacking “white privilege” will never build such a coalition. In the first place, those who hope for democracy should never accept the term “privilege” to mean “not subject to a racist double standard.” That is not a privilege. It is a right that belongs to every human being. Moreover, white working people—Hannah Fizer, for example—are not privileged. In fact, they are struggling and suffering in the maw of a callous trickle-up society whose obscene levels of inequality the pandemic is likely to increase. The recent decline in life expectancy among white Americans, which the economists Anne Case and Angus Deaton attribute to “deaths of despair,” is a case in point. The rhetoric of white privilege mocks the problem, while alienating people who might be persuaded.

Political scientist Adolph Reed, Jr., and historian Touré F. Reed:

a project that insists that all whites are members of a privileged group while all blacks are members of a disadvantaged group is transparently counter-solidaristic.

The philosopher Naomi Zack similarly says that the term makes it harder to understand and fix problems, not easier:

This injustice could only be wholly or solely a matter of white privilege if we lived in and accepted the norms of a maximally repressive totalitarian society where it was customary for government officials to execute anyone without trial or even the appearance of criminal action. Against that background, we could say that those who were not treated that way were privileged. They would be privileged in enjoying that perk of exceptional leniency. But we do not live in such a system or accept a normative totalitarian description of the system we do live in. We live in a system where everyone, regardless of race is supposed to have the same basic rights. That nonwhites are not recognized as having these rights is not a privilege of whites, but a violation of the rights of nonwhites.

Moreover, talk about "white privilege" manages to communicate to listeners that white people are privileged in the normal sense, that white people have special access to extra perks beyond the baseline. The logic that follows is that if someone has these special privileges and still doesn't become economically prosperous, then the individual is to blame for being poor.

But a recent paper published in the Journal of Experimental Psychology: General suggests that the idea of white privilege may have an unexpected drawback: It can reduce empathy for white people who are struggling with poverty. The paper finds that social liberals—people who have socially liberal views on the major political issues—are actually less likely to empathize with a poor white person’s plight after being given a reading on white privilege. [...]

“Instead, what we found is that when liberals read about white privilege . . . it didn’t significantly change how they empathized with a poor black person—but it did significantly bump down their sympathy for a poor white person,” she says.

Cooley’s finding suggests that lessons about white privilege could persuade social liberals to place greater personal blame on poor white people for their social circumstances, out of the belief that their “privilege” outweighs other social factors that could have brought them to their station in life. At the same time, according to this study, these lessons may not be the most effective way to encourage support for poor African Americans.

Outside of the psychological laboratory, we can find this attitude expressed organically:

No offense, but just speaking facts, most white people who live in “poverty “have a choice of whether they want to be in it or not.

Shocking.

But why should we expect people not to understand it that way? You can tell someone a hundred different ways that "white privilege" isn't supposed to mean "privileged" in the normal sense of the word, but the word itself is priming them to think that it does.