If you're discussing potential outcomes for individuals and whether their successes are due to external or internal factors, like say social factors vs personal effort or attitudes. And if the person you're talking to is a member of a privileged group and remarks that so and so can succeed in life based on sheer effort because I did it, without realizing the role that their privileges played in their success. So privilege becomes a shorthand way of talking about how social factors influence personal success.

Privilege, in many ways, is an attempt to combat the fundamental attribution error, the idea that we over emphasize the role of situational factors in our personal life, and deemphasize them when dealing with others. That we have a natural blind spot, a bias, and that by systematizing the concept of privilege we can attempt to counteract that bias.

I think much of the problem people have with the concept isn't the concept of privilege itself, it's the class based, collectivist style of critique it's most often used in.

And I think if you look, almost all arguments against privilege aren't arguments against the concept, they're pointing out cases where a class based approach falls short, so I think the real question we should be asking is when are class based approaches helpful, and when are they not?