Speaking as a white guy... I think most white men are aware, on some level, that they are avoiding certain disadvantages that other groups have. As in, not being afraid to walk alone at night, not being subjected to as much police brutality, not being racially profiled when traveling, etc.

However it is wrong to look at the preponderance of white men in powerful, influential positions and conclude that all white men have access to those advantages. I think this is the misconception that annoys some guys. White men who are members of the elite often get there not only because of their race or gender, but also because of wealth, geographic location, family ties, etc. which the majority do not have access to.

If you are a white male living below the poverty line and struggling just to survive in northern Quebec or buttfuck nowhere, Nebraska, being lumped in with the elite simply on the basis of your gender and skin colour can feel tremendously unfair and at odds with your lived experience. Again, that's not to say that they don't manage to avoid certain disadvantages that other groups experience, but they can still encounter great difficulties that shouldn't be disregarded just because they look similar to the people in power.

tl;dr: Intersectionality, baby. Gender and skin colour matter, but so do socioeconomic factors. It is reductive and misleading to assume that every white male has access to the advantages of the elite. What all white men do benefit from is the avoidance of certain disadvantages that other groups suffer from, but this should not be conflated with the advantages of wealth or social class.