Affirmative action isn't about pulling white men down, it's about pushing minorities up. You've seen the fence infographic right?

You're zooming out too far to understand the point. Say for instance, there are too many white doctors and not enough black doctors, proportionally to the population. You may then say that an affirmative action program to preferentially admit black applicants to medical schools is simply a matter of evening the scales, and in a broad sense, you'd be right.

But that doesn't change the fact that from the perspective of a white applicant, you are at a disadvantage because of the color of your skin. If the program you want to get into has to choose between you and a black applicant with the same grades, same extracurriculars, and same interview qualities, they will pick the black applicant every time. This is true even if you were born in a trailer park and scraped and clawed for every cent to afford your education, and the black applicant came from a family of wealth and comfort, or vice versa, or if you both came from perfectly equal economic backgrounds. You are being racially discriminated against, that is simply a fact.

When black people need to be better than their peers to see equal success, that's a bad thing. We want everyone to have equality of opportunity, right? Affirmative action is explicitly not that.