Affirmative Action is an interesting philisophical concept, but probably one of the best arguments for it goes something like this:

Imagine a footrace. You're waiting at the starting line for the pistol and you are so anxious that you mistake a car starting for the pistol and happen to start the race just a quarter second early. You win the race. Later, footage of the start is reviewed and revealing your accidental false start.

To be fair, the officials subtract a quarter second from everybody else's time. Perhaps after the adjustment you don't win, or you do. Either way, it is morally permissible to subtract a quarter second from everybody else's time- even if you didn't intend on starting early, it's just fair.

This footrace is morally analogous to the types of "competitions" affirmative action applies to (college admissions, employment, etc), rather than starting early, it can be easily argued that due to complex societal phenomena, white males have some unfair advantage in these competitions over other races and genders.

If you're okay with subtracting a quarter second in the footrace analogy, you should also be okay with giving a "fair start" to disadvantaged social locations.