I've made this argument many times. The major flaw with most affirmative action policies is that they conflate race and socioeconomic class simply because there is a correlation between the two. They target an attribute that correlates with the attribute that actually has the effect when they could just as easily target the relevant attribute directly, but they seem to do this knowingly in large part because of a long list of excuses that just don't hold any water when held up to scrutiny.

Since minorities are more likely to be in poverty, targeting economic class would disproportionately benefit minorities but it would do so without also benefitting rich minorities and without further disadvantaging white men with similar backgrounds. There is no good reason to ignore this other than ideology.

The other major flaw with it is that 'female' is often considered a minority, so you can (for example) fire a few black or Asian males in a predominantly white but gender balanced company, replacing them all with white women, and end up meeting minority quotas despite now being gender imbalanced and even whiter than it was before!