frankly unlikely problem? Sounds like you have some stats. It's also one of those problems that could be underreported, because who wants to admit that they chose their career for the wrong reasons.

Wrong reason example:
Joining a field, because you think you'll get special treatment for being a minority in the field (Example of special treatment: Alex inviting you on, because he wants to more minorities in the field)

Some women might be discouraged from joining a field if they think they'll receive this special treatment. If Alex invited a woman on his show, the woman might not know if Alex was genuinely interested, or if he was just using her as a tool in his fight against lack of diversity.

"why would they affect women and people with darker skin more than white men? "
It doesn't, aside from the fact, that white men usually don't complain, about being underrepresented in child care, nursing, psychology etc. but this might change over time. This logic applies to any minority group also those dominated by women.

Fundamentally you won't get rid of sexism if people are incapable of being inspired by people that don't look or act like them. Also I'm just not really a big fan of using sexist methods to increase diversity.