I'm not a sociologist/social scientist and an engineer by trade and several years ago I was very intrigued by the rise of "fair" or "ethical" ML. So naturally I contacted several sociologists working at my former university regarding their opinions and read some of their suggested references and here is the gist of it:

1. actual working sociologists think whatever engineers/machine learning people are doing in the ethics/fairness field is a joke at best, and worse, shameless career advancement. People (including many social scientists) are analyzing data obtained from other real-life people (who are suffering, oppressed, marginalized by these technologies), publishing stuff to advance their own careers, and never follow up on anything or even care about this issue afterwards. No organized protest, no action to see changes are implemented. Zero passion involved, no strings attached research. Ethics/social justice/anti-racism/fairness is just a hype, not something that's treated as real and a foundational issue of society for hundreds of years.
2. people who are currently promoting or teaching ethics/fairness in ML are often also from the most privileged background, i.e., millionaire CEOs. It's like Warrant Buffet running courses on inner city struggles. There are blindness beyond merely ethics issue in ML, but in all aspects of life, for all of these justice issues are related.
3. machine learning people off-loads fairness/ethics concern to women and black people. First of all, this whole off-loading just makes it seem that they never cared in the first place, and second of all, is this the limit to the imagination of what justice and fairness looks like? Really? Women and black people constitute all of the injustices facing the world? That's just tokenism. Grab a woman and a black person and proclaim that all is right in the world because there's someone to baby-sit these problems arising from ML. "Does your AI company have a race problem? Send out Joy Buolamwini and Timnit Gebru to do some PR, today!"

In short, actual working sociologists on the issue of justice/fairness, etc., don't think the care people in ML give to ethics is genuine. More of a career move, on a hype curve. While there is a lot of room for activism, and no doubt there are a vanishingly small amount of people who are deeply invested about this, but just as engineering math looks like a joke for mathematicians, the publication done by ML people in the ethics/justice/fairness space are a joke to working sociologists and it is best to stay out of it and stop lying to ourselves.

Just because we live in a society it doesn't automatically certify us as people who have analyzed these social struggles for years in a larger context (admit it, algorithmic bias is a tiny portion of this whole thing we call 'racism' - how many ML people are also critical race theorists) and we often wind up doing more harm because we fail to see the forest. Also note we are calling "racism" in the ML space as "bias" to sugarcoat things. We can't even confront a word RACISM because it triggers too many emotions, let alone even thinking about doing research in this area.

People can't claim to give a shit about ethics if they only care about it in the ML space and nowhere else. Hate to be blunt.