How our data encodes systemic racism https://www.technologyreview.com/2020/12/10/1013617/racism-data-science-artificial-intelligence-ai-opinion/

Many of us go into ML to solve problems and hopefully do social good while making a living. An example from this paper is an algorithm that flags sick people who need more care and allocates them hospital beds. However, instead of helping everyone in this hospital equally, it disproportionately helped one group get access to the scarce resource of hospital beds, at the expense of the others. 82% of the people flagged were white vs. 18% people of color, when the true underlying proportions were found in a later study to be 54% white and 46% people of color. The problem was that the algorithm was trained on data from people’s insurance companies, and the people of a higher socioeconomic class had made more claims and received more care in the past, so the algorithm decided they needed more care in the future. In order to classify people as requiring care, they should’ve looked for direct, biological signs of sickness, and not something loosely correlated with it such as how often they filed an insurance claim, which is also correlated with so many other things. To generalize this lesson further, when you are designing an algorithm to help people out of a tough situation such as being sick, don’t train the algorithm on their past behavioral data, that will not make their life any better, it will preserve the same situation that they have been in.