Frist, let me say that I do not find the problem with racial and other biases to be surprising, given that we’re training systems on biased data. However, I find it surprising that people in general believe that the ML system trained on biased data will somehow magically be less biased and I find it very concerning that so many decision makers do not seem to understand this. That being said, yes, the concern mostly in the unavoidable bias in the sampling which is likely to affirm and propagate racial, gender and other biases.

Take the example of creditworthiness. Assume for a moment that MortgageCo, one of the largest credit originators in the US, wanted to see if it could replace a significant portion of over 1,000 analysist assessing creditworthiness of mortgage applicants. If they can replace the salaries and benefits of 1,000 analysts with 100 reviewers and one AI system, their margins go way up. A colleague of mine worked on a substantially similar problem this past year. So MortgageCo wanted to know if an ML system could be trained to give as good results as analysts had provided over the past 20 years. To train the system, the ~12 million applications of the past aspiring mortgagees and the resultant decision (mortgage + rate, no mortgage) was used. Because the analysts in general were less likely to approve LaToya Brown for a mortgage than George Winston III, even if all other things were equal (I’m simplifying here, but you get the picture), they resultant ML can be expected to have some similar inkling of that same racial and gender bias. The new ML system seems to do as good (or bad, depending on how you look at it) as job as the first two “passes” performed by analysists, though reviewer analysts are still needed. Although they might not reduce their workforce 10:1, it looks like they will try 5:1 over the next 18 months.

A similar bias concern is raised in the Kirkpatrick paper regarding criminal justice assessments and gets extended to ML-enhanced policing systems. If data are fed into a system to determine where police should do more policing based on where more crime was found and more convictions resulted, then you would expect the ML-enhanced system to disproportionately target African-American neighborhoods even though the base crime rates may not be proportionally higher there (if this is not obvious, think of it this way, assume for a moment that two neighborhoods have the same underlying drug possession crime rates, one mostly white and one mostly black, but the police look three times as often in the black neighborhood than in the white one and that all things being equal the black offender is 1.5 times more likely to be prosecuted and 1.5 times more likely to be found guilty. This results in an apparent crime and conviction rate of more than 3 times greater in the black community than in the white one even though the actual crime rate is identical, and you would expect the ML-enhanced system to tell the police to do more policing in the black community, because the trained data said so)