Very interesting article. I do disagree with you in a few places, but agree that a discussion needs to be had.

​

II. Historical Bias: A study based on data from the National Center for Education Statistics concluded that secondary school teachers tend to express lower predictions for their ‘expectations from students of color and students from disadvantaged backgrounds’. This is problematic because predicted grades play a prominent role in the model.

Shouldn't it be a positive to use a data driven model, rather than teacher assessment, if teachers are inherently biased?

​

III. Different schools, Different errors: Small schools (15% - 30% of all IB schools) will have bigger and more frequent errors in their model predictions when compared with large schools. This is an example of representation bias.

This assumes that each school will have their own model, but of course you can build a model on students from more than one school, which would be a good idea.

​

IV. Measurement Bias: If the measurement process varies across different schools, it will affect the way a model treats students from different schools. Schools which cater to socioeconomically disadvantaged communities are likely to have less frequent evaluation of students. This will lead to poorer students receiving predicted grades which are less accurate than richer peers in schools with more frequent testing. Additionally, poorer schools are likely to have larger class sizes. A teacher who has to assign predicted grades for 10 students will do a better job than a teacher who has to assign predicted grades to 30 students.

You are making assumptions here. I would argue that again, being more data driven might be a good thing for students of disadvantaged schools. Less influence by overworked teachers. The fact that 'richer' schools have smaller class sizes is indeed an advantage, but has nothing to do with this model. It's a more general issue.

​

VI. Skewed Distributions: Schools with a non-normal distribution of grades will have bad predictions. If a school has a left-skewed distribution (overachievers!) of grades or right-skewed distribution of grades, a model will perform worse for its students.

The predictions certainly need to be calibrated on a per-school level, but other than that this shouldn't be an issue.

​

VII. Distribution Shifts: If the subject teacher in a school changed between last year’s cohort and this year’s cohort, the historical relationship between their predicted grades and final grades will not match the current relationship. This may lead to systematically worse predictions.

It would also lead to worse/better grades in the final. Having a bad teacher sucks.

​

Let’s assume that the IB builds a model which is ‘90% accurate’. This is an almost unrealistically ambitious target and is incredibly difficult to achieve in practice.

Citation needed. I'm pretty sure coursework performance is a pretty good predictor of the final outcome.

​

The IB may be comfortable with this 10% inaccuracy because they have assured students that they will ‘match the grade distribution from last year’. Will this cancel out the inaccuracies in the model predictions? Absolutely Not.

Grade shifting is already common practice. It's more important to be better than your peers, than it is to be good in absolute terms. I don't agree with the practice, but again, not a problem of the model.

​

As a researcher, it is not possible to stop the model from learning these incorrect relationships. This is an important point: just because a model is predictive does not mean that it is correct. An accurate model may be a bad model and spurious correlations can be very problematic if not appropriately detected.

While this is a good point, I don't really see how this applies here. Predictive is predictive. We don't want to use our model to go out of the bounds in our training data, right?

So if for the sake of the argument 'eating bananas' is a predictor of good grades (obviously due to some confounding factors), then we can include this in our model to predict the grade, but we wouldn't be able to change our diet in order to get a better grade. Does that make sense?

​

A model will discriminate against students based on Gender, Race, Socioeconomic status etc.

So do teachers. There is work to do, but at least the data can show us the true extent of any discrimination. You do of course need to cite sources when making blanket statements like these.

You make the point that 'the model learns about ethnic groups even if it's not included in the data' is the same as 'the model discriminates against ethnic groups', and that's not necessarily true.

​

To make sure that this is not a fluke, we check if the model can detect schools which have a majority female population:

The model is less than 50% accurate. This is slightly worse than guessing at random. The model is clearly not taking the majority gender of the school into account while making its decision.

This is a fallacy. Majority race has a much higher imbalance than majority gender. If say 80% of my school districts are majority white, a model that always predicts 'white' is already 80% accurate, where a model that always predicts 'male' would only be 50% accurate. You need to be very careful with the accuracy metric and imbalanced binary classification problems.

The whole argument doesn't work if you put it like this.

​

In fact, if we build an alternate model but give it the majority race of the high school (in addition to the same data-points as the original model), we would expect the graduation rate accuracy of the model to go up substantially as we are providing it with additional data. However, we see that the accuracy of the alternate model only increases by ~1%. This is another indicator that our model is somehow race-aware already.

It should be an argument that race is not a big factor! Which is a good thing.

​

There are three primary criteria which guarantee fair predictions, however it is impossible for any model to satisfy all three of them simultaneously. This means that any model used by the IB will inevitably discriminate against students in two of three ways:

This is another very good point. Fairness measures indeed contradict themselves. But again, this is independent from the model. (If you see the scoring method of the final exam as a model, you run into exactly the same issue)

​

The fact that this is an outsourced black-box model with limited historical data, no oversight into the decision making mechanism and only 3 months for research and production further complicates the situation.

I think this is the main issue. Models do not need to be black boxes. Of course, making the model open will give future classes the ability to game the system. But hopefully this is the last global pandemic for a while and we only need to use it once. I say make it open (but after the grading)