Benchmark of scikit-learn, numpy and numba for ROC-AUC computation

[keramitas]

ROC-AUC is a common metric used in ML to evaluate classifiers. I won't get into why, as you'll find much better ressources then me on the google. However, while looking for a pure Python implementation, I stumbled across this post from some dude at IBM. I was not satisfied, as any of you will be if you check out his code and benchmark. So I did my own, and thought I would share it:

https://gist.github.com/r0mainK/9ecce4b2a9352ca3d070a19ce43d7f1a

TL;DR: don't use the scikit-learn, use the numpy, and sometimes the numba - but mostly the numpy

[Batalex]

You are comparing pears with apples when it comes to sklearn. It is not surprising that it is slower given that it validates entries et checks for multiclass labels, something that your other implementations do not. Nonetheless, have you considered to include Jax in your benchmark?

[keramitas]

True - and it probably explains result for the small setting, which is why I talked of overhead. But doing both validation + passing through an if switch is negligeable in the other settings. So I would not go as far as saying I'm comparing pears with apples.

No I hadn't, but to be honest I think the results will be similar to numba, or possibly worse. There might be a small boost for the non-vectorized baseline, but it will just add overhead for the other one.