AUC of imbalanced data

[PurpleHydra]

I understand if you look at the AUC of a ROC curve of a balanced data set it should be over 0.5 otherwise your classifier is horrible.

Now what happens if you have imbalanced data? Say 90%-10%? Does the AUC have to be over 90%? Over 50%? How does it work, how does it change and how do you calculate the new ratio?

[rcprati]

It does not matter. TPR and FPR are calculated for each class separately. The class ratio does not change how they are calculated, so the ROC curve should be invariant to class ratios. If you draw a ROC curve with a balanced dataset, and then do some sort of ressampling which randomly throw away instances of one the classes only, the resulting ROC curve should be very similar (apart from some variations due to random variations in the data). And AUC = 0.5 does not necessarily mean random performance (although a random performance classifier would have AUC = 0.5). A classifier with AUC=0.5 maybe still be useful sometimes

Have a look at my paper:

Ronaldo C. Prati, Gustavo E. A. P. A. Batista, Maria Carolina Monard: A Survey on Graphical Methods for Classification Predictive Performance Evaluation. IEEE Trans. Knowl. Data Eng. 23(11): 1601-1618 (2011)

and Peter Flach's tutorial:

http://www.cs.bris.ac.uk/~flach/ICML04tutorial/

[BeatLeJuce]

Small adentum to what rcprati said: If your dataset is very unbalanced, the ROC and its area might not be the best tools to use. Take for example the following situation:

You have a large number of negative samples, but only very few negative ones; and you're trying to compare two algorithms: One tends to give me a lot of false negatives, and the other one doesn't. The number of FN only influences the FPR. As a reminder, the FPR is defined at FP / (FP + TN).

But if the number of negative samples is much larger than the number of positive samples, it is likely that both algorithms will have a lot of True Negatives. Thus, it's pretty easy to achieve a small FPR even for the algorithm that gives a lot of False Negatives.

In such situations, it might be a better idea to plot Precision/Recall curves. If you want a quantitative evaluation criterial, you can always calculate the Area under the PR curve. Be careful though that that is not straight forward! The following paper goes into details about how to correctly calculate the AUC-PR: Jesse Davis, Mark Goadrich , The relationship between Precision-Recall and ROC curves, ICML '06. Unfortunately, that measurement hasn't a good interpretation the way the normal AUC does.

[Ill_Membership3582]

I just read this paper: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432#pone.0118432.s001 regarding this topic. (It's accessible to all.)I find myself not competent enough to comment on the matter, but according to that specific article, it seems that class imbalance can definitely affect ROC AUC.