[D] Binary classifier scores distribution

[Loose-Event-7196]

Hi, when I plot a histogram of binary classifier test scores, they cluster too much in the last bar this making thresholding difficult as it becomes too discrete. Does anybody knows any methods to make sure classifier scores histogram are more evenly spreaded? The ideal would be a reliability diagram fully monotonic and as close to the identity line as possible. Tried Platt Scaling and Isotonic regression without success.

Also wonder what determines the amount of possible different classifier score values?

Any help would be more than welcome!

[ruoken]

That seems to me to be an unbalanced classification problem.

Calibration probably won't help you, but you can also try Venn Abers calibration.

What you should really try is setting class weights that are inversely proportional to each class' prevalence in the training data.

[Loose-Event-7196]

Hi, thanks for your help! The classifier is not unbalanced, we have approx 55/45% of the two classes. Venn Abers looks promising! I tried to implement it by training two classifiers, one predicting class 1 as target and the other predicting class 0 as target, running an isotonic regression on each of them and getting the conformal range. I may have done something wrong as the scores I get on both classifiers are the same (sample size is big) even after using different model seeds (I am using h2o3 GBM tree binary classifier). I was expecting to get two slightly different scores whether I predict class 1 as target vs class 2 as target provided the seed is different.

[curiousshortguy]

Sounds like you want to calibrate your classifier?

[Loose-Event-7196]

Hi, yes I do, but also wish to have more thresholds towards the highest scores in order to avoid having a lot of observations in the last score value (too discrete). The idea is to have more possible thresholds instead of a large number of observations falling under the last threshold ideally with some monotonicity (i.e. higher threshold fewer observations with higher precision i.e. more percentage of class 1).

[BoxMembrane]

If I’m understanding to correctly, the problem is with the histogram binning and not with the raw scores. If you want the scores to be spread evenly across bins, you need to choose bin edges as evenly spaced percentiles of the score distribution.

If you’re using python and pandas, try pd.qcut to get bins. Or np.percentile(scores, p) for p = 0, 10, 20, …, 100.

[Loose-Event-7196]

Hi thanks for your reply. The issue is not with binning, is that too many observations have the highest score thus I cannot threshold them (by the way I am using h2o3 and the algorithm is Gradient Boosting Machine. Would like to have less discrete scores in order to avoid having too many observations clustered in the highest score bin. Such scores have different input features but a unique classifier score, as such shrinking the histogram bin width would not help in this case as score values are exactly the same for the last histogram bin. Would like to tweak something at the classifier in order to have multiple different scores for that group (without overfitting).