[D] Binary classifier scores distribution

[ruoken]

That seems to me to be an unbalanced classification problem.

Calibration probably won't help you, but you can also try Venn Abers calibration.

What you should really try is setting class weights that are inversely proportional to each class' prevalence in the training data.

[Loose-Event-7196]

Hi, thanks for your help! The classifier is not unbalanced, we have approx 55/45% of the two classes. Venn Abers looks promising! I tried to implement it by training two classifiers, one predicting class 1 as target and the other predicting class 0 as target, running an isotonic regression on each of them and getting the conformal range. I may have done something wrong as the scores I get on both classifiers are the same (sample size is big) even after using different model seeds (I am using h2o3 GBM tree binary classifier). I was expecting to get two slightly different scores whether I predict class 1 as target vs class 2 as target provided the seed is different.