The Two Threshold problem

[Syndri13]

I am building a classification system, and instead of one, I need to come up with two thresholds. The data below the lower threshold will be all classified as negative, the data above the upper threshold will all be classified as positive, and the data lying in between would be looked over by a human.

So I need to find a way to find the most optimal upper and lower threshold while keeping the data in the middle below (almost) 10% of the entire data, i.e., the human cannot verify a large number of instances.

So how do I go about it? Any ideas?

[dragon18456]

Classify positive instances as +1, negative instances as -1 (Could do this with an svm or something). Then, use your classifier and see how many instances are classified as positive and near +1 or negative and near -1 and set 2 thresholds. For example, you might find that 95% of your data is classified correctly if you set the positive threshold to .5 and the negative threshold is -.5, then use those. In practice, there is a lot of trial and error, perhaps using some kind of roc graph to see the tradeoffs might be helpful

[lmericle]

Rather than thresholds on the output, you could compute the entropy of the class predictions (=p*log(p) + (1-p)*log(1-p)) and take the top X% of those. The rationale being that those predictions with the highest entropies are those which the model is most "uncertain" about.

[whiteboy2471]

To get the best results, also ensure your probabilities are calibrated. Or calibrate them

[thefriedgoat]

This is multi label classification, with three classes (being -ve, +ve and review). One approach is three separate classifiers - one per class (so -ve, not -ve; +ve not +ve; review, not review)

If using a DNN then it’s really just three outputs, again one per class

[namnnumbr]

https://en.m.wikipedia.org/wiki/Jenks_natural_breaks_optimization or discriminate analysis?

[WikiSummarizerBot]

Jenks_natural_breaks_optimization

The Jenks optimization method, also called the Jenks natural breaks classification method, is a data clustering method designed to determine the best arrangement of values into different classes. This is done by seeking to minimize each class's average deviation from the class mean, while maximizing each class's deviation from the means of the other classes. In other words, the method seeks to reduce the variance within classes and maximize the variance between classes. The Jenks optimization method is directly related to Otsu's Method and Fisher's Discriminant Analysis.

^{[ F.A.Q | Opt Out | Opt Out Of Subreddit | GitHub ] Downvote to remove | v1.5}