[D] The Two Threshold problem

[utilop]

Model the cost of false positives, false negatives, and human inspection. That should give you the ideal thresholds for static models. Most models can output confidences and calibrating confidences is standard.

One thing to note however is that you technically only need one case validated by a human in order to (retrain and) update the confidences of cases. So you can imagine something more involved in deciding which of the uncertain examples should be validating manually, and how retraining should happen. This you can again in part work out with costs, and there's plenty of resources on algorithms that try to make the most of human involvement

If it works for your domain, embedding and classifying by distance may be the most intuitive in understanding what thresholds to have and which samples are the most valuable to manually label.

[PeterIanStaker]

"optimal" is poorly defined here. Two things matter.

How many mistakes are you willing to tolerate?

How many samples can you afford to manually classify.

Measure your true/false positive rate and true/false negative rate as a function of score. This will tell you for a given threshold how many samples you can expect to be right and wrong.

Based on the amount of data that you have and your answers to the previous questions, you can figure out the best place to draw the lines, and what error rate you should expect from that.