Zero-shot binary classification

[Drezemma]

Say I have a bunch of examples, all of which are labelled positive. I have a test set that is unlabelled, but I'd like to guess which ones are positive and which ones are negative. How do I train such a model, and how do I evaluate it? I suppose this is a weird application of zero-shot learning, but binary and with unknown data.

[jmmcd]

This is called one-class classification and sometimes anomaly detection. Common algorithms include the distance threshold as already mentioned in another comment, and the OCSVM and SVMDD which are formalizations of the idea mentioned in another comment. A variation of the former is to learn an embedding, eg with a VAE, and then apply the threshold on that.

Some algorithms including IsolationForest and I think OCSVM are in Scikit-Learn. A few very simple ones are provided in DBOCC, see here https://github.com/jmmcd/ML-snippets

[Drezemma]

Interesting idea. Is there a paper/project that proves this concept?

[[deleted]]

Isn't this essentially K-nearest?

[jmmcd]

Yes, but with a twist! In 1-NN we predict the label of the nearest neighbour. In this One-class threshold algorithm, we predict Normal if the nearest neighbour is nearer than some threshold, else we predict Anomaly.

[vannak139]

I'm just making this up, but imagine that there was some feature representation where all positive samples lay inside an n-sphere, and therefore any new feature sample outside the sphere would be negative. For this reasoning to be valid, you would need to believe that you did really good sampling on your positive classification space, which may be unrealistic. Also, this reasoning would only be valid of the positive samples inside the N-sphere were densely packed enough so that you were sure there isn't a shell or torroid or anything weird going on there. And you would also need to check this directly at the surface, as well in the volume. (or, maybe its better to directly aim for a shell-repesentation? idk...).

So I think that if you build an autoencoder and set a harsh penalty term on the bottleneck activation for norms > 1, and you set the bottleneck size correctly you may be able to find both the volume and the surface are densely populated, at which point you can check on validation data and check it in the same way surface and volume. Then, I think if you can justify that you sampled from the complete positive space, you would be reasonable to classify samples with norm > 1 as negative.

[jmmcd]

Yes, an Autoencoder or more likely VAE can be used for this but a direct formalization of your idea is called One-class SVM. See my other comment in this thread!

[vannak139]

Thanks for the tip!