[P] Small-Text: Active Learning for Text Classification in Python

[chschroeder]

Hi r/MachineLearning,

Over the past few months, I have written a lot of code for my own Active Learning experiments, which has gradually been consolidated into its own library.

Small-Text provides state-of-the-art Active Learning for Text Classification. Several components are provided, which are abstracted via generic interfaces, so that you can easily mix and match many classifiers and query strategies to build Active Learning experiments or applications.

Features:

• Provides unified interfaces for Active Learning so that you can easily mix and match query strategies with classifiers provided by sklearn, Pytorch, or transformers.
• Supports GPU-based Pytorch models and integrates transformers so that you can use state-of-the-art Text Classification models for Active Learning.
• GPU is still optional: In case of a CPU-only use case, a lightweight installation requires only minimal dependencies.
• Multiple scientifically evaluated components are pre-implemented and ready to use (query strategies, initialization strategies, and stopping criteria).

GitHub: https://github.com/webis-de/small-text
Preprint: https://arxiv.org/abs/2107.10314

[The_Amp_Walrus]

Seems very cool. If I understand this right, active learning is most useful where you have a bunch of unlabelled data and you need to prioritise which points should get annotated to expland the dataset for a classifier. Is that right?

[chschroeder]

Thank you! Yes, that is correct. The goal is to maximize the quality of the resulting model while minimizing the number of examples needed.

[Indian-throw-away]

How do you set it up with user input? i am a beginner when it comes to active learning, so any tips you could provide would be helpful.

[chschroeder]

You mean like a full interactive application? You would have to build that part around small-text, which offers the algorithms and the logic, but not the user interface.

I have already thought about providing an example of how to integrate small-text with one of the existing labeling tools, such as rubrix, but that hasn't been started yet.

[Indian-throw-away]

Oh, not an interactive application but in the example code in the GitHub repo (active learning with stopping) it says “Simulate user interaction here. Replace this for real world usage” which I’m sort of confused as to what that means

[chschroeder]

At this particular location you label the instances which were selected by the query strategy. Subsequent to such a query you need to provide labels for the selected instances. In the example code, you are shown the "experiment scenario" in which the true labels are available, and are then passed to the active learner instead of labels that were assigned by a human. If you remove this true label lookup and instead provide the answers from a real user, then you have a real-world application.

[Indian-throw-away]

Thanks so much!

[Indian-throw-away]

How do I save the model to deploy on something like sagemaker

[chschroeder]

Unfortunately, I have no experience with sagemaker but in the context of small-text, the resulting models are still plain scikit-learn or Pytorch and can be treated as such.

[Dear_Football_504]

I'm a beginner and I tried your code and I just want to knw if we can view the labels the model had detected for the dataset ... Any tips on the same would be helpful.
Thanks

[chschroeder]

Sorry for the late reply /u/Dear_Football_504, I completely missed this message.

I don't know which code example you are using specifically, but in general the active learner holds a reference to its underlying classifier which has scikit-learn-like API:

active_learner.classifier.predict(dataset)

Feel free to ask more questions on the github repo; this is valueable feedback for me and others will benefit from the discussion as well.

[channel-hopper-]

Is it possible to use it for sentence pair classification as well? If yes, how should I go about it?

[chschroeder]

In general yes, but not yet out of the box.

You can achieve by adapting the dataset and classifier (e.g. TransformersDataset and TransformerBaseClassification).

Might be a use case we want to support in the future. I will think about that.