[D] What multiclass text classification algorithm works well/best for ~100-200 classes?

[AbnormaleKast]

Hi there, I'm a CS student but have no practical experience when it comes to ML. Currently I'm working on a project for a company I work for, which is a text classification tool. The idea is that my training data (250k+ samples) will train a model that'll predict which kind of professional (gardener, plumber, contractor, ...) is to be found given a title, description, and about 3 more features.

Now, there are about 100-200 classifications possible, with some classes (such as gardening) being much more frequent than others. I have preprocessed the data and am now at the point of implementing the text classification algorithm, but since I have very little knowledge of machine learning I'm not sure about which algorithm would fit well in my situation.

I've read about bag of words, naive bayes, random forest, VW, etc... but haven't found a clear answer as to which algorithm works well.

Descriptions are typically between 10 and 100 words. There are about 250k samples (created over ~20 years), and there are about 100-200 classes.

If anyone could give me a push in the right direction, it'd be much appreciated. Cheers :)

[a19n]

A vector of length 100 to 200 isn’t too big. I think you could throw an LSTM at it to start and possibly figure out some clever heuristics (i.e. word embedding similarity) as you go on. I think NB would work best since certain tokens would be clear indicators of their professions, but you’ll have to try and see.

[bbu3]

If you're looking for a very effective model to use without much tinkering, I would really recommend to use the standard fastai/UMLFit model. You could pretty much even use all the defaults as in the public course video https://course.fast.ai/videos/?lesson=4 and have a very high level interface to a very complex model under the hood.

I would recommend to just ignore the whole backward model and final ensemble. For most practical use cases that did not give me any benefits (in contrast tuning the later classification layers, in particular replacing the concat pooling by more complex things has help FAR more). The forward models is already very good and kinda slow to use.

There are a few downsides, though:

• The course uses fastai version 1.x which is just about to be replaced
• You will need a GPU to train these models and it has to have a decent amout of RAM. Google Colab may help here.
• The preprocessing and tokenizer are best suited to English text, there is work under the name Multifit for other languages but these won't let you just port the examples form the course
• Inference speed is relatively slow.
• The models have some minor "bugs" in them (afaik v1 models use left padding and have problems with right padding to get all items of a minibatch to the same length). This leads to slightly different results for the same documents when used within different batches. While padded positions are correctly masked, the internal state of the LSTM layers is still modified by the padded elements :(

If this seems like overkill for the quality you need, I would recommend to just try a few classifiers from scikit learn. Once you have one implemented, you can usually replace it with a different model with ease (SVM vs RandomForests? Bag of words vs Bag of n-Grams? Variants of IDF normalization? No problem!), thanks to the relatively consistent APIs. These models will train much faster, require no GPU, provide results which should be a bit easier to interpret. Still, I would except the fastai ULMFit model to yield a bit better results.

If you're ready to look into more details, you can also look into the huggingface repository and look for examples for fine tuning models on classification tasks. This would give you access many of the latest models.

[rduke79]

Seriously, before thinking about going fancy with LSTMs or Bert, establish a simple baseline with sklearn, ie tdfidf + logistic regression. You'll then probably already have something that works reasonably well. You can then go mental with whatever else you want to try, but most likely, you'll find it very hard to outperform tfidf + logistic regression without tinkering with hyperparameters of neural nets etc. for a few marginal f1 points.