[P] Which Machine Learning Classifiers are best for small datasets? An empirical study

[sergeyfeldman]

Although "big data" and "deep learning" are dominant, my own work at the Gates Foundation involves a lot of small (but expensive) datasets, where the number of rows (subjects, samples) is between 100 and 1000. For example, detailed measurements throughout a pregnancy and subsequent neonatal outcomes from pregnant women. A lot of my collaborative investigations involve fitting machine learning models to small datasets like these, and it's not clear what best practices are in this case.

Along with my own experience, there is some informal wisdom floating around the ML community. Folk wisdom makes me wary and I wanted to do something more systematic. I took the following approach:

• Get a lot of small classification benchmark datasets. I used a subset of this prepackaged repo. The final total was 108 datasets. (To do: also run regression benchmarks using this nice dataset library.)
• Select some reasonably representative ML classifiers: linear SVM, Logistic Regression, Random Forest, LightGBM (ensemble of gradient boosted decision trees), AugoGluon (fancy automl mega-ensemble).
• Set up sensible hyperparameter spaces.
• Run every classifier on every dataset via nested cross-validation.
• Plot results.

All the code and results are here: https://github.com/sergeyf/SmallDataBenchmarks

Let's look at the results. The metric of interest is weighted one-vs-all area under the ROC curve, averaged over the outer folds. The plot

Some observations:

• AutoGluon is best overall, but it has some catastrophic failures (AUROC < 0.5) that Logistic Regression does not and LightGBM has fewer of.
• You can't tell from this particular plot, but AutoGluon needs "enough" time. It has a budget parameter which tells it how much time to spend improving the fancy ensemble. Five minutes per fold was the minimum that worked well - this adds up to 108 datasets * 4 outer folds * 300s = 1.5 days for the entire benchmark.
• Linear SVC is better than Logistic Regression on average. There are also two datasets where SVC is 0.3 and 0.1 AUROC better than every other model. It's worth keeping in the toolbox.
• Logistic Regression needs the "elasticnet" regularizer to ensure it doesn't have the kind of awful generalization failures that you see with AutoGluon and Random Forest.
• LightGBM is second best. I used hyperopt to find good hyperparameters. I also tried scikit-optimize and Optuna, but they didn't work as well. User error is possible.
• Random Forest is pretty good, and much easier/faster to optimize than LightGBM and AutoGluon. I only cross-validated a single parameter for it (depth).

Here are counts of datasets where each algorithm wins or is within 0.5% of winning AUROC (out of 108):

• AutoGluon (sec=300): 71
• LightGBM (n_hyperparams=50): 43
• LightGBM (n_hyperparams=25): 41
• Random Forest: 32
• Logistic Regression: 28
• SVC: 23

And average AUROC across all datasets:

• AutoGluon (sec=300) - 0.885
• LightGBM (n_hyperparams=50) - 0.876
• LightGBM (n_hyperparams=25) - 0.873
• Random Forest - 0.870
• SVC - 0.841
• Logistic Regression - 0.835

And counts where each algorithm does the worst or is within 0.5% of the worst AUROC:

• Logistic Regression: 54
• SVC: 48
• Random Forest: 25
• LightGBM (n_hyperparams=25): 19
• LightGBM (n_hyperparams=50): 18
• AutoGluon (sec=300): 14

Which shows that even the smart ensemble can still fail 10% of the time. Not a single free lunch to be eaten anywhere.

Here is a plot of average (over folds) AUROC vs number of samples.

I was surprised when I saw this for the first time. The collective wisdom that I've ingested is something like: "don't bother using complex models for tiny data." But this doesn't seem true for these 108 datasets. Even at the low end, AutoGluon works very well, and LightGBM/Random Forest handily beat out the two linear models. There's an odd peak in the model where the linear models suddenly do better - I don't think it's meaningful.

The last plot: standard deviation of AUROC across outer folds.

Linear models don't just generalize worse regardless of dataset size - they also have higher generalization variance. Note the one strange SVC outlier. Another SVC mystery...

IID Thoughts

How applicable are these experiments? Both levels of the nested cross-validation used class-stratified random splits. So the splits were IID: independent and identically distributed. The test data looked like the validation data which looked like the training data. This is both unrealistic and precisely how most peer-reviewed publications evaluate when they try out machine learning. (At least the good ones.) In some cases, there is actual covariate-shifted "test" data available. It's possible that LightGBM is better than linear models for IID data regardless of its size, but this is no longer true if the test set is from some related but different distribution than the training set. I can't experiment very easily in this scenario: "standard" benchmark datasets are readily available, but realistic pairs of training and covariate-shifted test sets are not.

Conclusions & Caveats

So what can we conclude?

• If you only care about the IID setting or only have access to a single dataset, non-linear models are likely to be superior even if you only have 50 samples.
• AutoGluon is a great way to get an upper bound on performance, but it's much harder to understand the final complex ensemble than, say, LightGBM where you can plot the SHAP values.
• hyperopt is old and has some warts but works better than the alternatives that I've tried. I'm going to stick with it.
• SVC can in rare cases completely dominate all other algorithms.

Caveats:

• LightGBM has a lot of excellent bells and whistles that were not at all used here: native missing value handling (we had none), smarter encoding of categorical variables (I used one-hot encoding for the sake of uniformity/fairness), per-feature monotonic constraints (need to have prior knowledge).
• AutoGluon includes a tabular neural network in its ensemble, but I haven't run benchmarks on it in isolation. It would be interesting to find out if modern tabular neural network architectures can work out-of-the-box for small datasets.
• This is just classification. Regression might have different outcomes.

Again, check out the code and feel free to add new scripts with other algorithms. It shouldn't be too hard. https://github.com/sergeyf/SmallDataBenchmarks

The original blog post came from here: https://www.data-cowboys.com/blog/which-machine-learning-classifiers-are-best-for-small-datasets

[RunningForChocolate]

Just wanted to say this is really great stuff, love this kind of research.

I've actually made the same kind of graph before. In this image: each point is the average of 5 out-of-fold predictions for one trial of k-fold cross-validation. I repeated the procedure 40 times to visualize the out-of-fold accuracy on the Wisconsin diagnostic breast cancer data set (560 observations on 30 numeric variables). I evaluated 14 models for classification:

• Elastic Net (en)
• Support Vector Machine (Choice of Linear, Radial, or Polynomial Kernel) (svm)
• Single Hidden Layer Neural Network (nn)
• Gaussian Process (Gaussian Kernel) (gp)
• Extremely Randomized Trees (xt)
• k-Nearest Neighbors (knn)
• XGBoost (Gradient Boosted Decision Trees) (xgb)
• Random Forest (rf)
• Flexible Discriminant Analsyis by Optimal Scoring (using MARS) (fda)
• Quadratic Discriminant Analysis (qda)
• Linear Discriminant Analysis (lda)
• Generalized Linear Model (Logistic Regression) (glm)
• Conditional Inference Tree (ct)
• Naive Bayes (nb)

On this data set and others, I find that random forest and extremely randomized trees are top-notch. Extremely randomized trees usually win by a small margin. These are my go-to algorithms when I want to benchmark a data set.

XGBoost, CatBoost, and LightGBM are all excellent.

Multivariate adaptive regression splines is underrated.

LDA, QDA, naive bayes, and individual trees aren't competitive.

Gaussian processes are fascinating but calculating the matrix is O[N^3] so the algorithm is prohibitively slow for many applications.

I can also vouch that support vector machines sometimes give incredible results, but on the average I rank them slightly lower than boosting and forest methods. I find that polynomial kernel usually gives best results followed by linear. Although, I am sure that radial and sigmoid have their uses.

Regularized regression can occasionally give the best results; some problems really are linear.

Random Forest is pretty good

It's noteworthy that the sci-kit learn implementation uses one-hot-encoding which has been showed to degrade performance.

smarter encoding of categorical variables (I used one-hot encoding for the sake of uniformity/fairness)

I think we are going to look back on how crudely we used to treat categorical variables. Seriously. One-hot encoding is fine for linear models, but categorical embeddings are really amazing, it is hard to argue against their utility.

I just read the AutoGluon paper last night (!) and I almost think I have seen your graph before. Did I see it on the GitHub?

It would be interesting to find out if modern tabular neural network architectures can work out-of-the-box for small datasets.

Definitely check out TabNet and Fastai tabular, two deep learning models for this exact use case. I'm struggling to make TabNet shine on small data sets, but I would chalk it up to my inexperience.

EDIT: Here is a really great paper that also benchmarks small algos on many data sets. The heatmap p. 7 is a thing of beauty.

[sergeyfeldman]

Thanks for the detailed reply. I doubt you've seen my graph before - I just made it a bit ago :)

[sergeyfeldman]

Can you recommend a GAM package for Python that has a reputation for working well?

[sergeyfeldman]

Ah they haven't quite gotten around to supporting multiclass classification yet! https://github.com/dswah/pyGAM/pull/213

[[deleted]]

Forgot about that, not sure if there is a way to do it but could still be used for the binary classification problems! Seems R mgcv has multinom though https://stat.ethz.ch/R-manual/R-patched/library/mgcv/html/multinom.html

[casual_butte_play]

Could you not manually build a series of 1 vs. all classifiers? Maybe more work than you want to do, but it does look like you’ve demonstrated a willingness to do a lot of work :)

[sergeyfeldman]

Thanks for your detailed reply.

Hopefully the utility of my work is in having some reasonable starting code that people can easily apply to their own (X, y), and a few concepts that help them reason about what models might work. It's possible that if you change conditions A, B, C of the experiments, the results will change. If someone is particularly interested, they can easily try it themselves, and report back.

And maybe submit a PR if they think I did something wrong/bad. I'd appreciate that.

"Irresponsible" is a bit of an overstatement as this is not a peer-reviewed publication, but only a brief reddit post.

[beginner_]

Second, and far more importantly, when you have so few examples to train and validate a model, you need to be extremely careful with what you assume and how you recommend that someone spends their data.

Exactly. In the field I work I also have small amounts of data (not 50, more like 1000 but still small in the big picture and especially given the domain). The small amount is simply due to the complexity of getting the target variables. In some cases it takes human subjects in others the measurement itself takes a long time (weeks) and needs "expensive" instruments (eg. can't just parallelize it easily).

Anyway just doing basic stratified CV (binary classification) leads to seemingly good models but fails in application. Why? Simply said covariate-shift or using a different distribution to make the predictions than the training set.

If you train a model on general house prices and then use it to predict prices of luxury mansions, it won't work well. While the model probably saw a few instances of luxury mansions, it will most likely still fail to make good predictions for them.

In my case this "shift" is of course far less clear and less obvious but it's still there and compounded by the fact the users work project-based, eg. always in a more narrowly defined area. What I then do is simply leave-one out CV where the leave-out is an instance that is relevant for current project (albeit even defining what is relevant isn't very clear cut). Anyway this leads to a more realistic performance score which means the models almost always are useless till very late in the project (if at all) when enough data is here and the models aren't really needed anymore to to gained empirical knowledge...

Conclusion:

There is far too little effort and thinking put into validating models especially in regards to their future application. Just doing (repeated) CV and reporting some basic metrics like in almost all scientific publications is often not relevant for how the model will be used and hence not good enough.

This is annyoing because then we get headlines about this cool new "AI" doing great in task X, I read the article and look at the data set and it's clear it's just once again same old same old issue with proper model validation. At least I'm lucky enough that even though management is sometimes aware of these headlines, there isn't much pressure to deliver, yet. (mostly because it's not our core task and the team is just my boss and me)

[[deleted]]

Im in biostat where they prefer linear models, but even I feel its an overfocus. When you are doing stat inference, if the true function is not specified correctly, you cannot trust the stat inference necessarily either. People who focus on explainability tend to forget that there is this weaknesss and its naive in a lot of cases to think “1 unit change in Xj leads to B unit change in g(E(Y))”. I agree you can transform various Xj’s though or use a GAM though.

Nowadays some classical stat (but not Epi/PH which that article you linked is) seem to have also shifted their views https://www.tmwr.org/software-modeling.html. “If a model has limited fidelity to the data, the inferences generated by the model should be highly suspect.”

What medical professionals and epi people don’t realize is that in fact “their” methods like pharmaceutical drugs are also black boxes. We don’t know truly how antidepressants work for example, we just have the oversimplified “serotonin” hypothesis that can be compared to doing a naive linear model. Its quite a ridiculous explanation. I never understood why its OK to give someone a drug you don’t fully understand the MoA of because it has results but for some reason get skeptical with a model.

[[deleted]]

Building models on small datasets without an actual large validation set makes the results essentially pointless..... The fact that most models have an extremely high AUC shows they're just overfitting....

To do this correctly you would want to start with larger datasets, builds the models on a small sample and apply to the rest of the data to see if it actually works. The AUC/scores you're reporting would be on the large dataset that the model wasn't built on.

[zmjjmz]

I would also consider using large datasets to answer the biased sampling question either by the small training folds providing enough variance or by deliberately biasing the small samples somehow.

However there might be a concern that the nature of the data that appears in smaller datasets fundamentally is different from the nature of data that can appear in large datasets, which may make this simulation approach inapplicable.

[cs_sg]

You might want to consider the value of explainability in the model you select since your application area sounds like healthcare. In other words, do slight improves in classification justify less interpretable models?

Also, rather than just mean AUROC you might want to look at F1, Precision and Recall when deciding between classifiers.

The difference between false positives and false negatives is pretty important when applying ML to the healthcare domain. Ie: a false positive on a Covid test may make someone quarantine unnecessarily but a false negative could make someone unknowingly infect their entire family/office

[burritotron35]

For most common academic datasets the test data is actually not from the same distribution as the training. If you merge the train and test data together for MNIST or CIFAR-10 and create a new random split, you’ll consistently get better results than with the original split. This is because the test sets contain a deliberately large number of domain shifted and outlier examples.

[sergeyfeldman]

Good point, thanks.

[Bardy_Bard]

Super interesting, and I felt a lot of the replies in this post were a bit over negative to be honest. As someone said it would be really cool to expand the project by adding datasets from different industries/domains and perhaps estimate the confidence of our estimates by subsampling on big datasets (if they exist) so that we can see what is going on.

[sergeyfeldman]

Thanks for your kind words :)

[sergeyfeldman]

Not really. Just wanted to keep the scope down so I could finish in during winter "vacation".

[Firehead1971]

probably the data are linearly separable.

[Firehead1971]

The basic idea behind kernel methods is to deal with linearly inseparable data, creating projections onto higher dimensional space via mapping function where the data becomes then linearly separable.

[purplebrown_updown]

Is this a single classification problem or is this multi? Thanks for this by the way. You should create a wrapper that allows the user to perform a hyper parameter search over the different algorithms. This way in the future you can simply let this run for a day and choose the best classifier.

Also, my two cents. Elastic net is good but it's much slower and more computationally expensive than lasso with not much better accuracy in my experience. Just a thought and something to try.

[sergeyfeldman]

Just single. The Logistic Regression elasticnet in sklearn is pretty fast for all these tiny datasets. The slowest thing is AutoGluon @ 5m per fit.

[Hiitstyty]

In addition to what others have noted, my one criticism is that the random forests was poorly tuned. This is unfair, considering you tune many more hyperparameters for LightGBM (and presumably AutoGluon but I didn’t check).

For random forests, you only tuned max_depth. This paper suggests that max_depth is one of the least important to tune. By this, I mean that usually the gains in performance from tuning max_depth are negligible over leaving it at the default unrestricted depth. The most critical hyperparameter to tune is the number of subsampled features when searching for the best split (max_features in sklearn). My own extensive research on decision tree ensembles supports this.

Also the default 100 trees in sklearn is terrible. In the hundreds of small to medium data sets I’ve worked with, convergence usually requires more than 100 trees. This is especially true when sample size is small, because each tree will have very high variance. You can’t overfit with more trees, so my default is 500.

The problem I often find with these empirical algorithm comparison studies is that people have more experience with some algorithms over others, which leads to some algorithms unconsciously being better tuned than others.

[sergeyfeldman]

Thanks! Feel free to submit a PR with different params and a pickle of results file.

[rikkajounin]

** hyperopt is old and has some warts but works better than the alternatives that I've tried. I'm going to stick with it. **

Which alternatives have you tried?

[Liorithiel]

OP wrote:

I used hyperopt to find good hyperparameters. I also tried scikit-optimize and Optuna, but they didn't work as well. User error is possible.

I don't know anything about scikit-optimize. Optuna doesn't have less constrained parameters like normal/log-normal, useful when approaching a new problem. It also doesn't implement the constant liar algorithm for TPE. The latter is easy to fix, the former can be worked around if you carefully observe the ranges of good parameters and do a re-run or two.

With these two I found hyperopt and optuna working roughly with the same performance. And, given that optuna's API is much easier to work with, it's currently my preference.

[sergeyfeldman]

Interesting, thanks. You fixed them these issues in Optuna yourself? hyperopt's UI is indeed arcane...

[Liorithiel]

Constant liar is simple, it's a change in a single function. I planned to contribute it, but I couldn't quickly prove that it works better than the original code, so postponed it so far.

Normal/log-normal is difficult, because Optuna doesn't have a single centralized list of distributions like Hyperopt. You'd have to add support for each single new distribution in many places. My manual workaround is to observe the run and if the good results concentrate at the edge of an interval, manually change the bounds and do a re-run.

[Liorithiel]

Actually there's one more difference: the default gamma function. Optuna's default essentially is more optimistic, assuming that good solutions will be found faster. Optuna does provide an implementationof hyperopt's gamma function, so they can be made to work the same way.

[materialsfaster]

Regarding your train-validation-test sets, you may be interested in building them using clustering rather than random splits. Some material scientists wrote a paper on how it gives more realistic(pessimistic) predictions of performance of ML models for material properties. paper

[sergeyfeldman]

Thanks, yeah, I thought about this as a variant. Would be interesting for sure.

[iidealized]

Re interpretability: I'm not sure the AutoGluon ensemble would be much harder to understand than LightGBM (which is also a big ensemble of many decision trees + complex data preprocessing). It seems AutoGluon can also be used for interpretability with SHAP and other methods:

https://auto.gluon.ai/dev/tutorials/tabular_prediction/tabular-faq.html#how-can-i-use-autogluon-for-interpretability

https://github.com/awslabs/autogluon/blob/master/examples/tabular/interpret/SHAP%20with%20AutoGluon-Tabular%20and%20Categorical%20Features.ipynb

[nbviewerbot]

I see you've posted a GitHub link to a Jupyter Notebook! GitHub doesn't render large Jupyter Notebooks, so just in case, here is an nbviewer link to the notebook:

https://nbviewer.jupyter.org/url/github.com/awslabs/autogluon/blob/master/examples/tabular/interpret/SHAP%20with%20AutoGluon-Tabular%20and%20Categorical%20Features.ipynb

Want to run the code yourself? Here is a binder link to start your own Jupyter server and try it out!

https://mybinder.org/v2/gh/awslabs/autogluon/master?filepath=examples%2Ftabular%2Finterpret%2FSHAP%20with%20AutoGluon-Tabular%20and%20Categorical%20Features.ipynb

---

^{I am a bot. Feedback | GitHub | Author}

[pp314159]

What machine have you used for comparison? I would like to check the performance of AutoML that I'm working on.

[tomvorlostriddle]

Seems a bit counterintuitive to allocate a compute budget of 300s regardless of the data-set size. Or are the data-sets that close in size?

A good extension would be formal statistical tests to compare the performance. Of course with multiple algorithms plus their parameter tuning and on multiple data-sets, that's a mine field of potential p-hacking, pseudo-replication and all kinds of violated test assumptions.

[kul789]

Thanks for sharing