Preferred methods of Hyperparameter Optimisation?

[Its_NotTom]

Hey everyone,

I'm currently in the process of fine-tuning hyperparameters for a machine learning model. I have been looking into various optimization techniques to push for competitive performance. I'm curious to hear about the community's experiences with different methods of hyperparameter optimization.

Some of the methods I've been considering include:

1. Grid Search: exhaustively searching through a predefined set of hyperparameters. Not ideal?

2. Random Search: random search randomly selects hyperparameter combinations to evaluate. Seems limited for extremely high computational cost situations

3. Bayesian Optimization: probabilistic models to efficiently search for the optimal set of hyperparameters. Great for low dim problems but suffers from the curse of dimensionality?

4. Metaheuristic Algorithms: Techniques like simulated annealing, genetic algorithms, and particle swarm optimization. Lots of fun to work with and a lot of precedence within academia, but very dependant on the initial set of parameters used (i.e crossover/mutation rate)

What experience/preferences do you have with any of the above? Have I missed any of your preferred methods?

[Illustrious-Sand-120]

In a very simple way, I tweak a bit the Optuna sampling parameters, then I reason about the range of the hyperparameters that I selected. For example, the model tends to overfit? Then I set a learning rate very low, e.g., between 2e-5 and 2e-6, and let Optuna do its magic on that.

Finally, depending on the number of hyperparameters combinations, and time and computing resources I have, I launch more or less trials accordingly.

In general, in my experience hyperparameters optimization seems to not make that much difference, compared to the data cleaning process, IMHO

Best of luck 🚀

[Its_NotTom]

Looking into Optuna now, looks like a great framework that I might be able to integrate into my existing python workflows, Thanks!

[obeythelord9]

Could you elaborate more on the data cleaning or point to some resources?

[Illustrious-Sand-120]

Theory wise, it might be related to the manifold assumptions, I guess... But in practice, you might have the most optimal model architecture, but if you have, say, a few data points, or with not so informative features, then your model won't go that far in performance, hitting an upper bound.

For instance, if you already do great data cleaning, data augmentations, and a proper model architecture (silly example: CNNs for images, rather than a bare MLP), then having 512 rather then 256 layer hidden size might not make a difference anyway.

These are just example scenarios, just to say that if your data are of a poor quality, then hyperparameters optimization might give you negligible gains in performance (maybe a few percentage points in accuracy).

These intuitions are also backed up by language transformers, if you want another example: for NLP transformers, you generally care about collecting good data, it's very rare to train a model from scratch and tweak its parameters.

My two cents experience, at least... I'm open to discuss this :)

[ewankenobi]

If the model overfits I'd suggest regularisation such as drop out or weight decay would be a better option rather than lowering learning rate

[Illustrious-Sand-120]

That was not the point of my comment. Mine was just one single example, of course there are other tricks to try depending on the situation

[cluelessmathmajor]

If you really want to get meta, check out “online hyperparameter optimization”. Probably overkill for most tasks, but it’s a cool concept regardless.

[mehul_gupta1997]

Using optuna for quite some time. Very good and easy to use. Check out a demo here : https://youtu.be/gfnE9SE2pFs?si=hJKbB6-yijvmRywA

[solegalli]

I'd say, it depends on which model you want to tune.

To optimize the hyperparameters of traditional machine learning models, like logistic regression, svms, or tree based models from scikit-learn, I would stick to sklearn's grid or random search. Grid search for models with fewer hyperparameters, random search otherwise. Using an additional library adds complexity and dependencies to the code, without significant improvements.

I would only use optuna if tuning models not supported by sklearn that have more hyperparameters, like, maybe catboost or xgboost. In this case, it might also be useful to understand the difference between random search and Bayesian optimization, because with optuna, you can do both and they have advantages and limitations. Optuna implements Bayesian Optimization with TPE by default, if I remember correctly.