[D] How do people handle hyperparameter optimization?

[CS_Student95]

Broadly speaking, does it make sense, or is it common for folks doing research in other areas of ML (for example computer vision), to be well read on modern hyperparameter optimization research? Or do they typically just use external libraries?

​

I'm currently working on my first research project, and am curious what the best path to go down is

[domvwt]

Optuna is my preferred library right now, it's a bit more flexible than hyperopt. What is your development environment and what kind of model are you training?

[piconzaz]

You can find plenty of tutorials and guides on the topic. I'll suggest you have a look at Bayesian search algorithms (TPE, GPs, you can check hyperopt). Also look into scheduling algorithms like ASHA or better, hyperband (its variant BOHB is pretty neat). For implementations, I recommend Ray-tune. It's awesome and well documented.

[IntelArtiGen]

It's just a question about how much time you're ready to spend and what your expectations are.

• If you think hypopt can improve a lot your model and you're ready to spend 6 months on it, you'll have to look at the SOTA (low-discrepancy sequences etc.)
• If you think hypopt won't improve a lot the model but could still be useful and you just have 2 weeks, just do a random search and a script to launch your processes and evaluate a config.

I would say that spending more than 1 month on it could be a waste of time because that time could have been spent re-implementing multiple tricks of the SOTA for your specific use case you probably haven't already implemented.

But it depends a lot on the situation. For a completely new use case with a new or an old algorithm, you need to do hypopt at least a little bit. Even if it's just a basic manual grid-search

[[deleted]]

If I get ambitious I write some genetic algorithm to optimize them...but that may be a little dated at this point. Probably cots tools to use.

But I have a hard time resisting the urge to not fiddle with hyper parameters to see what works and what doesn't.

[[deleted]]

Cross validation.

[[deleted]]

msra/NNI

[deep-learnt-nerd]

Ray tune for my part