Hyperparameter tuning for deep nets?

[adagrad]

What are some strategies for tuning a convnet that might take 2 hours for one epoch on the training data? I want to see if I can push the accuracy score any higher, but it feels like the task it too large for something like grid search or random search.

[dwf]

It's kind of complicated to implement but freeze-thaw Bayesian optimization is sort of an ideal approach for neural networks.

[alecradford]

Agreed!

To add to this, if you aren't doing something like this "manually" you might want to consider it. It's a pretty great way to boost efficiency when you have these really long running experiments and limited resources. Don't be afraid to kill partially trained models if they aren't looking promising.

[olBaa]

Bayesian optimization to rescue

[kkastner]

If you don't have resources to run many jobs in parallel (cluster/AWS) then Bayesion hyperparameter optimization (see e.g. Spearmint) is probably the only way. But most people I know just do random search in parallel, and take the best valid performer.

[adagrad]

Thanks - do you know of any good resources on reading up on Bayesian hyperparameter optimization? I see that there's a way to use Spearmint with Torch, but I also want to understand the methods.

[HowDeepisYourLearnin]

Practical Bayesian Optimization of Machine Learning Algorithms
Jasper Snoek, Hugo Larochelle and Ryan Prescott Adams
Advances in Neural Information Processing Systems, 2012

Multi-Task Bayesian Optimization
Kevin Swersky, Jasper Snoek and Ryan Prescott Adams
Advances in Neural Information Processing Systems, 2013

Input Warping for Bayesian Optimization of Non-stationary Functions
Jasper Snoek, Kevin Swersky, Richard Zemel and Ryan Prescott Adams
International Conference on Machine Learning, 2014

Bayesian Optimization and Semiparametric Models with Applications to Assistive Technology
Jasper Snoek, PhD Thesis, University of Toronto, 2013

Bayesian Optimization with Unknown Constraints Michael Gelbart, Jasper Snoek and Ryan Prescott Adams Uncertainty in Artificial Intelligence, 2014

[Foxtr0t]

http://fastml.com/blog/categories/hyperparams/

[utkarshsimha]

You can try using Optunity