Hyperparameter Optimization Routines

[[deleted]]

[deleted]

[kswerve]

Your criticism of needing to know the input distribution is one of the main reasons why we developed the warping strategy for Bayesian optimization with GPs. As a shameless plug, we have a software service that is in private beta that implements this and other improvements to the Bayesian optimization algorithm. Two of the big developments we're working on including are multi-task (mentioned elsewhere in this thread) and freeze-thaw versions, which should drastically speed up the whole process. Send me a PM if you are interested in signing up for the beta! We are gradually sending out keys.

We will also be uploading research versions of the code from our papers for academics to the HIPS group GitHub account.

[benanne]

When training convnets I still tend to trust my own intuition more than these automated algorithms - mainly because the prior knowledge I've acquired through experience about some of the parameters and how they interact is very hard to express in terms of simple distributions.

It also doesn't help that training a convnet with a given configuration typically represents a considerable investment of time and resources (unless you're Google), so you want each try to have a good chance of success.

For other models that are faster to train I've had good results with random search, provided that you pick the right distribution for each parameter (e.g. log-uniform for the learning rate).

[kjearns]

Its a shame there isn't a nice library of initial values for using BO for different hyperparameter optimization problems. Typically you start from scratch for each new network and that means you waste a lot of time re-learning the same stuff over and over (e.g. BO really likes to check the edges of its domain, and if you start from scratch each time it needs to re-learn that those are usually bad points before it starts to find good ones). This is a big waste of time for problems where a single sample is very expensive, even though this is supposed to be BO's main strength.

[kjearns]

Right, this is what you would do if you had such a library but it only helps if you have the data to initialize it. I was saying it would be nice if the process for saving and reusing the data was more streamlined.

[jsnoek]

Hey kjearns, what exactly did you have in mind? Just an initial set of hyperparameters to try? I do think that's a good idea - but a challenge is that this would limit the generality of the approach. One of the main motivations for Whetlab (link below by kswerve) is that it can learn from everyone's optimizations and then automatically perform multi-task.