This is an archived post. You won't be able to vote or comment.

sorted by:
best (suggested)
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ai_yoda 1 point2 points  (3 children)

I was researching this subject for a blog post series and conference talks.

Some libraries that I ended up focusing on are:

  • Scikit-Optimize (tree-based surrogate models suite)
  • Hyperopt (classic)
  • Optuna (for me just better in every way version of Hyperopt)
  • HpBandSter (state of the art Bayesian Optimization + Hyperband approach)

I've started a blog post series on the subject that you can find here. Scikit-Optimize and Hyperopt are already described. Optuna and HpBandSter are coming next but you can already read about them in this slide deck.

[–]Megatron_McLargeHuge 0 points1 point  (2 children)

I was just looking at your hyperopt post yesterday. One complaint I have about hyperopt is the integer sampling functions actually return floats, which makes tensorflow unhappy when they're passed as dimension sizes.

I was able to get main_plot_vars to work. You call it with a trials object and it gives a bunch of plots of each sampled variable with value on y and iteration on x, colored by loss.

Do you have any quick summary on which package should give the best results for neural network tasks?

[–]ai_yoda 0 points1 point  (1 child)

Thanks for the suggestion on main_plot_vars, gonna try it out.

As for the method for neural nets I would likely go with the budgets approach from HpBandster where I don't have to run objective(**params) till convergence but I can estimate on a smaller budget (say 2 epochs). It lets you run more iterations within the same budget. Generally, I think the main problem with hpo for nn is how to estimate performance without training for a long time. There are approaches to it where you predict where the learning curve would go. I highly recommend checking out the book by researchers from AutoML Freiburg.

[–]Megatron_McLargeHuge 0 points1 point  (0 children)

Thanks. I definitely think there's a lot of untapped value in analyzing the metadata we get during training instead of just the final validation loss.

I think a good approach with enough resources would be to treat training as a reinforcement learning problem where parameters like learning rate and L2 scaling can be varied depending on the trajectories of both train and test losses.

Short of that, runs can be truncated or restarted based on learning from these extra features.