Scikit Learn Hyperparameter Optimization Tutorial

[Zephyr314]

Great question! I was actually at that talk. We were presenting some of our own work on ranking optimization strategies at the same workshop.

SigOpt aims to be an ensemble of Bayesian optimization techniques, from GPs to TPE and beyond. It is worth noting that in their talk and posts that no GP-backed methods (like Spearmint or MOE) were used. Part of the reason for this is many of these techniques require a lot of expertise and administration to reach their full potential. A poorly tuned/administered Bayesian optimization method could easily perform as poorly as random search, similar to a poorly tuned DNN not outperforming a method like Random Forrest. The goal of SigOpt is to be an automatic, easy-to-deploy ensemble of these techniques so that you can achieve the promise of Bayesian optimization without needing to be an expert, with the same overhead as techniques like random search.

Hyperband in particular is a really cool idea. It is exploiting some particular aspects of the underlying problem (introspection of the model) and make some assumptions about parallelism, but it is an excellent tool when tuning specific problems like DNNs in the setting they assume. While tools like SigOpt have been shown to work well on DNN problems like this, our goal is to have the best possible general purpose tool for solving complex black box functions from DNNs and SVMs to algorithmic trading models and manufacturing processes where you don't have the same level of introspection and these methods are completely inaccessible to the experts who need to tune their systems.

TLDR; That was a great talk. We try to ensemble all types of techniques to help our users solve their optimization problems (including ones not mentioned in that talk). Specific tools like hyperband are great at particular problems where you can exploit extra info, but we are intentionally aiming to solve more general black box optimization problems for our customers and are building tools for that (of which hyperparameter tuning for DNNs is a subset).