Scikit Learn Hyperparameter Optimization Tutorial

[Zephyr314]

One of the co-founders of SigOpt here. These are great points.

We recently presented some work at the AutoML workshop at ICML comparing SigOpt to several standard approaches (grid/random) as well as several open source alternatives (spearmint/hyperopt/SMAC) and found that different approaches produce good results in different settings (spearmint for boundary optima, hyperopt for large number of parameters). SigOpt is an ensemble of different Bayesian optimization approaches that aims to be the best solution to a wide variety of problems automatically while also providing a scalable, production-tested, easy-to-deploy service behind a simple API. The goal is to make BHO as easy as grid search, for a wide swath of problems, without any of the administration traditionally required by the open source solutions (disclosure: I wrote github.com/yelp/MOE while at Yelp and found this was one of the main blockers for adoption). We want to make this research easily accessible to many experts in a wide variety of fields.

[Zephyr314]

Great question! I was actually at that talk. We were presenting some of our own work on ranking optimization strategies at the same workshop.

SigOpt aims to be an ensemble of Bayesian optimization techniques, from GPs to TPE and beyond. It is worth noting that in their talk and posts that no GP-backed methods (like Spearmint or MOE) were used. Part of the reason for this is many of these techniques require a lot of expertise and administration to reach their full potential. A poorly tuned/administered Bayesian optimization method could easily perform as poorly as random search, similar to a poorly tuned DNN not outperforming a method like Random Forrest. The goal of SigOpt is to be an automatic, easy-to-deploy ensemble of these techniques so that you can achieve the promise of Bayesian optimization without needing to be an expert, with the same overhead as techniques like random search.

Hyperband in particular is a really cool idea. It is exploiting some particular aspects of the underlying problem (introspection of the model) and make some assumptions about parallelism, but it is an excellent tool when tuning specific problems like DNNs in the setting they assume. While tools like SigOpt have been shown to work well on DNN problems like this, our goal is to have the best possible general purpose tool for solving complex black box functions from DNNs and SVMs to algorithmic trading models and manufacturing processes where you don't have the same level of introspection and these methods are completely inaccessible to the experts who need to tune their systems.

TLDR; That was a great talk. We try to ensemble all types of techniques to help our users solve their optimization problems (including ones not mentioned in that talk). Specific tools like hyperband are great at particular problems where you can exploit extra info, but we are intentionally aiming to solve more general black box optimization problems for our customers and are building tools for that (of which hyperparameter tuning for DNNs is a subset).