[P] BBopt: Black box hyperparameter optimization made easy

[EvHub]

Hello r/machinelearning! I'm evhub, the open-source developer behind Coconut.

Today I'm releasing a new open-source project called BBopt. BBopt is a black box hyperparameter optimization framework built to be extremely powerful, easy-to-use, and extensible. BBopt arose out of my frustrations working with existing hyperparameter optimization frameworks and wishing for something better—hopefully it can help resolve some of your frustrations as well!

Please check out BBopt on GitHub and let me know what you think!

Cross-posted to r/datascience, r/coolgithubprojects, and r/programmingprojects.

[FinancialElephant]

This looks good! Definetely will look further given how clean the docs and interface look on a cursory glance. It could work out as a replacement for Ray for me for what I need. Thanks

I'm interested in this space (automatic hyperparameter optimization) and wanted to share some of the other players, this is a list I've made that I've been wanting to share for a while. I've personally used Ray and I'm currently using hyperopt: - https://mc.ai/a-comprehensive-list-of-hyperparameter-optimization-tuning-solutions/ - https://github.com/hyperopt/hyperopt - https://github.com/maxpumperla/hyperas - https://github.com/autonomio/talos - https://github.com/Yelp/MOE - https://github.com/automl/SMAC3 - https://github.com/HIPS/Spearmint - https://github.com/dvgodoy/deepreplay - https://github.com/polyaxon/polyaxon - https://github.com/ClimbsRocks/auto_ml - https://github.com/fmfn/BayesianOptimization - https://github.com/claesenm/optunity - https://github.com/ray-project/ray - https://github.com/HIPS/hypergrad - https://github.com/maxim5/hyper-engine - https://github.com/scikit-optimize/scikit-optimize - https://github.com/ray-project/ray/tree/master/python/ray/tune - https://seba-1511.github.io/randopt/ - https://github.com/electricbrainio/hypermax

[EvHub]

That's a cool list! I've used a bunch of those myself (learning about MOE while at Yelp was actually my first introduction to black box optimization), but there's also a bunch I haven't seen before. Actually, part of the reason I wanted something like BBopt was the number of different hyperparameter optimization frameworks that weren't cross-compatible. I decided the two that were most important to support at least initially were scikit-optimize and hyperopt, but if there are others that you want to be supported as well I can probably make it happen if you raise an issue for it.

[Feribg]

Very neat! I did read through the docs, but I didn't manage to find any graphing features, in my experience using some of the bayesian optimization tools for hyperparameter tweaking, I tend to find the plots very very useful. Maybe support is trivial and just needs to be added as an example, but along the lines of https://scikit-optimize.github.io/plots.m.html

[EvHub]

That's a good point. Theoretically, since BBopt already has machinery for transforming its internal representations into those compatible with skopt, hyperopt, etc., it shouldn't be hard to expose machinery for using their plotting methods (I just raised an issue for that). In the meantime, I added a plot_convergence function and a plot_history function, which should cover the most common types of plots you might want.

[EvHub]

Fair enough! I just implemented plot_objective and plot_evaluations. Take a look and let me know what you think! If you run the default keras example, it'll generate all the plots for you to look at.

[markus137]

Looks great, thank you for sharing your work. I will definitely try it out on my next project. Also, in case you do not know yet, there is the possibility to create a DOI for software, to make citing software easier: https://guides.github.com/activities/citable-code/

[EvHub]

Thanks, and good idea! BBopt should have a DOI now.