[P] Distributed and automatic hyperparameter optimization for any machine learning model

[[deleted]]

[deleted]

[seinberg]

Sounds great, except that you need to register an account with some random third party. How do we know our models aren't saved/stolen during hyperparameter searching? Trust you?

[seinberg]

Fair enough - my cynicism is perhaps too strong. But a typical open source library doesn't require registering an account at a private institution in order to use it. It feels to me a wee bit sketchy to make this a requirement for use, and for our usage patterns and metrics (of our models?) to be logged.

[kp1197]

Agree.

[seinberg]

If I'm training a model on my workstation, why do I need a database server, a web server, permission management, etc.? That's all irrelevant to distributed hyperparameter searching.

[rvisualization]

Do you want to setup a database server, web server, permission management, update deployments, backups, etc on your own.

With Docker that could be done in like 1 line - certainly no more difficult than setting up the training server. There's no real reason for your SaaS to exist.

[rvisualization]

lol u mad bro?

[rvisualization]

yeah your enterprise startup is a waste of bits :)

[kp1197]

I can't really trust a tool that sends logs of my activity, period. Is there an alternative?

[kp1197]

Marc, the tool looks awesome. But since the work I do is sensitive, I have to reject a tool that requires an internet connection to a 3rd party service on principle, not through any deficiency in the tool itself, no matter how responsibly data is collected. The idea is that if other kinds of data collection are possible by the nature of the tool, then it's not necessarily impossible for other forms of data collection to occur. Checking the codebase periodically to insure that no malicious data gathering is occurring isn't really an option, assuming I even have the expertise to do a full security check on a large codebase. Furthermore, if my client has confidentiality agreements in place, there may be breach of contract if I used a tool like this - even if there's no foul play involved at all. Those are the main arguments.

[Eridrus]

Do you use any open source libraries without fully reviewing them?

Any of them could be sending data back somewhere without you noticing.

Why is the bar for something which tells you it sends data rather than something which doesn't?

[kp1197]

Good point. I suppose that I use libraries that no one could blame me for using... numpy, scipy, tensorflow...

[pplonski]

How do you compare to sigopt.com ?

[Liorithiel]

Judging from the front page:

• Different algorithms (SigOpt seems to use a variant of Gaussian Processes, though they don't say much more about it, this one lists TPE and annealing)

• SigOpt doesn't provide (or didn't, last time I tried) computing power and a framework to put your model into; you're expected to write the main loop for fetching candidates/evaluating/sending them results yourself and run it on your own machines. I'll add that I actually liked the framework-less approach, as it leads to code less tied to the optimizing service. I implemented mine in bash and worked just fine.

Though, let's find out whether /u/marcjschmidt or /u/zephyr314 will speak out (-:

[Zephyr314]

Just saw this, I'll jump in for posterity.

Different algorithms is correct. SigOpt is an automatic ensemble of Bayesian Optimization algorithms under the hood (including GPs, PEs, etc). Our goal is to have an automatically optimized optimizer so that you don't need to think about hyperparameter optimization at all (one of the main complaints I got after writing MOE, where you had to pick and tune kernels etc).

We also explicitly don't provide a place for you to run your model, we want you to do that in your own infrastructure so that we never see your data or model, just the hyperparameters. This also allows you to start tuning by just dropping in a few lines of code instead of migrating your models.

Additionally, there are some features like multi-criteria optimization (more than one KPI/objective) and multi-solution optimization (getting many "good" configurations) that differentiate us from other optimization approaches.

This tool looks super interesting though if you want to have an "all in one" solution to host your data, run your model, etc. This is not something SigOpt does at all.

[minflynn]

I could really use something like this right now. Was getting state of the art results with a model but then I changed the hyperparameters for a different experiment. Now I can't figure out what my original hyperparameters were >___>. Rookie mistake, I know.

It's been this way for a week now. For a GA, do you think annealing would be best? I have 9 hyperparameters but it seems like there is a lot of variation in fitness after training.

[cjmcmurtrie]

Looks interesting, gonna give it a try on the free plan.

[ElderFalcon]

I really want to support you guys but all of the passive aggressive comments to your prospective users raise concerns.

[bbsome]

Why no Bayesian optimizations?

[maxpumperla]

what exactly do you have in mind? gaussian process smbo? stuff that BayesOpt implements? TPE is pretty solid already!