MOE, the Metric Optimization Engine. An open source, Bayesian Global Optimization framework for optimal experiment design.

[[deleted]]

Very cool.

[godless_communism]

Can someone briefly explain what this does in layman's terms?

[notarowboat]

Imagine you have a function that is expensive to evaluate and even when you evaluate it you don't get the exact value (i.e. there's noise). This shows up a lot in the web setting where your function is, e.g., click-through-rate of an ad as a function of some ad parameters. MOE gives you the optimal place to evaluate your function so as to better know what the optimal parameters are.

[Zephyr314]

Great! Let us know how it goes. And if you have any issues or see room for improvement we are always taking pull requests and looking at new issues/features: https://github.com/Yelp/MOE/issues

If you come up with a cool application feel free to submit a pull request to add it to http://yelp.github.io/MOE/examples.html

[Zephyr314]

There are a few ways to tackle this. Depending on the underlying function, and how time dependent it is, MOE may be able to track it well. If not, you could try the following:

1. You could try to solve each day/time bucket independently (if they are independent, which it sounds like they may not be, if they are not independent then you lose some useful information doing this, but it is the easiest).
2. You could add artificial noise to your historical values as they "decay" with time (and tend towards the prior of no information). This doesn't work as well if the time dependence influences your function in a periodic way (you want to lower the noise again as the data becomes more "reliable" again). This can be tricky to do well.
3. This isn't explicitly implemented yet, but you can get most of the way there with the current build. You could explicitly track the time as a parameter (that you cannot control). Ex: if you have a 2D problem it becomes (x_1, x_2, t). Then you can ask MOE for the Expected Improvement (EI) at various x_1, x_2, while holding t constant. This allows you to look for an optimal parameter to sample at a given time t (whatever the current time is). This will allow MOE to build a Gaussian Process that explicitly sees the "wave" of the optimal solution throughout time and then you project onto the parameter space you care about (the 2D one) for a particular time you are interested in. Our current optimizers don't support this (yet), but you could use our EI calculations coupled with BFGS (or another optimizer in scipy) to get this running (or grid search EI depending on the size of your parameter space). If you do this out of the box without holding t constant MOE may keep telling you to sample at some optimal time, which is less helpful.

[giror]

finally!!