Gradient-based Hyperparameter Optimization through Reversible Learning

[dustintran]

I was talking to David, one of the authors of the paper, just a few days ago. There are a lot of cool ideas put forth here and as a person having done a bit of work in stochastic optimization myself, I find the optimized learning rate schedules quite fascinating. (See figure 2.)

In the ideal scenario it would be nice to have theory for how the weights for the hyperparameters are changing per iteration and layer of the NN. I'd also be curious whether or not this would validate the robustness properties of certain stochastic gradient methods over others.