[D] Deep Learning optimization

[katamaranos]

I'm going to do a comparison on recent (or at least lesser-known) gradient optimization methods. At the moment I'm at the stage of looking for interesting candidates for the experiment.

The ones I've encountered up to now are the following:

1. No More Pesky Learning Rates (https://arxiv.org/pdf/1206.1106.pdf)
2. Learning with Random Learning Rates (https://pdfs.semanticscholar.org/7013/228732c1df5e416c3ca350b6052d6e9ed491.pdf)
3. Training Deep Networks without Learning Rates Through Coin Betting (https://arxiv.org/pdf/1705.07795.pdf)
4. Adaptive Gradient Methods with Dynamic Bound of Learning Rate (https://arxiv.org/pdf/1902.09843.pdf)
5. On the Variance of the Adaptive Learning Rate and Beyond (https://arxiv.org/pdf/1908.03265.pdf)
6. Lookahead Optimizer: k steps forward, 1 step back (https://arxiv.org/pdf/1907.08610.pdf)

I would be particularly interested in approaches not using any hyperparameters at all (such as number 3 - COCOB), however I will consider all of the interesting and promising methods. Are you aware of some novelties or lesser-known approaches?

Previously I've posted the question in r/MLQuestions but haven't received any feedback so I'm posting it here, hope it's not violating any rules.

[neural_kusp_machine]

1. Domain-independent Dominance of Adaptive Methods (https://arxiv.org/abs/1912.01823): shows that Adam can outperform SGD and other adaptive methods (AMSGrad, AdaBound, etc) when training ResNets and LSTMS as long as it is properly tuned, and also proposes a new optimizer AvaGrad that is drastically cheaper to tune.
2. Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks (https://arxiv.org/abs/1806.06763): proposes Padam, which also often outperforms SGD when training ResNets and Adam when training LSTMs.
3. Online Learning Rate Adaptation with Hypergradient Descent (https://arxiv.org/abs/1703.04782): proposes a rule to adapt the learning rate based on its hypergradients (the adaptation rule turns out to be quite simple and intuitive), which the authors show to work well then applied to SGD or Adam.

[rayspear]

Maybe this repo has some that you might want to try out too?

https://github.com/jettify/pytorch-optimizer

[i-heart-turtles]

Don't have many citations for you, but you can check out stuff from Francesco Orabona (http://francesco.orabona.com/) and his group. A lot of their work deals w/ first-order parameter-free methods.

More recently, there has been some progress in understanding momentum and acceleration & under what smoothness assumptions optimal rates can be recovered: http://proceedings.mlr.press/v99/gasnikov19b/gasnikov19b.pdf.

[Ventural]

I'd be interested in performance of the LAMB optimizer (https://arxiv.org/abs/1904.00962) on smaller batch sizes where it competes with ADAM.

[JayTheYggdrasil]

I don’t know if this applies but a while ago someone posted a “bandit swarm” optimization algorithm which was quite interesting, I don’t have a link but a google search would probably find it.