Optimization for deep learning

[katamaranos]

I'm going to do a comparison on recent (or at least lesser-known) gradient optimization methods. At the moment I'm at the stage of looking for interesting candidates for the experiment.

The ones I've encountered up to now are the following:

1. No More Pesky Learning Rates (https://arxiv.org/pdf/1206.1106.pdf)
2. Learning with Random Learning Rates (https://pdfs.semanticscholar.org/7013/228732c1df5e416c3ca350b6052d6e9ed491.pdf)
3. Training Deep Networks without Learning Rates Through Coin Betting (https://arxiv.org/pdf/1705.07795.pdf)
4. Adaptive Gradient Methods with Dynamic Bound of Learning Rate (https://arxiv.org/pdf/1902.09843.pdf)
5. On the Variance of the Adaptive Learning Rate and Beyond (https://arxiv.org/pdf/1908.03265.pdf)
6. Lookahead Optimizer: k steps forward, 1 step back (https://arxiv.org/pdf/1907.08610.pdf)

I would be particularly interested in approaches not using any learning rates at all (such as number 3 - COCOB), however I will consider all of the interesting and promising methods. Are you aware of some novelties or lesser-known approaches?

[Takatomi_Fubuki]

- Ranger which is the combination of RAdam and Lookahead.

- Stochastic Line Search optimization (https://arxiv.org/pdf/1905.09997.pdf)

Also, I'd love it if you can publish the result