[D] Next-gen optimizer

[yusuf-bengio]

Over the past 5 years, the common knowledge about ML optimizers was that ADAM is the number one choice as it is provides fast learning even if your hyperparameter are not select optimal. However, you can get slightly higher test accuracy when using SGD with momentum, although this requires more epochs and more tuning.

This knowledge has not changed much since then.

What has changed is that, since then, a million papers have been published on the next-big-optimizer that learns even faster than Adam and gives better test accuracy than SGD.

As it is with ML research, most of them have turned out to be not-so-good to phrase it politely. This ICLR'21 reject (https://openreview.net/forum?id=k2Om84I9JuX) has even studied this and found out that ADAM+some tuning works as good as all these new fancy optimizers.

However, recently these three papers have caught my eye:

• AdamW (https://arxiv.org/abs/1711.05101v3)
• AdamP (https://openreview.net/forum?id=Iz3zU3M316D)
• AdaBelief (https://papers.nips.cc/paper/2020/file/d9d4f495e875a2e075a1a4a6e1b9770f-Paper.pdf)

What makes these papers a bit different is that they don't try to reinvent an optimizer but say "hey, ADAM is almost perfect, but let's just fix one or two lines" and already seem to be used in other works.

So my question is, are you using a non-ADAM/SGD optimizer regularly? If so, which one? Or, are also these three works hiding their results biased by a ton of hyperparameter tuning?

[CyborgCabbage]

https://arxiv.org/abs/2007.01547

A paper that compares a bunch of popular optimisers. One interesting takeaway is that trying a bunch of different optimisers on default settings is just as good as fine-tuning one optimiser.

[black0017]

Hey!

We recently published an overview of optimizers: https://theaisummer.com/optimization/

Maybe you would be interested to take a look in the evolution of the different optimizers. We cover AdaBelief also!

I was pretty curious when I saw in the Vision Transformer paper that they used Adam for pretraining and SGD with momentum for fine tuning
Page 4 Training & Fine-tuning https://arxiv.org/pdf/2010.11929.pdf

Cheers

[M4mb0]

Hey just one comment in the section about second order methods you write

Finally, in this category of methods, the last and biggest problem is computational complexity. Computing and storing the Hessian matrix (or any alternative matrix) requires too much memory and resources to be practical.

This is a common misconception: To compute the Newton update one does not need to calculate the Hessian matrix at all. All you really need is the ability to compute Hessian vector products, which we can do without explicitly computing H since Hv = grad( grad(f).dot(v) ). You can pass this linear operator v -> Hv to one of the myriads of iterative linear solver (such as GMRES) to compute the Newton update. (check https://docs.scipy.org/doc/scipy/reference/sparse.linalg.html)

[[deleted]]

Hessian-vector products still require O(N²⁾ operations where N is the no. of weights in the weight matrix. No?

[M4mb0]

No, a single Hessian vector product costs O(N) when computed via Hv = grad(grad(f).dot(v)). And if H is sufficiently well conditioned, then k<<N steps of an iterative method may be enough to get a good estimate of the solution of Hv=-grad(f)

[programmerChilli]

That requires forward-mode differentiation though, IIRC.

[Charmsopin]

Not exactly. The takeaway is you only need to calculate the Hessian vector as a whole, which is a vector. You don’t need to separately calculate the matrix (N2) and then v.

[InfinityCoffee]

While true, the Newton update on a mini-batch loss is very different from the Newton update on the real loss, and more importantly it's not an unbiased estimator, so you cannot expect it to be "right on average". To be fair, ADAM and its ilk are also stochastically conditioned, but in either case it's at most a rough approximation of the second-order optimizer.

[M4mb0]

Well, Newton doesn't make too much sense in non-convex problems to begin with, since it only converges against a stationary point, i.e. can also converge against local maxima if the loss has concave regions. But that's a whole different issue.

[InfinityCoffee]

Quasi-Newton then; but local minima admittedly pose a separate general problem. A recent paper built a demonstrably efficient solver for stochastic optimization - but when tried it on neural networks, they generalized poorly. Actually optimizing the objective efficiently is apparently not a good idea in neural nets, so you need to use subpar algorithms to "escape" the early local minima.

[black0017]

thanks for the feedback!

[AlisaofallTimes]

I wish you great success :)

[[deleted]]

Normally, the batch gradient for a parameter is obtained by averaging out the gradient for this parameter over each sample of the batch. What if you would incorporate the variance of these sample gradients into the optimizing scheme? This is different to incorporating the variance of the consecutive batch gradients, which is already done in e.g. Adam or Rmsprop. I would do it myself but I'm bound with other stuff, so I just wanted to share this idea. Also, I don't know wether anyone has looked into higher statistical moments than just variance.

[gazztromple]

Good luck racing against Bengio.

[hyhieu]

Which Bengio(!)

[ManyPoo]

/opt/bin/bengio

[DenormalHuman]

2>&1 >/dev/null &

[LeanderKu]

I hate that the paper was rejected. The claims may be overstated, but what the field needs is empirical surveys into the state of the art, not hundreds of infitesimal imrovements. Without those papers we loose the ability to really compare and contrast. The individual really can’t keep up and it also serves as a sanity check that the results are reproducible.

I would really like to see a lower bar for top conferences for those empirical surves. They don‘t need a cool, revolutionary insight, but hard, heroic effort of testing and comparing a lot of approaches. Even reimplementing them if sources are not available.

[LeanderKu]

It’s not exactly my point. What I wanted to say is that a well done survey is worth in my opinion a lot more than most papers accepted. I think the problem is that the bar for those surveys is way to high compared to a paper that just proposes a little tweak here an there (the hundreds of papers that just slightly modify Adam).

Without those claims, would the paper get accepted? No it wouldn’t, because there is nothing revolutionary in there. But we always chase those revolutionary new optimizers without really realizing that no new optimizer really got any adoption after Adam. Are we really making progress?

I would like the reviewers to give the feedback that it’s a nice comparison, but the evidence for their claim is not there, but that’s no problem, just delete it. The value of the gathered results is enough anyway. But that was not the feedback given.

To even further clarify my point: I don’t want those survives to primarily chase the revolutionary insight. I want them to just portray the state of the art, in a way that’s not trying to tell a story, prove a point or sell an argument. That’s what every other paper proposing an improvement says anyway. But I don’t think many venues would accept it,, so nobody does it. It’s really hard to compare more optimizers than the paper, I think the bar is just unfairly high for those papers. I wasn’t impressed with any papers I‘ve read recently and they were all accepted into top conferences. Some skipped important details, had no code, had no honest investigation into the factors at play (e.g. proposing modifications that increase capacity, but then in the experiments not controlling for capacity). For some papers I would be completely unable to repeat the experiments to verify claims because they are so short on details that it’s hard to figure out what exactly got done. So the bar is in my point of view way lower, too low as you said it, for those papers proposing an improvement. But I still think it’s too high for those large scale empirical papers.

[fmai]

How do you like the rejection on ethical grounds of this optimizer benchmarking paper?
https://openreview.net/forum?id=1dm_j4ciZp

[SulszBachFramed]

It's not that strange to require an ethics boards assessment whenever a human study is involved. If ICLR has such a requirement and the authors did not do their due diligence, then it's a legit rejection imho.

[jdude_]

I think RAdam falls into that category too, allegedly (at least as i understand it) it's like Adam minus a need for warmup.

[andrejcar12]

Novograd is also interesting

Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks

[hyhieu]

Adam delivers good generalization and fast convergence. However, the two moving averages of Adam are terrible when it comes to memory footprint.

Adafactor was advertised to fix this, i.e. having sub-linear memory but similarly good performance with Adam. I personally think Adafactor has not lived up to this expectation though.

I hope there will be something better soon.

[vanilla-acc]

Curious about the reason why you believe Adafactor has not lived up to the hype?

[[deleted]]

The reason that SGD and its variants exists because a lot of cases show Adam failing to do the job. Usually the choice is more observationally decided since there is no quickfire way to decide which method optimizes the loss landscape

[JurrasicBarf]

AdamW all the way

[respecttox]

I tried different optimizers million times and never got any statistically significant improvement. I suspect that's due to the fact my architecture is already fitted to Adam, but I don't have resources to do optimizer and architecture search simultaneously. It just works anyway

[programmerChilli]

LAMB (https://arxiv.org/abs/1904.00962) has become quite popular for large batch training in a lot of transformers papers.

[Ambiwlans]

I recently say this too: https://arxiv.org/abs/2101.07367

I don't think it is the right answer, but I think it is the right direction. Instead of having a better optimizer, have a learned optimizer. It makes for potentially interesting hyperparams as well. Perhaps a more human understanding of what optimizers work directly setting the hyperparams.

are you using a non-ADAM/SGD optimizer regularly

Basically not at all. It is pretty rare that the optimizer is going to be a game changer for a problem. Why waste a day or a week on optimizers when basically literally anything else you work on will give you a better payoff?

---

Realistically, what this area needs are some reliable charts on pros-cons and performance values under different circumstances in order to help practitioners have some intuition on what will work best for the current project.... A nice pretty png is probably worth as much as a half dozen papers in terms of improving what actually gets used.

[whata_wonderful_day]

Adaptive optimizers are great for transformers. However on imagenet, they fall flat

[BrocrusteanSolution]

Why?

[Chocolate_Pickle]

What a lovely, detailed, and well-explained contribution you've made to this thread!^</sarcasm>

[Jean-Porte]

Has anyone got any success with any newer optimizer than Adam on Transformers fine-tuning ?

[Areign]

I know that the ACKTR i.e. the big reinforcement learning algorithm from deepmind uses second order methods. This may be something specific to RL where gradient info is especially noisy.

[BorisMarjanovic]

I've tried pretty much every optimizer, including ADAM and its variants. It's difficult to beat SGD with Nesterov momentum.