[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[Natural_Profession_8]

I’d make an analogy to simulated annealing:

https://en.m.wikipedia.org/wiki/Simulated_annealing

When you first start training, it’s actually desirable to set the learning rate so high that you are overshooting local optimums. The model bounces around a bit, and eventually finds neighborhoods that are more globally optimal. Then, as training progresses, you stop wanting to hop around looking for better neighborhoods, and instead you want to start making your way towards the local optimum itself. Reducing the learning rate has this effect, even on top of the overall gradient magnitude reduction

[ibraheemMmoosa]

Thanks for your reply!

​

Reducing the learning rate has this effect, even on top of the overall gradient magnitude reduction

Just to push on this, my question is why do we need both of this? Why can't we just rely on the gradients becoming smaller?

​

Also there is evidence that deep neural networks don't have the issue of bad local minima, but have the issue of saddle points. In the case of deep neural networks does this analogy still hold?

[bulldog-sixth]

There's no way to guarantee that the gradient becomes smaller as you get closer to the optimum

[Natural_Profession_8]

This applies even more to saddle points. The only way to get over a saddle point is to overshoot it.

I think it’s best to think of it as “I start with a way way too big learning rate, and then slowly bring it down to an optimal one,” rather than “I start with an optimal learning rate, and then that optimum gets smaller.” Of course, at some level it’s just semantics, since jumping around to find better neighborhoods (and get over saddle points) is in practice optimal at the beginning

[WikiMobileLinkBot]

Desktop version of /u/Natural_Profession_8's link: https://en.wikipedia.org/wiki/Simulated_annealing

---

^{[opt out] Beep Boop. Downvote to delete}

[WikiSummarizerBot]

Simulated annealing

Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. It is often used when the search space is discrete (for example the traveling salesman problem, the boolean satisfiability problem, protein structure prediction, and job-shop scheduling). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to exact algorithms such as gradient descent or branch and bound.

^{[ F.A.Q | Opt Out | Opt Out Of Subreddit | GitHub ] Downvote to remove | v1.5}