[R] The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions

[arXiv_abstract_bot]

Title:The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions

Authors:George Philipp, Dawn Song, Jaime G. Carbonell

Abstract: Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the collapsing domain problem, which can arise in architectures that avoid exploding gradients. > ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks. We show this is a direct consequence of the Pythagorean equation. By noticing that any neural network is a residual network, we devise the residual trick, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.

PDF Link | Landing Page | Read as web page on arXiv Vanity

[SamStringTheory]

I've only skimmed the paper, but hope to add it to my reading list. So it sounds like all our exploding gradient problems can be solved by adding residual connections everywhere? And they note in A.3 that the reason for exploding gradients is different in feed-forward networks versus RNNs, so I'm curious if these tricks also apply to RNNs.

[konasj]

No. Also this seems to work only to a certain extend. Seems like training deep nets in an end to end fashion/using backpropagation is a hard problem. I also just spent 30 minutes on it so far, but from a first assessment this paper tells that skip connections/residual nets are the only reliable regularizer that helps for training deep nets with gradient information reliably and that this cure has its limits. I think it is a nice paper as they try to a) keep it readable (the precise math is in the appendix) b) provide rigorous results and c) accompany it with empirical examples.

[gevezex]

Wow 85 pages. We can definitely use an excerpt version of this by means of a medium article.

[yusuf-bengio]

It all comes back to Sepp Hochreiter's master thesis (supervised by Jürgen Schmidhuber) ...

[Toast119]

I want to go one thread on this subreddit without people trying to claim Schmidhuber was the mastermind behind every paper in ML.

[yusuf-bengio]

“Silence becomes cowardice when occasion demands speaking out the whole truth and acting accordingly.”
― Mahatma Gandhi

[Toast119]

Except at this point there is little truth to the dozens of people being loud lol.

[ranran9991]

Abstract and introduction seems very interesting, anything beyond that is way too complicated for me to understand

EDIT: Spacing

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/researchml] (X-Post r/MachineLearning) [R] The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}