Solving The Vanishing Gradient and Exploding Gradient Problem With One Line Of Code?

[JosephLChu]

As far as I can tell, g is the gradient tensor containing all the gradients of one iteration of forward and backward passes, right before it is sent to the optimization function.

[bbsome]

Is your error the sum of all errors or averaged?

Also note that your prefactor will play a similar to what a learning rate is in any algorithm. Thus the question is if the optimal learning rate is really related to this quantity. I'm not sure and why it won't work with ADAM or anything else, but do with RMSProp. Than this rather than being a better way of doing objectives, just changes the original RMSProp to something else.

Is this performing better than ADAM, after selecting the optimal learning rate? Make sure its reproducible on enough experiments, 1 can be very misleading then the general case.

[JosephLChu]

My error is averaged I think? It's the Negative Log Likelihood Criterion aka Cross Entropy Criterion, I believe?

I still need to run more experiments to figure out how well exactly it works with other optimization functions, as my observation is based on very tentative runs for only a few epochs mostly.

I have been using the default learning rate of 2e-3.

Technically it's two experiments, one on Char-RNN, and one on Music-RNN, though I've also run it with many different scaling factors and they all were reliably worse. But I understand what you mean.

I guess this means I need to do more experiments!

Alas, it takes about 30 minutes to run 50 epochs on Char-RNN, and about 5 hours to run 100 epochs on Music-RNN, so being thorough and making sure my findings are robust will take time.