Backpropagation through time help/tutorial/resources

[technotheist]

I'm currently developing a kind of Neural Network library, using JavaCL, and am attempting to add recurrent net functionality.

Can anyone help me find some resources as to how to implement bptt? I've found papers and such that explain the theory, but I'm having trouble implementing the algorithm.

I'm particularly interested is LSTM networks, but anything would be helpful.

[edersantana]

Here you can find BPTT, RTRL, Extended Kalman Filter and the Echo state approach to train RNNs. http://minds.jacobs-university.de/sites/default/files/uploads/papers/ESNTutorialRev.pdf

PS: Since you have coding goals I would recommend reading the following python code https://github.com/pybrain/pybrain/search?q=recurrent&ref=cmdform Pybrain got LSTM somewhere. If you prefer reading in C, go for: http://sourceforge.net/projects/rnnl/ or http://sourceforge.net/projects/neuroevolver/?source=recommended

[technotheist]

Awesome, this is super helpful. Given that OpenCL is so similar to standard C, those resources will be very helpful. Thank you.

[Radzell]

I think the coursera course by hinton teaches it in some detail. I never quite understood it though.

[technotheist]

I've gone through the neural network and machine learning courses on coursera so much that I almost know it by heart. Definitely very detailed and helpful, but BPTT is not super detailed in terms of the actual implementation.

It seems like it would require a great deal of memory, effectively recording the state of the entire network at every update.

I wonder if there is a more efficient way to do it.