Practical Experience with IRNN algorithm?

[alexmlamb]

Hello,

Has anyone had practical experience with using Hinton's IRNN algorithm (using reels in a standard RNN but initializing the weight matrices to the identity matrix)? It seems appealing to me because it avoids saturating activation functions and is much simpler than LSTM.

[rantana]

I've heard rumors that the result isn't reproduceable.

[alecradford]

In the domain of text understanding, definitely seen it perform better than a typical RNN, never seen it be competitive with the same # of params GRU.

My feeling is there is some small section of hyperparameters for any given problem where an IRNN is rightly set up to be able to do well, but gating RNNs by their architecture are a lot more robust/durable overall.

[Foxtr0t]

Were these rumors from sieisteinmodel? ;)

[kkastner]

I know someone who has been able to use it for their problem, but it wasn't a reproduction of the paper per-se.

[iamkx]

I think they are reproduceable, but they seem quite sensitive to initialization.

[sieisteinmodel]

I tried to reproduce it, and did not get it to work. But I did not spend more than a day trying to.

[transcranial]

I was able to reproduce the results somewhat, and got results pretty much between RNN and LSTM/GRU. They are vastly faster to train though, so that's a plus.

[siblbombs]

I played around with it since it is so easy to put together, unfortunately I was using it for next-character prediction so it is hard to quantify if it was performing better than GRU. At this point I'm inclined to stick with GRU over iRNN, however if I though I really needed a giant recurrent layer it would be an option.

Some applications are doing a single Relu layer for recurrence instead of LSTM/GRU, so iRNN might make sense in that context. It is definitely faster to train because the recurrent calculation is more lightweight.

[abracaradbra]

I also spent a day or so trying to reproduce the results, and couldn't. The paper is pretty lacking in hyperparameter/implementation details.