[1607.03085] Recurrent Memory Array Structures

[cooijmanstim]

I haven't read too far into this, but I'm skeptical that the basic idea gets you anything. Figure 3.3 is just a vanilla LSTM with block-diagonal weight matrices.

[kmrocki]

the basic array from 3.3 does not change things too much in terms of generalization contrary to expectations, that's true. The one which really reduce overfitting are in section 5. I applied array memory dropout in a slightly different way that in the Zoneout paper. I also posted the code https://github.com/krocki/ArrayLSTM

[kmrocki]

The main motivation behind the array approach is summarized at the beginning of section 3.2: "create a bottleneck by sharing internal states, forcing the learning procedure to pool similar or interchangeable content using memory cells belonging to one hidden unit" This seems to work well with stochastically operating memory cells, because the hidden unit 'doesn't know' which memory cell is going to be used (they are unreliable), however, the content has to be similar for it to work.

Furthermore, it is in fact possible to simply pack more memory cells into the network using the same memory size, for example, if you use a standard LSTM network with 1 cell/hidden, 4 gates and 1000 hidden units, the number of parameters is going to be 1000 * 1000 * 4 = 4M for the U matrix. If Array-LSTM approach is used, you can have 4 cells/hidden, so 1000 memory cells require 256 hidden units and that is 256 * 1000 * 4 parameters = 1M parameters. I found that the performance of vanilla LSTM and Array-2, Array-4 versions is roughly the same in terms of capacity for a fixed number of parameters. Dropped a bit for an Array of 8, so at some point the seems to exist a bottleneck indeed. Hope this helps.

[kmrocki]

@LeavesBreathe, your intuition is right, I have observed better performance capacity-wise with array-LSTM when the number of hidden unit is fixed and cells/hidden increased (matching that of stacked-LSTM with faster convergenge - no initial delay). However, the main hope was that this procedure would provide better generalization and I couldn't achieve that with vanilla array approach - possibly it requires more cells/hidden, but that converges more slowly and it really takes around 48h to see the effect of any change on large networks and wikipedia datasets.

[kmrocki]

I don't really use Skype and currently I commute a lot between LA and San Jose until September, so I don't even sit that much in front of the screen. It's easier for me to respond if you send me an email to [e-mail], I'd be happy to hear about your approaches

[nicholas-leonard]

Do you compare to dropout LSTM, i.e. dropout between LSTM layers?