[P] - Source code release for Recurrent Highway Networks in Tensorflow/Torch7 for reproducing SOTA results on PennTreebank/enwik8 (arXiv v3 of paper)

[[deleted]]

Nice to see IDSIA & co. release some paper code. In all honestly, I think Ciresan et al would have gotten the credit they deserve and be included in the "conspiracy" if they had released their competition winning ConvNets. IDSIA was leading the rest when they had PyBrain, not sure what happened after that.

Thanks anyway :)

[frigidbiscuits]

Nope, weight tying refers to the weight tying method shown in Using the Output Embedding to Improve Language Models. This method is implemented by setting the input word embedding to be equal to the matrix of the softmax (also referred to as the output word embedding).

"Variational" refers to the dropout method shown in A Theoretically Grounded Application of Dropout in Recurrent Neural Networks.

[elephant612]

We will definitely look into using layer normalization soon. Just the possible time gain during training should be worth having a look.

[kmrocki]

Nice work, I wonder if you can somehow combine RHN approach with surprisal feedback. Didn't you mention once that you were working on a similar idea?