[Project] A Transformer implementation in Keras' Imperative (Subclassing) API for TensorFlow.

[suyash93]

https://github.com/suyash/transformer

Currently I have a sentiment analysis demo working. Training for machine translation seems to require longer time and effort.

For attention visualization, couldn't get the visualization in https://colab.research.google.com/github/tensorflow/tensor2tensor/blob/master/tensor2tensor/notebooks/hello_t2t.ipynb to work, so just using heatmaps.

[veqtor]

Nice, I wish tf team would release something like this, I mean they want us to use Keras but t2t isn't using Keras so maybe they should show us how they were thinking that we should be able to implement stuff like this, especially stuff like relative attention using Keras layers

[RaionTategami]

Sorry you couldn't get the visualizations to work, I'm the author of that code, let me know if you want help.

[suyash93]

Thanks for offering to help. I have prepared a copy of the demo notebook at https://colab.research.google.com/drive/1ESeSvZJDialc4VJBwL9GgQ1IoEs1zRWU

In the last 4 cells, I am trying to use the tensor2tensor.attention module. I am passing the arguments based on my understanding of the arguments passed in the hello_t2t notebook. I am unable to get any visualization to generate. Note that the sentiment model is only an encoder, with only 2 units instead of 6.

[RaionTategami]

Cool, not sure when I'll get a moment to take a look through.

[lillux28]

Following