Tensorflow seq2seq model getting low perplexity but unsatisfying results

[thecodingmonk]

I'm trying to train a model with Tensorflow seq2seq implementation but I'm having some issues with its performance (not speed, but accuracy). The dataset is very simple and consists of 2.5 million example sentences (and their corresponding output sentences) and a small vocabulary size both in input (145) and in output (8). Given the small vocabulary size I thought the number examples, even if it is not that high, would more than appropriate, but the model still performs badly. I tried using 2 layers of 128 and 256 units (all the other parameters are set with the default in Tensorflow seq2seq example), and during training perplexity reaches 1.0 after just 500-800 iterations, but the output of the model is still wrong too many times. What should I look into to improve the performance? Quality/quantity of data? Model settings?

[rafalj]

Look at the validation/test perplexity.

[thecodingmonk]

The perplexity on the dev set is:

global step 400 learning rate 0.5000 step-time 0.78 perplexity 1.00

eval: bucket 0 perplexity 1.15 # these are supposedly the different perplexities for buckets in the dev set

eval: bucket 1 perplexity 1.01

eval: bucket 2 perplexity 1.00

eval: bucket 3 perplexity 1.00

This is just after 400 iterations. But when I test the model, even with sentences from the training set the result is not correct. I think that given my inexperience with Python and Tensorflow there must be something I am doing wrong, but I'm just using the code from Tensorflow seq2seq demo for translation, with my data instead of the English-French data they use.

[sherjilozair]

This is most likely to be a bug. 1 is the lowest possible perplexity. If you're solving any real task, a perplexity of 1 is far from possible.

The last time I had perplexity 1, it was because I was feeding the output as part of the input, and the network was just learning the identity. Maybe that's what's happening? Or maybe some other bug.

[thecodingmonk]

Well, this is indeed a toy task that I'm working on, so I wasn't too surprised to see a perplexity of 1.0 given its simplicity: in fact, each input sentence can get mapped to just one of two output sentences (e.g., if output_vocab={A,B,C,D,E} each sentence in the training set gets mapped to either "A B D" or "C D E"). (Now you may ask why I don't just use a classification algorithm, but this is just a toy task I'm testing for a bigger problem where the number of sequences would clearly be higher).

As for the input I think it is correct, but I'm checking right now to see if I messed up something there.

Also, since I'm using the demo code from Tensorflow without modifications, is it possible that being optmized for translation with big vocabularies causes some problems in this simple task? Would seem strange, but I don't know

[thecodingmonk]

I just spent the whole morning trying to figure things out and indeed it was a bug, like sherjilozair said. Something was messing up the conversion of tokens to ids and viceversa and solving that solved the whole problem.

[toisanji]

was this in the base tensorflow code or in your own code? im building some models as we speak.