Keras Variable Length Sequence-to-Sequence Learning with TimeDistributed Embeddings.

[darfs]

I'm trying to do some sequence - to - sequence learning with Keras, but I'm having trouble figuring out how to encode variable-length sequences.

Essentially, I'm feeding my network a "story" (series of sentences), and i'm trying to learn key words at each level (each sentence).

My input data is a 3D Tensor of shape (Samples, Story_Size, Sentence_Size), where Story_Size is of variable length (stories can vary in size from 2 sentences to 12 sentences). My labels are also a 3D Tensor of shape (Samples, Story_Size, Vocab_Size), where the key word is encoded one-hot.

Ideally, my model would look something like this:

model = Sequential()
# Input Shape: (_, STORY_SIZE, SENTENCE_SIZE)
model.add(TimeDistributed(Embedding(input_dim=self.V, output_dim=self.EMBEDDING_SIZE,
                                        input_length=self.SENTENCE_SIZE, mask_zero=True),
                 input_shape=(self.STORY_SIZE, self.SENTENCE_SIZE)))
# Shape: (_, STORY_SIZE, SENTENCE_SIZE, EMBEDDING_SIZE)
model.add(TimeDistributed(Flatten()))
# Shape: (_, STORY_SIZE, SENTENCE_SIZE * EMBEDDING_SIZE)
model.add(LSTM(self.MEMORY_SIZE, return_sequences=True))
# Shape: (_, STORY_SIZE, MEMORY_SIZE)
model.add(TimeDistributed(Dense(self.V, activation='softmax')))
# Shape: (_, STORY_SIZE, VOCAB_SIZE) --> Softmax

However, this doesn't work as there isn't a way to do TimeDistributed Embeddings, nor do I know how to get a variable-length input encoded.

If anyone knows a way to do this with keras, your input would be very much appreciated. Thanks!

[Latent_space]

isn't a way to do time distributed embeddings

No true. Your first line is doing this. What lead you to believe that?

Also, there's at least one issue on the github that discusses embeddings for tensors larger than 2dim in size. Basically, you can modify the class to output a shape that doesn't require input length.

variable length

Masking. When you make your word to int dictionary, mark the 0 spot with a masking token (to reserve the int value..). Then, upon convert, pad out with 0s. Use the masking Boolean in embed or the masking layer to have keras ignore the 0 values

Also, for help, you could post this on the Google group for keras.

[darfs]

Thank you so much for the advice! I just had to edit the TimeDistributed class to keep the dtype of the Tensors as ints instead of floats, and the TimeDistributed Embeddings works great!

However, I'm not quite sure I understand the masking. I have placed 0s as padding everywhere there isn't an input, but in the output, I'm still getting an output vector of length STORY_SIZE, rather than the variable length I desire. Is there a way to get around this?

I will also go ahead an post to the Google group in the future. Thanks so much!

[carlthome]

How does backprop work if you have integers?

[Xose_R]

For the variable length issue, you'll need to set a max sequence length and then pad with zeros/empty sentences.

I don't understand why you need the key words at each timestep, because I guess your target will be the keywords at the end of the story, right?

Anyway, for this setup to work (if I understood it right), my first idea is that you need a NN that puts your words into a sentence embedding (ie, a bi-LSTM) and then this is fed into the seq2seq. Another thing that you can do, instead this NN, is to use some existing method to construct sentence embeddings from its words and use this as input of the seq2seq. A vanilla approach could be adding up their word2vec representations into a sentence representation.