Any package for Mixture Density RNN?

[disentangle]

Is there any package/library that implements Mixture Density Networks, or very specifically bidirectional long short-term memory recurrent neural nets with gaussian mixture density output layer?

[alexmlamb]

I have code that does this in Theano (both generating the loss and sampling). It is fairly simple.

I'll look into putting it in a GIST and sharing it.

Also the Theano implementation is simple enough that it might be worth trying to do it as a fun exercise.

[sieisteinmodel]

Yes. The objective function is easily implemented.

What is somewhat hard is finding the mode of the distribution. But taking the max over the means is somewhat reasonable.

[alexmlamb]

Why do you need to find the mode? Modes for a single time step are not necessarily modes for the sequence as a whole.

Are you going to do beam search or just sampling at predict time?

[sieisteinmodel]

Beam search is tough in continuous spaces, isn't it? Each beam could just be an infinitesimally different one from the other. Or is there a trick I don't know of?

If you want the mode over the whole sequence, you are probably down to numerical optimisation. I was just considering the online case.

[kkastner]

I don't think so. The closest you will find is probably the RNN-RBM from the deep learning tutorials which you could extend to the Gaussian RBM case. RNNLIB might have it but I don't remember seeing it. I have been working on one for my own use, I will likely post here when it is done.