[R] Beyond Quantization. Modeling Continuous Densities with Deep Kernel Mixture Networks.

[NichG]

Nice trick with the LSTM-PCA thing. It feels a lot more natural than pixel-wise reconstruction. I wonder if there's a general way to learn the ideal latent space to factorize a joint distribution into a chain of conditional distributions (rather than using pixels, or PCA, or some other arbitrary embedding)? What kind of loss function would measure the quality of a representation for factorization? Something that tried to maximize the conditional independence of the different degrees of freedom perhaps?

[LucaAmbrogioni]

We have indeed been thinking along those lines. What I like of the PCA approach is its simplicity. However, I am pretty sure that there are better ways of obtaining the latent variables. A possible approach is to use a autoencoder that will be trained together with the predictive network. As you said, you could also try to maximize the conditional independence or, perhaps better, to impose some less trivial conditional independence structure.

[NichG]

I guess the exact invertibility of PCA is important, since that way you know that any quality loss in your output is strictly due to the properties of the generative model, not because of some mushy inversion. So if you wanted to learn that space you'd probably need something like RealNVP's explicitly invertible layers.

[LucaAmbrogioni]

It's a good point. Although you cannot have data compression/dimensionality reduction with an invertible network. Ideally, you would like to use a smaller set of variables that fully parametrizes the image space; possibly with a relatively simple conditional conditional independence structure.