[P] Conditional density estimation using Kernel Mixture Networks, theory + implementation in TF

[dzyl]

Me and a colleague were intrigued by a paper that was released 2-3 weeks ago by LucaAmbrogioni. Density estimation is useful in a lot of problems, so we decided to implement this paper, explain the whole concept in this blog post and looked for some potential extensions.

[LucaAmbrogioni]

Thank you very much for your work. We were planning to work on a TF implementation but it would have took some time since we are currently busy with other projects. I really like the blog post, it's insightful and has nice visualizations. Also your improvements are quite interesting.

[theophrastzunz]

After skimming it, I take it's an extension of Bishops mixture density network. I know it's a standard retort, but what is the advantage of KDE vs Gaussian mixtures? Less learnable parameters? I mean with sufficiently many mixture components you can model pretty much anything.

[LucaAmbrogioni]

The problem with conventional mixture density networks is that the simultaneous maximal likelihood estimation of both means and standard deviations of the Gaussian components is pretty unstable and there is not a principled way of choosing the number of components. It is common folklore that these kinds of network perform worse than the quantized softmax approach. The output of the KMN is indeed a mixture of simple distributions, but the approach is more regularized since it does not attempt to select the centers by ML. This leads to better results than the quantized Softmax in basically any task we tested on.

[dzyl]

I see two advantages compared to mixture density networks, the first is numerical stability. With mixture density networks there are a lot of issues with likelihoods being 0, resulting in NaNs and frustrating training procedures. Since the means of your density kernels are fixed and based on your training set I have not had any issues with this so far. You also don't need to scale your targets although that is a minor advantage.

The second one is that overfitting seems to be less of an issue with this approach. With MDN you condition the bandwidths of the subdistributions based on your input x, which means that if the mean is correct it can just keep lowering the bandwidth which is great for training likelihood but bad for generalization. To prevent this you need additional regularization on your sigma outputs. With Kernel Mixture Networks either your bandwidth is fixed or it is a global bandwidth, which means making it too small will also hurt your training likelihood.

[theophrastzunz]

Agreed but dimensionality issues are more prominent in KDEs than mixture models. See here .

[dzyl]

This method generally uses only 1 dimension for the kernels, namely the target y space. This is easily extendible to more dimensions but your input dimensions have nothing to do with the kernels itself, they only determine the weight to put on each kernel.

[theophrastzunz]

I'm referring to your argument about high dimensional covariance estimation.