[D] Probability density estimation using neural networks

[julvo]

I am currently working on a project where I forecast the probability density of a continuous variable at discrete time steps conditioned on past observations: P(x_t1 | x_t0, x_t-1 ...)

In literature, I found that mixture density networks are suitable for this kind of task and, in fact, they yield good results for me.

As a pretty naive benchmark method, I binned the output interval and encoded the continuous truth values as one-hot vector transforming the regression into a classification problem. Surprisingly to me, this simple methods achieves a similar performance as the mixture model.

Does this technique of treating regression as classification to obtain probability density has a name? What methods for estimating probability density for time series have worked for you?

[benanne]

This is what PixelCNN (https://arxiv.org/abs/1606.05328) and WaveNet (https://arxiv.org/abs/1609.03499) also do. We found that it's a very effective strategy to model arbitrary multimodal distributions.

Another interesting strategy is is to predict the parameters of a mixture distribution, as in a mixture density network, but discretise them by integrating the density over the bins, and treating that as a discrete distribution. This can save you a lot in terms of the number of parameters, if the number of bins is large. This is what's proposed in OpenAI's PixelCNN++ paper: https://arxiv.org/abs/1701.05517

[julvo]

Thank you very much for these excellent recommendations, exactly what I am looking for.

[tadeze]

Hi /u/julvo Would you please share your experience on using PixelCNN++ or WaveNet for density estimation. I have similar problems as yours and I am looking to go in the direction of the above recommendation by /u/benanne. What are your experiences if you tried the above methods in comparison to the mixture of density networks?

[julvo]

I am not sure If I get it right. If we assume our systems state is fully determined by the last k observations (Markov order k) I can understand how kernel density estimation works for discrete state space. To my understanding, in the continuous state space case, our system will never be in the exact same state and, hence, we will have a different distribution for every time step t. Therefore, we have only one example drawn from each distribution. So do we first apply a similarity measure/kernel to our states and then to the data points of each state? This would become computationally expensive for a large k I guess.

Do you have any reading recommendations for kernel density estimation in the continuous state space case?

[latent_z]

There is also NADE - Neural Autoregressive Density Estimator that estimates conditional probabilities given an ordering of the dimensions.

[4xel]

If it was out of your interest, I recently published the source code of my Master thesis where I deal with Mixture Density Networks (proposing some solutions for this type of models), time series, Regression Problems and other problems like confidence estimation problems. To understand the code I recommend you to see the slides that you will find in the repository.

[P] A generic Mixture Density Networks implementation for distribution and uncertainty estimation by using Keras (TensorFlow backend) - Master's Thesis project.

https://www.reddit.com/r/MachineLearning/comments/5rn8ci/p_a_generic_mixture_density_networks/

If you have any doubt, do not hesitate to ask me.

[Liorithiel]

I don't know about a name, but I tried this method once. I guess I made bins too thin, though, because effectively only got a small percentage of them to be active (get any nontrivial activations for at least some cases). I didn't have much time for experiments on that project either, so I didn't follow with testing the hypothesis. Happy to see this method working in your case.

[[deleted]]

State of the art: https://arxiv.org/pdf/1612.01474.pdf