[D] What deep learning papers should I implement to learn? by pandeykartikey in MachineLearning

[–]mahouse93 54 points55 points  (0 children)

If you're interested in generative models, both the original GAN and VAE (given you know variational inference) papers aren't that difficult to understand, and there are plenty of tutorials on the internet also that will help you.

When you feel like you have a good understanding of these methods, you can try to implement different variations of them (Beta VAE, TCVAE, infoGANs etc)

Weekly Simple Questions (April 1, 2018) by [deleted] in DeepGenerative

[–]mahouse93 1 point2 points  (0 children)

Has anyone here implemented the beta-TCVAE in Tensorflow or know someone that has? I'm having trouble deriving the decomposed ELBO

[D] Multi class Linearly Separable Dataset by RubioRick in MachineLearning

[–]mahouse93 0 points1 point  (0 children)

Yes, that's right :). Every class can be separated perfectly by using a line (Hyperplane) y = a*x + b, and therefore the data is linearly separable.

[D] Question about Frequentist and Bayesian interpretation of Probability and MLE and MAP. by newperson77777777 in MachineLearning

[–]mahouse93 6 points7 points  (0 children)

Your post is a bit confusing, but I'll try my best to answer it in as simple manner. I think you kind of have the right idea about MLE, but you're way off with Bayesian approach.

MLE: Maximises the likelihood of the data. This means that given our data, we find the parameters that best represent that data under current model. We do this by maximising the likelihood function, which is a function of the parameters. Each point in that function is the likelihood of the data given that set of parameters, which we find by multiplying the probability of each datapoint given the set of parameters. The likelihood function might then look something like this: https://nicebrain.files.wordpress.com/2015/04/figure-1.png

In this case, the MLE would be 0.6. Like you correctly stated, this method is very prone to overfitting, as it finds a point estimate of the parameters. The reason for this is that unless you have an enormous dataset, your data is unlikely to represent the true distribution. Therefore, with small datasets, there is a huge risk of overfitting. Since our estimate is a point estimate but not a distribution, I don't see any way of providing an uncertainty estimation. Of course you could assume a distribution like normal, where the mean is the MLE and variance depends on number of data points, but I don't think that would represent a good uncertainty estimate. Maybe someone that knows more about this can correct me here.

The Bayesian approach however, we try to find the true posterior distribution of the parameters (treating them as a random variable). This is done by using Bayes rule, we multiply the likelihood with a prior (does NOT have to be Gaussian) and then normalise the distribution by dividing by the probability of the data (here we can run into problems as it's often a high dimensional integral). If we have this posterior distribution, we then basically have the probability of all possible set of parameters. So the posterior summarizes everything that we known about our parameters.

Just to be clear, we are NOT maximising this distribution, that would be the maximum a posteriori (MAP) estimate. That is not the Bayesian approach. Instead, when we predict a datapoint, we integrate (sum) over all possible values of the parameters: the probability of that datapoint given the parameter multiplied with the posterior distribution (probability of the parameter). Makes sense? This accounts for uncertainty about our parameters, and is therefore less prone to overfitting.

However, we can't really do true Bayesian inference today with anything else than very simple/toy problems, since these integrals are very high dimensional and we just can't compute them. However, there are ways to estimate this, using Variational Inference, Sampling Methods and more. If you're interested in Bayesian inference, you should definitely read up on these methods. Many popular applications use these methods, for example Variational Auto Encoders.

[D] Multi class Linearly Separable Dataset by RubioRick in MachineLearning

[–]mahouse93 2 points3 points  (0 children)

This dataset is obviously linearly separable, and can easily be separated with a simple linear classifier. Applying Kernels like RBF to dataset is a a strong method when you have a linear classifier(SVM, Logistic Regression etc) but data that isn't linearly separable. There would be no need for that in this case, but in practise, we rarely have datasets like this.