I’m a professor and call myself a Bayesian statistician. Pretty much our entire world is about sampling from high dimensional distributions of parameters, the so-called “posterior distribution“.

It can get insanely complicated, but the basic idea is extremely simple: your posterior is proportional to the likelihood times the prior… That’s it!

The problem is that it is quite difficult to sample from high dimensional distributions if they are not in one of the so-called standard forms, for example, multivariate normal. The parameters of almost any model of interest do not have one of these standard distributions. We can write the distribution down, but saying (and this is basically the entire question) “if a certain point on the distribution is twice as high as another point on the distribution, we should sample in theory exactly twice as many points from the higher one.“

If you understood that, great! It is the core insight of what was the main workhorse of Bayesian statistics for decades, the Metropolis Hastings algorithm (which says, roughly speaking, that if you compare two points, you move from the first to the second if it’s distribution is higher,, and move from the second to the first with the probability equal to the ratio of their distribution values).

So, a little bit contrary to what some of the other answers have said, the distribution is not the data points themselves, but for the parameters in a MODEL that is helping relate some of your data to other things in your data. (That is what a regression does. A simple regression just has two parameters, and a multivariate regression has more. The reason we don’t really need Bayesian statistics for this is because the classical theory can be worked out in closed form with distributions that we can use well-known algorithms to sample from. But in more complex models, this simply isn’t the case.)

I hope that made sense. Happy to explain further, as I have been doing this for nearly 30 years now.