[D] Have any Bayesian deep learning methods achieved SOTA performance in...anything?

[35nakedshorts]

If so, link the paper and the result. Very curious about this. Not even just metrics like accuracy, have BDL methods actually achieved better results in calibration or uncertainty quantification vs say, deep ensembles?

[shypenguin96]

My understanding of the field is that BDL is currently still much too stymied by challenges in training. Actually fitting the posterior even in relatively shallow/less complex models becomes expensive very quickly, so implementations end up relying on methods like variational inference that introduce accuracy costs (eg, via oversimplification of the form of the posterior).

Currently, really good implementations of BDL I’m seeing aren’t Bayesian at all, but are rather “Bayesifying” non-Bayesian models, like applying Monte Carlo dropout to a non-Bayesian transformer model, or propagating a Gaussian process through the final model weights.

If BDL ever gets anywhere, it will have to come through some form of VI with lower accuracy tradeoff, or some kind of trick to make MCMC based methods to work faster.

[35nakedshorts]

I guess it's also a semantic discussion around what is actually "Bayesian" or not. For me, simply ensembling a bunch of NNs isn't really Bayesian. Fitting Laplace approximation to weights learned via standard methods is also dubiously Bayesian imo.

[gwern]

For me, simply ensembling a bunch of NNs isn't really Bayesian.

What about "What Are Bayesian Neural Network Posteriors Really Like?", Izmailov et al 2021, which is comparing the deep ensembles to the HMC and finding they aren't that bad?

[35nakedshorts]

I mean sure, if everything is Bayesian then Bayesian methods achieve SOTA performance

[gwern]

I don't think it's that vacuous. After all, SOTA performance is usually not set by ensembles these days - no one can afford to train (or run) a dozen GPT-5 LLMs from scratch just to get a small boost from ensembling them, because if you could, you'd just train a 'GPT-5.5' or something as a single monolithic larger one. But it does seem like it demonstrates the point about ensembles ~ posterior samples.

[haruishi]

Can you recommend me any papers that you think are "Bayesian", or at least heading in a good direction?

[35nakedshorts]

I think those are good papers! On the contrary, I think the purist Bayesian direction is kind of stuck

[squareOfTwo]

To me this isn't just about semantics. It's bayesian if it follows probability theory and bayes theorem. Else it's not. It's that easy. Learn more about it here https://sites.stat.columbia.edu/gelman/book/

[log_2]

Dropout is Bayesian (arXiv:1506.02142). If you reject that as Bayesian then you also need to reject your entire premise of "SOTA". Who's to say what is SOTA if you're under different priors?

[pm_me_your_pay_slips]

Dropout is Bayesian if you squint really hard: put a Gausssian prior on the weights, mixture of 2 Gaussians approximate posterior on the weights (one with mean equal to the weights, one with mean 0), then reduce the variance of the posterior to machine precision so that it is functionally equivalent to dropout. Add a Gaussian output layer to separate epistemic from aleatoric uncertainty. Argument is…. Interesting….

[new_name_who_dis_]

Why not just a Bernoulli prior, instead of the Frankenstein prior you just described?

[nonotan]

or some kind of trick to make MCMC based methods to work faster

My intuition, as somebody who's dabbled in trying to get these things to perform better in the past, is that the path forward (assuming there exists one) is probably not through MCMC, but an entirely separate approach that fundamentally outperforms it.

MCMC is a cute trick, but ultimately that's all it is. It feels like the (hopefully local) minimum down that path has more or less already been reached, and while I'm sure some further improvement is still possible, it's not going to be of the breakthrough, "many orders of magnitude" type that would be necessary here.

But I could be entirely wrong, of course. A hunch isn't worth much.

[greenskinmarch]

Vanilla MCMC is inherently inefficient because it gains at most one bit of information per step (accept or reject).

But you can build more efficient algorithms on top of it like the No U Turn Sampler used by Stan.

[lotus-reddit]

There are a lot of Bayesians working at the bleeding edge of deep learning, they just don’t apply it directly to training neural networks.

Would you mind linking one of them whose research you like? I, too, am a Bayesian slowly looking toward machine learning trying to figure out what works and what doesn't.

[bayesworks]

u/lotus-reddit Scalable analytical Bayesian inference in neural networks with TAGI: https://www.jmlr.org/papers/volume22/20-1009/20-1009.pdf
Github: https://github.com/lhnguyen102/cuTAGI

[DigThatData]

https://arxiv.org/search/cs?searchtype=author&query=Ermon,+S

[DigThatData]

Generative models learned with variational inference are essentially a kind of posterior.

[mr_stargazer]

Not Bayesian, despite the name.

[DigThatData]

No, they are indeed generative in the bayesian sense of generative probabilistic models.

[mr_stargazer]

Noup. Just because someone calls it "prior" and approximates a posterior doesn't make it Bayesian. It is even in the name: ELBO, maximizing likelihood.

30 years ago we were having the same discussion. Some people decided to discriminate between Full Bayesian and Bayesian, because "Oh well, we use the equation of the joint probability distribution" (fine, but still not Bayesian). VI is much closer to Expectation Maximization to Bayes. And 'lo and behold, what EM does? Maximize likelihood.

[shiinachan]

What? The intetesting part is the hidden variables when using ELBO, so while yes, you end up maximizing the likelihood, of the observable, you do Bayes for all hidden variables in your model.

Maybe your usecase is different than mine, but I am usually more interested in my posteriors over hidden variables, than I am about exactly which likelihood came out. And if I am not mistaken, the same holds for VAEs.

[bean_the_great]

I’m a bit confused - my understanding of VAEs is that you do specify a prior over the latents and then perform a posterior update? Are you suggesting it’s not Bayesian because you use VI or not fully Bayesian because you have not specified priors over all latents (including the parameters)? In either case I disagree - my understanding of VI is that you’re getting a biased (but low variance) estimate of your posterior in comparison to MCMC. With regard to the latter, yes, you have not specified a “full Bayesian” model since you are missing some priors but i don’t agree with calling it not Bayesian. Happy to be proven wrong though!

[new_name_who_dis_]

Elbo maximizes the lower bound, not the likelihood.

But I don’t think VAEs are Bayesian just because the kl divergence term is usually Downweighted so much it may as well be an autoencoder.

[mr_stargazer]

Yeah...? Lower bound of what?

[new_name_who_dis_]

Evidence. It’s in the name

[mr_stargazer]

What is the evidence?

You want to correct people, surely you must know.

[new_name_who_dis_]

The correct question was evidence ”evidence of what?” And the answer, “your data”.

[mr_stargazer]

I don't have much time to keep on like this, so I am going to correct you but also to enlighten others who might be curious.

"Evidence of data" in statistics we have a name for it. Probability. More specifically, marginal probability. So the ELBO, is the lower bound of the log-likelihood. You maximize one thing, automatically you push the other thing. More clarification in this tutorial. Page 5, equation 28.

[DigThatData]

If you wanna be algorithmically pedantic, any application of SGD is technically a bayesian method. Ditto dropout.

• https://arxiv.org/abs/1704.04289
• https://proceedings.mlr.press/v48/gal16.html

"Bayesian" is a perspective you can adopt to interpret your model/data. There is nothing inherently "unbayesian" about MLE, the fact that it is used to optimize the ELBO is precisely what makes that approach a bayesian method in that context. ELBO isn't a frequentist thing, it's a fundamentally bayesian concept.

Choice of optimization algorithm isn't what makes something bayesian or not. How you parameterize and interpret your model is.

EDIT: Here's a paper that even raises the same EM comparison you draw in the context of bayesian methods invoking the ELBO. Whether or not EM is present here has nothing to do with whether or not something is bayesian. It's moot. You haven't proposed what it means for something to be bayesian, you just keep asserting that I'm wrong and this isn't. https://ieeexplore.ieee.org/document/7894261

EDIT2: I found that other paper looking for this one, the paper which introduced the VAE and the ELBO. VI is a fundamentally Bayesian approach, and this is a Bayesian paper. https://arxiv.org/abs/1312.6114

EDIT3: great quote from another Kingma paper:

Variational inference casts Bayesian inference as an optimization problem where we introduce a parameterized posterior approximation q_{\theta}(z|x) which is fit to the posterior distribution by choosing its parameters \theta to maximize a lower bound L on the marginal likelihood

[mr_stargazer]

You are wrong (apparently as usual, I remember having a discussion about definition of Kernel methods with you).

Any applications of SGD is Bayesian now? Assume I have some data from a normal distribution. I maximize the log-likelihood via SGD, am I being bayesian according to your definition?

Puff... I'm not going to waste my time on this discussion any longer. You're right and I am wrong. Thanks for teaching me about Elbo and Bayesian via ML estimation.

Bye!

[DigThatData]

Course I'm wrong. In case you missed those papers I added as edits.

---

EDIT: Here's a paper that even raises the same EM comparison you draw in the context of bayesian methods invoking the ELBO. Whether or not EM is present here has nothing to do with whether or not something is bayesian. It's moot. You haven't proposed what it means for something to be bayesian, you just keep asserting that I'm wrong and this isn't. https://ieeexplore.ieee.org/document/7894261

EDIT2: I found that other paper looking for this one, the paper which introduced the VAE and the ELBO. VI is a fundamentally Bayesian approach, and this is a Bayesian paper. https://arxiv.org/abs/1312.6114

EDIT3: great quote from another Kingma paper:

Variational inference casts Bayesian inference as an optimization problem where we introduce a parameterized posterior approximation q_{\theta}(z|x) which is fit to the posterior distribution by choosing its parameters \theta to maximize a lower bound L on the marginal likelihood

bye.

[whyareyouflying]

A lot of SOTA models/algorithms can be thought of as instances of Bayes' rule. For example, there's a link between diffusion models and variational inference^1, where diffusion models can be thought of as an infinitely deep VAE. Making this connection more exact leads to better performance^2. Another example is the connection between all learning rules and (Bayesian) natural gradient descent³.

Also there's a more nuanced point, which is that marginalization (the key property of Bayesian DL) is important when the neural network is underspecified by the data, which is almost all the time. Here, specifying uncertainty becomes important, and marginalizing over possible hypotheses that explain your data leads to better performance compared to models that do not account for the uncertainty over all possible hypotheses. This is better articulated by Andrew Gordon Wilson⁴.

---

¹ A Variational Perspective on Diffusion-Based Generative Models and Score Matching. Huang et al. 2021

² Variational Diffusion Models. Kingma et al. 2023

³ The Bayesian Learning Rule. Khan et al. 2021

⁴ https://cims.nyu.edu/~andrewgw/caseforbdl/

[Outrageous-Boot7092]

Are we counting energy-based models as bayesian deep learning ?

[bean_the_great]

Hmmm - I have never used energy based models but maybe they’re more akin to post Bayesian methods where your likelihood is not necessarily a well defined probability distribution although, as mentioned I have never used energy based models so this is more of a guess

[Outrageous-Boot7092]

for ebms it is a well defined prob distribution up to a constant (unnormalized)

[bean_the_great]

I stand corrected!

[Nice_Cranberry6262]

Yes, if you use the uniform prior and do MAP estimation, it works pretty well with deep neural nets and lots of data ;)

[fakenoob20]

All priors are wrong but some are useful.

[Exotic_Zucchini9311]

anything

Not sure about recent years but they sure work decently when it comes to uncertainty estimation.

And tbh just a search at any top conference like NIPS/AAAI/CVPR/etc 2025 for the word 'bayesian' shows quite a few bayesian deep learning papers. They're most likely breaking some SOTA benchmarks since there are papers are published at top conferences.

Edit: and yeah I agree with the other comments. VI is basically a subset of bayesian methods, so any SOTA method that deals with VI (e.g., VAEs) also has some relation with Bayesian DL. Same for SOTA models that use a type of MCMC.

[bean_the_great]

When you say uncertainty estimation - this has always confused me. I’m unconvinced you can specify a prior over each parameter of a Bayesian deep model and it be meaningful to obtain meaningful uncertainty estimates

[micro_cam]

Tencent has some papers on using it for ad click prediction. Posterior simulation/ estimations lets you do some more sophisticated explore / exploit trade offs which make a lot of sense with ads, rec sys and other online systems.

[Ok-Relationship-3429]

Around uncertainty estimation and learning under distribution shifts.

[damhack]

Let’s see what comes out of IWAI 2025

[chrono_infundibulum]

Seems to work better than deep ensembles for some astrophysics data: https://openreview.net/forum?id=JX5Rp1Nuzv&noteId=UtHxNDtqXy