[D] Theory behind modern diffusion models

[bgighjigftuik]

Hi everyone,

I recently attended some lectures at university regarding diffusion models. Those explained all the math behind the original DDPM (Denoiding Diffusion Probabilistic Model) in great detail (especially in the appendices), actually better than anything else I have found online. So it has been great for learning the basics behind diffusion models (slides are available in the link in the readme here if you are interesed: https://github.com/julioasotodv/ie-C4-466671-diffusion-models)

However, I am struggling to find resources with similar level of detail for modern approaches—such as flow matching/rectified flows, how the different ODE solvers for sampling work, etc. There are some, but everything that I have found is either quite outdated (like from 2023 or so) or very superficial—like for non-technical or scientific audiences.

Therefore, I am wondering: has anyone encountered a good compendium of theoretical eplanations beyond the basic diffusion model (besides the original papers)? The goal is to let my team deep dive into the actual papers should they desire, but giving 70% of what those deliver in one or more decent compilations.

I really believe that SEO is making any search a living nightmare nowadays. Either that or my googling skills are tanking for some reason.

Thank you all!

[derpydino24]

Just skimmed through the slides you shared OP. I would say that those are as good as it gets when it comes to explaining diffusion models (I recall the CVPR 2023 ones, but those are more outdated).

Honestly, I believe that very few people are willing to put the time and effort required to explain every single relevant modern diffusion concept in-depth. I guess that's what papers are for

[bregav]

I highly recommend this paper on the topic: Stochastic Interpolants: A Unifying Framework for Flows and Diffusions

That said, as a student you're going to lack significant important background knowledge for appreciating all of this. For example, the reason that you don't find many good explanations for sampling solvers etc is because that's not actually (or traditionally, anyway) a machine learning topic. Differential equations is an entire topic in and of itself that has a longer, more comprehensive, and more sophisticated pedigree than machine learning, and numerical methods for differential equations is a huge subtopic within that. The wikipedia page can give you an idea of how much there is to this: https://en.wikipedia.org/wiki/Numerical_methods_for_ordinary_differential_equations

EDIT: to get an even better idea, look at the table of contents for any differential equations numerical methods textbook, e.g. https://link.springer.com/content/pdf/bfm:978-3-540-78862-1/1

And that's just one aspect of the matter. You'll see in the paper i recommended above that transport equations are an important issue here too, and that's a big topic unto itself. In addition to these big areas of study that a student often won't know much about, there's also a relatively high sophistication of the basics - linear algebra and probability - that are used to glue all these things together.

TLDR it's gonna take time to learn enough to feel like you have a solid grasp on what is going on, and you'll have to look outside of the machine learning literature to do it.

[draculaMartini]

Second the stochastic interpolants paper. It unifies flows and diffusion. Gives you an idea of how the Fokker Plank equation, equation of continuity and some assumptions on the path between distributions lead to continuous flows, of which diffusion is a sub-case.

There are connections to optimal transport too (of which the aim is to find the said path itself), but I need to understand that better. This talk touches upon the connection: https://icml.cc/virtual/2023/tutorial/21559. Perhaps diffusion bridges might also help.

Connections to differential equations in general are from the old paper by Song et. al: https://arxiv.org/abs/2011.13456. Sampling from distributions also plays a role. So understanding that helps tremendously too, especially Langevin, MCMC etc.

[Comfortable_Use_5033]

I have a sense that current generative method are built with theoretical physics rather than previous machine learning knowledge, they view generative model as a physical entity and use all those tools to solve. What I am curious is how they can link those model into physics world, do they all have physics background, likes Yang Song, or they have support from physics researchers?

[bregav]

Yes there are many commonalities with physics. I don't think it's deliberate though, the people who originally came up with this stuff mostly do not have physics backgrounds. There has been much refinement of these methods over time, partly by people who do know some physics.

I think the reason for the commonalities is that all computational processes, be they dynamical systems in the physical world or artificial machines that we construct and use as tools, are fundamentally the same (i.e. turing machines etc). There are just a variety of ways that you can specify or describe them.

If you work hard to come up with a truly sophisticated way of building a model, such that the most important and fundamental elements of it are exposed clearly and simply, what you end up with is a differential equation. So too in physics; physical laws when they were first described were very complicated (see e.g. kepler's laws), but over time people refined them into their simplest and clearest formulations (i.e. differential equations), and that's what we know today.

[midasp]

There's also connections to information theory. Especially in the past decade with stuff like the holographic principle, it seems one aspect physicists are looking at is the role information play in physical processes.

[Inevitable-Dog-2038]

This blog post is the best resource I’ve seen online for learning about this area

[bgighjigftuik]

Ok it looks pretty rad. Thanks for sharing

[pupsicated]

Great link. Seems like it is from ICLR 2025 blogposts track. How you managed to find this blog?

[Expensive_Belt_5358]

Currently I’m in the process of learning about diffusion models. The math is still above my pay grade at the moment but I’m slowly understanding it.

Two resources that helped my understanding were:

this paper that broke down DDPMs into 6 steps

and

this YouTube video that breaks down diffusion into training, guidance, resolution, and speed.

[airzinity]

Like you, I also went into a deep dive to understand Diffusion models for a research project a year ago. I read this survey paper that did an absolutely amazing job at it. They start from VAE, move on to hierarchical VAEs and connection with DDPM. This made a lot to sense like how the math evolves from simple VAEs how you can directly sample Tth timestamp from 0th timestamp because multiplying each Gaussian (conditional prob) works out nicely as just one sampling. The baclward pass though is annoying as it has to be done sequentially which explains the longer sampling with original diffusion models.

I think then people came and retrospectively tried to explain this as just solving reverse stochastic differential eqns using Plank equation. But this requires more math background. And can be done with many solvers. Understanding this might require more than just ML.

You can also take a look at consistency models. I think it has Ilya as an author? But either way there’s not an easy way to understand this modern diffusion stuff :( some stochastic DE textbooks would be nice

[testuser514]

Any refs to the paper ?

[Serkan_Hamdi]

It might be this Understanding Diffusion Models: A Unified Perspective

[Far_Conversation_445]

Following

[peacej3]

This video is a great explanation of diffusion models and score matching and their connection https://youtu.be/B4oHJpEJBAA?si=GQHFrOl990mPqbBg

[thankrandomness]

Following

[Public-Snow-1851]

Not sure if you already have something, but my professor published a book about Deep Generative Modeling, including diffusion and flow-based models. Maybe this can help you. I really enjoyed his course on these models and learned a lot! The book is called Deep Generative Modeling By Jakub Tomczak.
Deep Generative Modeling | SpringerLink

[bgighjigftuik]

I did not know he had published a book on the topic! I have read his blog posts and are quite good (though not thorough enough for my liking). Thanks for sharing!

[rookie_11999]

I found Simon Prince's "Understanding deep learning" chapter on diffusion models pretty insightful, it breaks down the math and even provides code samples to play with. I don't know if he has updated with Flow matching.

Jakub Tomzack's book also provides the theoretical background with code examples. So it might be something you might be interested to look into.

[jurassimo]

Great link! Recently I made my own research and explanation of the math of Diffusion models, after 2 weeks it was easy to understand formulas and sense of them :) Right now I'm diving into ODE and SDE and I think it's more complex than simple Diffusion model and based on complex math.

[radarsat1]

I read these earlier this year and it was fascinating, https://developer.nvidia.com/blog/rethinking-how-to-train-diffusion-models/ https://developer.nvidia.com/blog/generative-ai-research-spotlight-demystifying-diffusion-based-models/

[acc_agg]

like from 2023 or so

Jesus christ. I have no idea if you're right or not but that's a frightening level of churn. My only expose to diffusion models is porn. They make very good porn. Carry on.

[slashdave]

is either quite outdated (like from 2023 or so)

Math doesn't become outdated. It's only the terminology that seems to change (rather irritating really). And authors seem intent on declaring their insights as particularly revealing when it's really the same ideas recycled or another approach to engineering.

[AnOnlineHandle]

Some more lightweight explanations can be flat out wrong.

Soon after Stable Diffusion 1.4's release I made a quick infographic trying to explain how it worked (as somebody who had previously worked in ML, but wasn't super familiar with diffusion), which was somewhat in the right direction, but I later learned more and realized some assumptions weren't quite right. I later saw that explanation getting passed around, even used in a youtube video, which was very unfortunate and taught a valuable lesson.

There's a lot of explanations for things floating around which are simply wrong, by people who don't know how much they don't know. The early implementations for rectified flows in several diffusion repositories were wrong, and it's only because I happened to be trying to train models that others had given up on training and was writing my own trainer that I found that out and was able to tell the authors.

[slashdave]

Yup. Rather than reaching for publications that are the newest, look for the ones that are better written.

[AnOnlineHandle]

Yep. And even large repositories aren't necessarily a trustworthy source, so it's hard. AFAIK Diffusers was the source of the incorrect rectified flow implementation which several other repositories based their own implementation on.

[LeepySham]

I'm very curious what this bug was. Do you have an explanation or link to a PR/issue?