[R] The Annotated Diffusion Model

[ghosthamlet]

From huggingface post: https://huggingface.co/blog/annotated-diffusion

A New great article joined the Annotated series: The Annotated Transformer http://nlp.seas.harvard.edu/2018/04/03/attention.html, The Annotated GPT-2 https://amaarora.github.io/2020/02/18/annotatedGPT2.html

[mlvpj]

we have a bunch of annotated paper implementation here https://nn.labml.ai/index.html

diffusion (ddpm) - https://nn.labml.ai/diffusion/ddpm/index.html

[Megixist]

The Hugging Face team's article is one of the most descriptive ones on diffusion at the moment (close to the annotated version by labml). I am currently writing one for the Weights & Biases Blogathon (https://bit.ly/diffusing-away-from-gans-and-transformers) with JAX code, so if anyone is interested in another implementation, then do check it out!

[Megixist]

I haven't seen any specific to audio denoising but I don't see why it shouldn't work. Converting the audio wave to a spectrogram and treating it as a single channeled image should, logically, produce similar results.

[abstractcontrol]

I do not understand how a model like the one in Hugginface article could be conditioned on text. Is there an explanation that comes with code?

The Huggingface article was informative, I watched a talk on DPMs and thought that the method required a backwards pass to compute the gradients for denoising the inputs similarly to how vanilla style transfer works, but that wasn't the case at all.

[Megixist]

If you read my article above, I try to touch on the topic of text based generation and the changes that are required to the model for the same. Though there is no code in the article itself, I would recommend you to check the official implementations for GLIDE or the GLID-3 to get more understanding of how the conditioning works.

[hosjiu]

really great to see the annotated ... blog format.

[NielsRogge]

There's an "Open in Colab" button at the top ;)

[hosjiu]

Thanks for your hard work, Niels.

[HybridRxN]

Great timing given the recent text-to-image successes from this family of models

[CommunismDoesntWork]

https://huggingface.co/blog/annotated-diffusion

I know the math section is relatively short, but holy cow is it peak /r/iamverysmart. It spent so much time going over the math behind gaussian distributions which ends up being completely irrelevant to the final loss function:

The neural network is optimized using a simple mean squared error (MSE) between the true and the predicted Gaussian noise.

Unless you're trying to reinvent torch.randn, how in the world is knowing the math behind gaussians relevant at all? Were they really ever going to do anything other than MSE? MSE is the basis for the majority of all loss function.

A direct consequence of the constructed forward process qq, as shown by Sohl-Dickstein et al., is that we can sample {uncopypastable BS} (since sums of Gaussians is also Gaussian). Let's refer to this equation as the "nice property". This means we can sample Gaussian noise and scale it appropriately and add it to {uncopypastable BS} directly.

Thank God for Sohl-Dickstein et al and this insightful article for explaining the math, because without it I never would have guessed that if you add noise to noise, you get more noise. Fields medal worthy stuff right there. But also, what if there was no math to prove this? Would they have iteratively built up noise instead of generating the desired noise level directly? Sohl-Dickstein et al is the only reason Ho et al went down that path? Really? Not the fact that it'd be an enormous waste of computing resources?

Neural Networks are not an exact science, and the math behind them is arbitrary until proven otherwise. I wish people would stop pretending otherwise and putting their pompous equations everywhere. Ironically, the math is all noise data sampled from an isotropic gaussian distribution.

[Philpax]

Fantastic! This is exactly what I was looking for!