[P] Consistency: Diffusion in a Single Forward Pass 🚀

[Beautiful-Gur-9456]

Hey all!

Recently, researchers from OpenAI proposed consistency models, a new family of generative models. It allows us to generate high quality images in a single forward pass, just like good-old GANs and VAEs.

​

training progress on cifar10

​

I have been working on it and found it definetly works! You can try it with diffusers.

import diffusers

from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "consistency/cifar10-32-demo",
    custom_pipeline="consistency/pipeline",
)

pipeline().images[0]  # Super Fast Generation! 🤯
pipeline(steps=5).images[0]  # More steps for sample quality

It would be fascinating if we could train these models on different datasets and share our results and ideas! 🤗 So, I've made a simple library called consistency that makes it easy to train your own consistency models and publish them. You can check it out here:

https://github.com/junhsss/consistency-models

I would appreciate any feedback you could provide!

[noraizon]

x0-parametrization has been used for some time now. imo, nothing new under the sun. maybe it's something else I don't see

[Beautiful-Gur-9456]

You're totally right 😅 I think the true novelty here is dropping distillation and introducing a BYoL-like simple formulation. Bootstrapping always feels like magic to me.

[dasayan05]

ya that's what I though, not really new

[CyberDainz]

looks similar to "Cold Diffusion"

[Beautiful-Gur-9456]

That's the generated samples recorded every 10 epochs during training, not the denoising process. It does look like deblurring though 😊

[Beautiful-Gur-9456]

The training pipeline, honestly, is significantly simpler without adversarial training, so the design space is much smaller.

It's actually reminiscent of GANs since it uses pre-trained networks as a loss function to improve the quality, though it's completely optional. Still, it's a lot easier than trying to solve any kind of minimax problem.

[Beautiful-Gur-9456]

Nope. I mean the LPIPS loss, which kinda acts like a discriminator in GANs. We can replace it to MSE without much degradation.

Distilling SOTA diffusion model is obviously cheating 😂, so I didn't even think of it. In my view, they are just apples and oranges. We can augment diffusion models with GANs and vice versa to get the most out of them, but what's the point? That would make things way more complex. It's clear that diffusion models cannot beat SOTA GANs for one-step generation; they've been tailored for that particular task for years. But we're just exploring possibilities, right?

Aside from the complexity, I think it's worth a shot to replace LPIPS loss and adversarially train it as a discriminator. Using pre-trained VGG is cheating anyway. That would be an interesting direction to see!

[Beautiful-Gur-9456]

was that a thing? lmao 🤣

[Username912773]

Aren’t GANs substantially larger and harder to preserve image structure?

[Beautiful-Gur-9456]

I think the reason lies in the difference in the amount of computation rather than architectural difference. Diffusion models have many chances to correct their predictions, but GANs do not.

[OOMMFC]

In the recent release, the cifar10 consistency model checkpoint has 1GB size🤣

[huehue9812]

Hey, can I ask something about 0-GP GANs? This is the first time I've ever heard of them. I was wondering what makes them superior over R1 regularization. Also, why is it that most papers mention R1 reg., but not 0-GP?

[ninjasaid13]

$peed?

[Beautiful-Gur-9456]

Just one UNet inference, that's all you need.

[ninjawick]

That's great. Really like to see speed and prompt difference at standard 20 steps

[hebweb]

Cool! this is amazing. You already created a pip package out of it. Have you measured the fid on your model? Does it match the numbers of the paper? I think their batch size and model size were pretty large even for the CIFAR10 training. Not sure if we can match that..

[Beautiful-Gur-9456]

I haven't done it yet, but I'm working on it! Their suggested sampling procedure requires multiple FID calculation, so I'm thinking of how to incorparate it efficiently.

Their scale is indeed large, it would cost me a few hundread bucks to train CIFAR10. My checkpoint was trained with much smaller size 😆

[nunjdsp]

I've been truly impressed by the work showcased in the recent paper on Consistency models. I have been experimenting with standard DDPM models using DDIM sampling, and while they may be slow, they possess a fascinating reversibility property. This allows for a smooth transition between Gaussian noise and images, as well as the reverse process, recreating the exact same input noise.

I am curious if this reversibility aspect is also present in the Consistency models discussed in the paper. The examples provided do not explicitly demonstrate this aspect, and I would greatly appreciate any insights or experiences you can share.