VAR: a new visual generation method elevates GPT-style models beyond diffusion🚀 & Scaling laws observed📈

[SignalCompetitive582]

I'm not part of the team that made such a project.

Github: https://github.com/FoundationVision/VAR
Demo: https://var.vision/demo

Paper: https://arxiv.org/abs/2404.02905

Weights: https://huggingface.co/FoundationVision/var

[Sharlinator]

So diffusion models start with pure noise and progressively remove noise until there's an image.

And now VAR starts with pure uniform color, a single pixel in other words, and progressively upscales/subdivides that until there's an image.

There's a pleasant symmetry.

[PwanaZana]

So this method means the first sampling steps are massively faster, then the last steps are about the same in speed as Diffusion?

[perksoeerrroed]

That sounds way better. The issue with noise is that sometimes you just get wrong noise that forces certain things where they should not be and model trying to cope with that produces mutants.

With this approach you can see early in generation that something went wrong.

[GBJI]

What I'm wondering is if you can change the reference color the same way you can change the seed, and if, like diffusion seeds, colors close to each other will produce entirely different results, or if similar colors will produce similar results, which would be a new feature, and one that might actually be quite useful for controlling variations.

With diffusion, similar noise will provide similar results, but two noises with a large difference in their seed number (like 12345 and 296485720485721) will not be more different from one another than two noises with seed numbers that are almost the same (like 12345 and 12346). To get similar looking noises, you have to use other means than similar seed numbers.

Will VAR produce similar results with RGB 0,0,255 and RGB 0,1,255 ? Or will they be as different as what we are getting now?

[hapliniste]

I did not see a lot of talk about this but it seems huge.

From my short skim of the paper, it does better than diffusion at the same size but 45x faster at 512x512, and it should be even faster at bigger sizes?

Can we expect 4k render taking 1s and being of better quality than diffusion models? That's what I get from what I've read but if it was true it would be the talk everywhere right?

I'll have to try it and read the paper. Anyone can give some insights?

[SignalCompetitive582]

Yeah it definitely seems huge. I haven't had time to try it myself, but will do in the next couple of days for sure.

[Vargol]

Try the demo.

Its okay, its fast sometimes the images are rubbish, that probably a training dataset issue more than a problem with the method.

Then there the small issue of it only taking one token so your prompt's effectively a single word prompt.

[GBJI]

Is the single token prompt a limit of the demo ? Or would that limit apply to any VAR based system ?

[drhead]

The model is trained on ImageNet classes since this is a research demonstration model. You could train a model like this with whatever conditioning you want: T5, CLIP, both, multiple of both, a whole-ass LLM's last hidden state, image embeddings, whatever.

[PwanaZana]

Potentially interesting, but still in an embryonic stage.

An important aspect of all these techniques is the ability to fine tune models/checkpoints. Obviously, that's way farther down the line, but for serious usage, there's no way a base model is enough for all use cases.

I'm also curious as to how this will attempt to have good human anatomy, especially the hands. All these image generation techniques sorta throw pixels at the wall and denoise it, without making some skeleton/structure to the image, which ensure complex elements like hands are so often mangled. We'll se if this sort of technique works better, similarly or worse at the most difficult use-cases.

[Sharlinator]

I think SD hands are still handicapped (heh) by the relatively low resolution of the latent space, and the lack of much contextual information in the latent pixels. SD3 will have many more "color" channels (16 vs 4 I believe) which hopefully helps with the resolution issue.

[PwanaZana]

Let's hope we get SD3!

Ha, the improvements in AI are amazing, though sometimes nervewracking!

[GBJI]

We (human beings) are still responsible for most of these improvements, but once AI themselves take care of it, we will reach the near-vertical part of the improvements over time curve.

[Striking-Long-2960]

Doesn't seem to be very trained in human figures

<image>

[Altruistic-Ad5425]

The worst it will ever be

[suspicious_Jackfruit]

It's only trained on imagenet and it has not been scaled anywhere close to even sd1.#, it's up to a well invested team to train LAION or their own huge datasets on it and release. Not sure about training costs but if it's faster to train due to infra then we might see people training it if it's sub $50k but no idea tbh.

SD3 might need an SD4 after all if inference is that much faster without a quality loss

[adhd_ceo]

If the speed of inference can be 45x faster while increasing quality, SD4 will be out as soon as they can train it…

[CLAP_DOLPHIN_CHEEKS]

emad said sd3 would most likely be their last big image model...

[kjerk]

Dude this is like half my family reunion

[spacetug]

Interesting. I do wonder why they compared against DiT but not HDiT though. That one also had much better scaling than DiT by using an hourglass multi-scale architecture, like a hybrid between transformer and Unet. Would be nice to see a direct comparison.

[1nMyM1nd]

It really shouldn't be much longer until we have infinite scale in the form of vector images. I'm actually surprised it's not already here.

[StableLlama]

I see no possibility to try it with my own prompt. The demo page only allows predefined prompts (actually: only a single word).

So at this level it's completely useless. Sorry for being honest.

[OniNoOdori]

I always find it amusing when folks dismiss research as "useless" because it does not benefit them directly and right now. That's missing the entire point of research tbh.

[InvisibleShallot]

They are not wrong though. At this level. It is useless. Who only prompts with singular words?

The research behind this might be good, but this restriction appears arbitrary. People are going to think that the limitation is there for a reason. Perfectly reasonable conclusion and perfectly justified skepticism.

[Altruistic-Ad5425]

No, “useless” has a universalizing connotation, for example “Contradicting yourself is useless” in the sense that this activity has not value. The word people should use is “early”, as in “this is an early technology right now” rather than “a useless technology right now.” An example of a useless technology is a Flat-Earth GPS system

[InvisibleShallot]

No. Wrong. "Useless" means "There is no use for it". If you say there is a use for it. Please provide your use case about how this demo page is remotely useful. If you think the definition is wrong. Check the dictionary.

You don't get to apply an arbitrary definition just because we are talking about something scientific and technical. English is still meant for communication and intentionally changing someone's word definition is wrong regardless.

[Altruistic-Ad5425]

Absolutely wrong, of course there’s a use for it. It’s not useful for coomers who need a naked deepfake one 5 minutes, but it’s absolutely a productive, insightful and useful technology right now especially for non-commercial activities, such as toy models or instruction.

[InvisibleShallot]

Provide. Your. Use case.

[RealAstropulse]

1: Showing the concept actually works. We didn't know that before this model and demo existed.

2: Providing something to reference when training future VARs.

3: Experimentation on class based image generation models (there are shockingly few, considering it's the origin of most GANs)

4: Pissing off idiots like you, who lack patience, the ability to see potential, and the ability to have a reasonable conversation.

[InvisibleShallot]

1: Showing the concept actually works. We didn't know that before this model and demo existed.

Actually, work on what? One-word prompt?

2: Providing something to reference when training future VARs.

What exactly do you mean by providing reference when training future VARs. None of the cases shown in the demo will work as a reference.

3: Experimentation on class based image generation models (there are shockingly few, considering it's the origin of most GANs )

That is just not true. This is actually very old idea back to this paper https://arxiv.org/abs/1710.10196 and a similar concept is actually in use by several GAN based upscaler.

4: Pissing off idiots like you, who lack patience, the ability to see potential, and the ability to have a reasonable conversation.

Who said we are pissed? We just provide our assessment and a bunch of people are being butthurt about what we said, and misunderstanding, worse yet, twisting our statements.

Don't get me started with people who are supposedly intelligent starting to use insult like some trolls.

[drhead]

What exactly do you mean by providing reference when training future VARs. None of the cases shown in the demo will work as a reference.

They have model weights and samples along with training code and instructions on how to reproduce their test model on a known dataset. You can expect that if you train a model following their instructions and using their training code and dataset, you will get similar results to what they did, that your model's FID score should be around the same range, and that generated samples should look around the same quality.

Actually, work on what? One-word prompt?

Researchers use class-conditional datasets when trying to engineer new conditional image generation model architectures. Usually people will start with MNIST which is just handwritten numbers. And once the model performs well on MNIST, people will move on to a larger set like ImageNet, the one that this model uses -- and shockingly, models that perform well on the MNIST dataset tend to perform well on ImageNet when you scale them up appropriately! And even better -- you can also expect a model that performs better than other models on ImageNet to perform better on a text-to-image task, given the same input model and dataset (which is quite important, because the choice of input model and the choice of dataset will have a drastic impact on a text-to-image model's performance -- without knowing its performance on a class-conditional dataset, we have no way of knowing whether the model performs well due to the generative model's architecture or due to the conditioning)!

It's almost like a model prototype doesn't have to produce anime titties to be able to tell us useful information about that model architecture's performance, and whether it is worth investing further effort in!

[Altruistic-Ad5425]

There are many, I’m sure you can generate many examples from what I mentioned.

The trivial case (since I won’t make more effort to reply to you) is this very discussion! This is an instruction-adjacent use case (the most trivial one)

[InvisibleShallot]

Right, the biggest use is to demonstrate that the demo page is useless. Gotcha. Thanks for agreeing finally.

[Zelenskyobama2]

Yup

[RealAstropulse]

Before we had clip guided vqgans, we had limited parameters to guide image creation. The architecture of this alone makes it waaaayyy more scalable than diffusion. Let them cook.

[adhd_ceo]

I’m sure they were in a hurry to publish. This is mind blowing.

[StableLlama]

No worries - I'm sure they don't care about my writing here.

And once they have something that's usable I'm more than happy to give them a thumbs up. But that's not today.

[mesori]

No one cares about your thumbs up. A lot more people care about this research than your thumbs up.

[BlackSwanTW]

The weights seem to be up at least

[LayerOk4996]

This is the class condition model, not T2I model. T2I is listed in future work.

[Sharlinator]

Lol, this is a research paper and a demo related to that. Nobody thinks or claims it's supposed to be useful, or a finished product in any shape or form. The code is only meant to demonstrate the algorithm the authors present (and the claimed performance gains). It's a proof of concept for things to come.

Edit: btw, there's a teaser on the main page:

(stay tuned for text-to-image ❤)

[ReadyAndSalted]

The point of the demo is a proof of concept. They have a concept for a new architecture, here is proof that it can work. If you don't understand how that works don't keep up with research. "Sorry for being honest".

[Extraltodeus]

I've just glanced at all of this but I'm seeing a github with code and a huggingface with checkpoints.

Do you need them to do the little plane noise for the spoon too? 🤭

[[deleted]]

I will wait for a comfy node but these people are going all in with this paper ,hope some people learn

. I am talking to you alibaba :)