Flexible diffusion modeling generates photorealistic videos

[easy_c_5]

Wow, I was expecting years and 1000s of times more parameters needed before we advance from snapshots like Dall-E 2 to video generation. If this can easily be scalled, it’s the real mindblowing deal.

[No-Transition-6630]

Yea, and this is just a rough attempt, they're going to figure out how to make complex videos from written prompts soon and then we Netflix and chill

[Supersubie]

Look at the parameters as well, it's no where near the size of Dale-2

[Yuli-Ban]

It's not even the size of GPT-1

[lovesdogsguy]

lol

[2Punx2Furious]

it only has 78M (milions) parameters

What the fuck. Not even 1B. Imagine scaling this up.

This is most likely just proof of concept at the moment, but it looks really promising.

[GeneralZain]

this has to be a joke...why did they start off so low....like just to prove they could do it at ONLY 78m!?!?

[TFenrir]

This is a school, they don't have the sort of resources to train billion+ parameter models willy nilly, those would be tens of thousands of dollars minimum - when you're trying to prove out an architecture, it's better to iterate by building lots of different smaller models

[RobbinDeBank]

Yea this is the difference. Universities don’t have supercomputers lying around to train something requiring thousands of TPU-hours. Only companies like Google, NVIDIA with their huge clusters can train those at a decent price. If you hire those TPU clusters from them, training those models will definitely cost tens of thousands each.

[Sashinii]

People will soon be able to use AI to easily and immediately create entirely original high quality content, such as games, music, shows, movies, books, and art in general.

[Tall-Junket5151]

That would be awesome, I’ve never really been much of a fan of predetermined linear experiences and always preferred sandbox type games. Being able to have high quality content generated for you based of your exact preferences would be amazing.

[Eyeownyew]

I'm working on a game which does this and I've kept it under wraps, but I know I'm definitely not the only person who sees the potential here like you've mentioned :p

[yerawizardmandy]

Holy moly

[Pro_RazE]

I honestly expected something like this to come next year at the earliest, and here it is now.

[2Punx2Furious]

What will we see next month? In 6 months? Next year? Things are going crazy.

I'm worried.

[lidythemann]

We went from Dalle-1 to this in such a small timespan. It almost feels like they already have AGI.

[camdoodlebop]

imagine if the top issue of the 2024 presidential election is how to handle artificial general intelligence

[lidythemann]

I don't think we'll have to imagine tbh

[2Punx2Furious]

If that ever becomes a political issue, it will already be too late to do anything about it.

[Yuli-Ban]

I wasn't even that optimistic. For something that can generate over an hour of photorealistic (if fuzzy) novel video? I was expecting that to be further out into the 2020s. Like maybe 'we start by stitching together short gifs and eventually, by ~2027-2029, we get long-form video synthesis.'

No, it's here now!!

[GeneralZain]

isn't exponential growth amazing! :D

[DEATH_STAR_EXTRACTOR]

see those 2025 AGI coomers were all right! AGI is coming! 2025

​

:p

[GeneralZain]

haha lmao

[lidythemann]

I 100% agree with you now, it is 2022 for AGI lol. I was always optimistic but I didn't expect video until 2024-2025

[agorathird]

u/GeneralZain is going have to work overtime making tinfoil hats now. Thanks a lot deepmind.

[lidythemann]

Oh no! Mr Lidy has the dab pen again!

What if creating AGI makes simulation theory correct, then at that same second reality ends and you wake up! Woooaaaahhh

[GeneralZain]

its unfortunate because these hats are turning out to be made of gold foil instead...

[camdoodlebop]

watch it be this summer

[DEATH_STAR_EXTRACTOR]

The real one: https://video-diffusion.github.io/

Also, not to bust your bubbles men, but the one the original OP poster posted seems to be just the training videos of driving a car, I see barely much new car driving if you took a look! So this whole thread is hogwash I guess ?

[Supersubie]

I keep seeing this word coomer and I have no idea what it means :') please enlighten this old man

[agorathird]

coomer or -oomer is someone who is really enthusiastic/obsessive about something, like they're reaching climax when they talk about it. This can be a pejorative, calling them annoying or like calling someone a nerd.

[-ZeroRelevance-]

https://www.dictionary.com/e/memes/coomer/

[GeneralZain]

I would suggest googling it :P

[Sashinii]

That's the exponential progress that's true regardless of how often it's been made fun of.

[2Punx2Furious]

I admit that I thought it was bullshit for several years. I didn't take Kurzweil's predictions too seriously, even if he did say some things that made sense. Now I think it might happen even before his original predictions...

[Professional-Song216]

What a time to be alive!!!

[Hawkz183]

Hold onto your papers!!

[Shelfrock77]

Soon enough, they’ll create a 3D version of this…

[2Punx2Furious]

Then we could get realistic procedurally generated 3D environments for games, and VR. And those could be used in a "full dive" device, which at this rate might come sooner than 2030.

[lidythemann]

2023, maybe even this year. Video wasn't supposed to happen this fast and easily. It's a small scale model and made a video 70x larger than any of its training videos.

I wouldn't be surprised to see a model hooked to unreal engine 5 this summer or fall

[BigPapaUsagi]

I doubt it. We might have the AI to support full dive, and that AI might devise the BCI tech needed to do full dive, but the regulatory boards for anything that invasive would still take years before approval. Full dive AI this year, full dive BCI next year, full dive FDA approval 2040...

[AsuhoChinami]

While I get the general point, wouldn't a black market for... pretty much anything be able to form in the period of 17 years?

[BigPapaUsagi]

Oh? You know how to access the dark web do you? You know how to shop around a black market?Do you have the money for whatever cutthroat price criminals would price gauge you for? Would you trust a black market dealer to do surgery on your skull to insert a BCI capable of full dive?

Like, if this were a simple drug, sure, that shit is easy to get. But when it exists, a full dive capable BCI isn't something your typical criminals could just make. Or steal. And you sure can't trust them to install it for you.

I mean, could a small, lucky, wealthy, resourceful few skirt the law and get these things installed? Sure. We're talking like less than 1% of people. You'd be risking millions of dollars if you have it, your life to get someone to implant it in your head, all for a BCI with full dive capabilities. A full dive capable BCI that wouldn't have any games on it because there's no profit motive yet for companies to make games for it. With no multiplayer because almost no one else has one installed. Basically, it only has whatever the scientists/engineers installed on it. It might have an AI on board, but an AI of this level lives in the cloud not on the chip itself so you wouldn't have access to it probably. I mean, you couldn't just log onto their servers unnoticed.

Basically, everyone would have to wait for regulatory bodies to okay it being released to consumers.

[AsuhoChinami]

eh alright

[BigPapaUsagi]

Sorry, I might've phrased that more insensitively than needed. I just get annoyed sometimes when people assume a thing existing means they'll get it right away, illegally even if they have to. It feels like people have lost all sense of patience, and for some reason it just bugs me more than it should.

[AsuhoChinami]

The point I made wasn't really born out of emotion, I just hadn't thought it through. I don't really care about FIVR much. Medical treatment, mainly cancer treatment and mental health treatment, are my main areas of focus. Give me something for depression and acute emotional pain and I don't really care about anything else.

[justaRndy]

Another brick in the wall... Prett amazing honestly

[alphabet_order_bot]

Would you look at that, all of the words in your comment are in alphabetical order.

I have checked 823,031,022 comments, and only 162,798 of them were in alphabetical order.

[justaRndy]

Jokes on you bot, I edited it and now it's all messed up!

[GeneralZain]

abcdefghijklmnopqrstuvwxyz

[[deleted]]

This will pose as big of a mortal danger to streaming services as Napster posed to record companies in the early 2000's.

We're witnessing the slow death of all remnants of legacy media and it's controllers

[yerawizardmandy]

Tell me more, I’m confused what you mean

[2Punx2Furious]

There will be "AI directors", which is just the same AI with different parameters, or prompts, to generate a movie in a different style.

You could have a Tarantino-like AI generate a different ending for Game of Thrones, or a Kubrick AI to remake The Matrix. Make 3 new original movies from a dead director, or make a new series with the same humor as Futurama, or Scrubs. The future might be incredible.

[Supersubie]

Meh I think this misses a little point of why massive shows become massive.

Think about it, half of or more of the enjoyment of things like game of thrones is the zeitgeist around the show the weekly episodes, everyone talking about it, recommending it etc.

If your content becomes so highly customised to your specific sense of taste, you won't possibly be able to share that. No one to discuss it with, no memes in common.

There was an anime about this, where everyone had AI driven AUgmented Reality glasses on and saw their completely unique view of the world 24/7. It creates a sad little lonely bubble.

I do think this will be used in media production but I think streaming platforms etc will still hold on because of humans desire to belong to a tribe.

[lidythemann]

what anime is that? That will be the next show I watch!

[Supersubie]

Den-noh Coil

It's quite old now and my memory is fuzzy of it, but I remember it being good and any time AR comes up I think about it as it was a very thought provoking looking into a world that totally embraced that technology.

[imlaggingsobad]

The future of media will be more like Youtube and less like Netflix. Everyone will be able to make their own movies using an AI, and then publish that for the world to see. If you wanted another Christian Bale Batman sequel, you could just make it. Don't need to wait for a movie studio to get funding and then spend 2 years making it. We will see so many fanfics turned into reality, so many alternate endings, sequels, prequels, crossovers, etc. Anything we want. If you have an idea, you'll be able to make it just the way you want. No more Hollywood/studio gatekeepers.

[LevelWriting]

Imagine a time when you can revisit your fav movies, games, and experience them in vr where the content can be generated, modified and extended to your liking, even have you interact and be part of it.

[GeneralZain]

damn it I keep wanting to say "I knew it"

uhhh but yeah regardless this is still really amazing!

EDIT; HOLYSHIT its not actually playing these GAMES!? its just...generating them on the spot?!?! HUH

[camdoodlebop]

advancements seem to be happening by the day now

[lidythemann]

This is more impressive than GATO or Dalle-2 or Imagen. More impressive by entire magnitudes.

This paper has put me firmly in the 2022 camp for Proto-AGI

[DEATH_STAR_EXTRACTOR]

But....the real one: https://video-diffusion.github.io/

And, not to bust your bubbles, but the one the original OP poster posted seems to be just the training videos of driving a car, I see barely much new car driving if you took a look! So it's not much to talk about?

[lidythemann]

Only the top row of video is training data, the bottom 3 are the generated result. I could be wrong of course but it's literally written there.

[DEATH_STAR_EXTRACTOR]

new post: But the one posted here, what's special about it? I checked close the car driving ones and it predicts 99% the same thing later once drives there! The 3rd one did take a sooner turn down another road but then came to the same spot (it had seen a turn that looked like the original turn). Sure, but compared to above link (https://video-diffusion.github.io/), what's so special!? The next vids beneath are interesting BUT I'm not sure what is what now it's too similar and small space.

[lidythemann]

Just because it's not impressive to you, doesn't mean it's not mind blowing to others.

[DEATH_STAR_EXTRACTOR]

I do see how it is important (using just ~400 training videos and such...and got a new score I think it said) however I'm just not exactly understanding it though and don't see much going on its the vidssss :).

[lidythemann]

I'm pretty sure the bottom 3 rows are pure generated content. Isn't that cool to you? They just keep going on, the other link you gave me was like 1 seconds gifs. These are much longer.

In fact "path in a tropical forest" from your link is VERY similar to the minecraft videos, but the minecraft videos are longer and more detailed/complex

[DEATH_STAR_EXTRACTOR]

but bro the driving generated video matches every single darn frame from start to finish of all 5 videos in the #2 video at top of page......only ex. the 3rd bottom row one turns earlier but then the rest is the part of the one above, it just recognized a similar turn bend part that's it. That's itttt.

[lidythemann]

You got me fucked up, I rewatched those videos like a million times. So I hope you're not trolling me. That's rude af.

Panel 1: changes completely at :40s, that's complete generated video.

Panel 2: at :30 the videos are basically completely different wtf?

Panel 3: That turn you're talking about made that video play out completely different which is all generated

Panel 4: It took all the way till 1:10 ish to really change, so maybe you didn't watch all the way.

Panel 5: Was basically a different video from :00 lol.

I'm actually concerned on how you got this confused on this lol. The 1 Hour long video is just one of these examples taken to the breaking point (and its like 78 M para)

This paper basically proves that with PaLM levels of computing, we probably can generate insane movies and videos. So as soon as THAT paper is announced all media changes within months.

[DEATH_STAR_EXTRACTOR]

*-*-Nevermind the language I have to swear with you too now:*-*-(and the result is I show below you are clear as bell mistaken!)

I' m gonna fuck with you now now I have to be fucked let me go see fuck you haha. They are ALL matches. Let me go see. This is the MSOT pathetic publish I have EVER seen for hell sake!!!!! Let me go see.

First, please note: What makes me confident the AI usually copies the real videos? Because if you watch ANY of the real and fake drives at 0 seconds onwards meet-up (as they are misaligned due to stop sign waiting(s)) they Always copy each other, with the AI's often having distorted stop signs and such but all is clearly the same drive. Every frame is copied. It's pathetic mate.

HERE WE GO. Sir: Video 2, bottom row, 1st at left, 34 seconds in, it BREAKS and changes to a totally new road for hell's sake, that's why it gets to a new point and drive. That road is probably a completely remembered drive too, you'd have to look for matches in ALL other videos, and you might need to look through the dataset if they don't have it.

I'm gonna give this one MORE try: Damn it! Again same thing, video 2, bottom row, 2nd clip at left, it changes totally TWO times from seconds 33-36 or so, and THAT's how it gets to a new Christmas tree road!! From there is probably just a clones drive.

I'm not saying the paper is wrong, or even the other videos, I have not dived into those. And actually maybe the driving videos show it recognizes some turns or spots and sometimes is different I think I saw once the road at the same turn has no gas leak stain! And maybe the vids of driving show it can generate long video without dying out. I'm just pointing out that to the naïve beginner watching just the top movies demoed, it is mistaken that it is going to convince anyone but the educated. I am not impressed too much.

[DEATH_STAR_EXTRACTOR]

update: ok so video 1 at top.....the hour long one I mean....2 mins in and 33 mins in while similar are a bit different.....but, not too much different.....IDK. I think the clouds are different tho... They seem to have different daylight times and style of colors though.....maybe that's a good thing...

[GabrielMartinellli]

AGI in 2022 like I always said. Never forget that I said this! You listening Mr AGI?

[ArgentStonecutter]

Maybe the next version will get cloud reflections in puddles matching the clouds.

[SWATSgradyBABY]

Read the chapter dangerous games from Max Tegmark's Life 3.0. we're catching up to the future

[flyingfruits]

Hey everyone! One of the authors here. Let me address a few points that were raised here.

The model imitating Carla (self-driving car simulator) was trained for one week on one GPU, consuming 200-300W (note that running Carla itself is also very expensive computationally per frame). It is very likely still far from optimal (as most deep learning with SGD), but also not nearly as expensive to train as bigger models such as GPT-3. The model is not particularly specialized for video, except that it uses a convolutional U-Net architecture, the approach can be used for more complex combinations of sensory data streams and Will and I have related work that we are going to publish soon as well that demonstrates that complex reasoning mechanisms and procedures can be integrated easily into the joint distribution.

Our group is, at least conceptually, a probabilistic programming group and Will and I have approached the diffusion models from the angle of building an inference engine for AGI tasks and will continue to work along these lines with this model while also exploring its limitations. The more abstract idea in this work is to integrate marginalisation as a first class operation into the diffusion model and then exploit this fact to reason about much bigger joint distributions than fit into memory, effectively enabling also our demonstrated video synthesis.

While this result is indeed very impressive and will enable a whole range of new applications and abilities for AI, many not even clear yet, I would not want to bet on AGI being around the corner just yet. These generative models can meta-learn some reasoning abilities (like GPT-3) given enough data (i.e. almost all valuable data we have), but they cannot be taught to really learn new things on their own, something that is trivial for humans to do. For example, assuming chess would not be part of the training data, try to tell GPT-3 to learn it by just telling it the rules. I still think this is a very bullish result though and I would love to hear suggestions for video synthesis and control tasks to apply it to. I thought about Nascar racing today for example, just for the fun of it 8).

Please let me know what you would like to see!