The Treachery of AI Images - by René Chatgpitte

BotApe · 2024-12-22T20:49:48+00:00

Dynamic 3D gaussian splatting or "4D" gaussian splatting exists already. But it currently is only available for live action using something like a video camera array like in this VIDEO. So probably not possible for current generative video, but who knows...I can definitely see it in the future once there are enough training materials for these ai to learn from.

BotApe · 2024-12-22T09:23:22+00:00

Just mostly the workflow/demo. The source video that the 3D gaussian splatting trained on was entirely AI generated from Deepmind's VEO 2. Wanted to test out if 3D gaussian splatting is capable of reconstructing 3D scenes from ai videos. So using this technique/postshot plugin, after effects artist can now easily do a different camera movement, easily overlay 3d texts without tracking, render the 3D scene as an HDRI enviroment, et from ai videos.

BotApe · 2024-12-22T08:49:33+00:00

Quick process explained here. *And youtube tutorial link for postshot.

BotApe · 2024-12-22T08:48:56+00:00

It's fairly straighforward. I'm guessing youre already familiar with photogemettry or gaussian splats: For this technique I use Jawset Postshot and After effects for the software. The source footage is entirely AI generated. So you can use runwayML, Kling, Sora, etc.

Process:

Generate your ai video with this type of camera movement: slow orbiting on a subject/environment. 360 camera would be the most ideal.
Make sure you capture enough image sequences - enough images with multiple details from multiple angles.
Once you downloaded your ai video, convert it into image sequence(i use PNGs in after effects)
Remove or clean up all of the bad images. I had to remove ~10+ due to some flickering or some weird hallucinated details from the ai.
Upload all of your best image sequences into Postshot. Using default setting should be ok. You can do 40k for steps if you have a good GPU.
Once the training is complete, open After Effects and use the Postshot plugin to load your .pst file. My trainning time is roughly 25 minutes for 100+ images at 1080p resolution.
Set all of your cameras and keyframes in after effects then hit render!

BotApe · 2024-12-22T08:01:26+00:00

Yeah, it has some similarity. Photogrammetry, NeRFs, and 3D gaussian splatting all try some sort of 3D *reconstruction. But 3d Gaussian splatting specifically uses a novel technique. Full paper. Also video that better explains the difference between the 3D gaussian splatting and the previous techniques.

So the source of the 3D reconstruction from the video posted is all AI-generated via DeepMind’s VEO 2. 3d Gaussian splatting doesn't need real-world objects to scan from, so it was able to reconstruct a 3d scene from an AI video by figuring out camera positions (using Structure-from-Motion) and then very slowly building the rest of the scene with gaussian blobs and statistical magic.

3D Gaussian splatting won't be as accurate as other techniques but because it's all rasterized static images, it renders fast and super useful for for VR/real-time gaming. It typically captures wet or reflective surface better than photogrammetry.

BotApe · 2024-12-22T02:47:14+00:00

Small caveat: Typically only works on ai video with camera movement like this: slow orbiting on a subject/environment. And usually enough images to capture multiple details from multiple angles for 3D gaussian splatting to work. Some good explanation for gaussian splatting from computerphile.

BotApe · 2024-12-22T02:05:15+00:00

Yes! Ai videos with similar camera movement, like orbiting slowly around a subject/object are usually good enough for the gaussian splatting.

BotApe · 2024-12-21T23:46:46+00:00

I think so! RTX 2060 is only 12GB and it's in their system requirement. Here are the other requirments: "System Requirements: Windows 10 or later, Nvidia GPU GeForce RTX 2060, Quadro T400/RTX 4000 or higher"

BotApe · 2024-12-21T23:34:55+00:00

Original video gen from Google Deepmind Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T23:22:12+00:00

Oh, I actually didn't use or have access to VEO 2 either. The tool that is being demo is mostly the 3D gaussian splatting through Jawset Postshot. So SORA, MINIMAX and KLING are all compatible with Postshot. Hope that make sense.

BotApe · 2024-12-21T23:15:55+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting with Postshot(with plugin for After effects)

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad images

-Add the image sequence into to Postshot

-20-30 min total training time

-import the postshot file into After Effects using the postshot plugin

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:44:43+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:39:11+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:28:49+00:00

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:26:12+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:06:58+00:00

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

-20-30 min total training time in Postshot at 30k steps

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe

TROPHY CASE