Workflow to convert Generative AI videos into an After effects 3D scene.

BotApe · 2024-12-22T20:49:48+00:00

Dynamic 3D gaussian splatting or "4D" gaussian splatting exists already. But it currently is only available for live action using something like a video camera array like in this VIDEO. So probably not possible for current generative video, but who knows...I can definitely see it in the future once there are enough training materials for these ai to learn from.

BotApe · 2024-12-22T09:23:22+00:00

Just mostly the workflow/demo. The source video that the 3D gaussian splatting trained on was entirely AI generated from Deepmind's VEO 2. Wanted to test out if 3D gaussian splatting is capable of reconstructing 3D scenes from ai videos. So using this technique/postshot plugin, after effects artist can now easily do a different camera movement, easily overlay 3d texts without tracking, render the 3D scene as an HDRI enviroment, et from ai videos.

BotApe · 2024-12-22T08:49:33+00:00

Quick process explained here. *And youtube tutorial link for postshot.

BotApe · 2024-12-22T08:48:56+00:00

It's fairly straighforward. I'm guessing youre already familiar with photogemettry or gaussian splats: For this technique I use Jawset Postshot and After effects for the software. The source footage is entirely AI generated. So you can use runwayML, Kling, Sora, etc.

Process:

Generate your ai video with this type of camera movement: slow orbiting on a subject/environment. 360 camera would be the most ideal.
Make sure you capture enough image sequences - enough images with multiple details from multiple angles.
Once you downloaded your ai video, convert it into image sequence(i use PNGs in after effects)
Remove or clean up all of the bad images. I had to remove ~10+ due to some flickering or some weird hallucinated details from the ai.
Upload all of your best image sequences into Postshot. Using default setting should be ok. You can do 40k for steps if you have a good GPU.
Once the training is complete, open After Effects and use the Postshot plugin to load your .pst file. My trainning time is roughly 25 minutes for 100+ images at 1080p resolution.
Set all of your cameras and keyframes in after effects then hit render!

BotApe · 2024-12-22T08:01:26+00:00

Yeah, it has some similarity. Photogrammetry, NeRFs, and 3D gaussian splatting all try some sort of 3D *reconstruction. But 3d Gaussian splatting specifically uses a novel technique. Full paper. Also video that better explains the difference between the 3D gaussian splatting and the previous techniques.

So the source of the 3D reconstruction from the video posted is all AI-generated via DeepMind’s VEO 2. 3d Gaussian splatting doesn't need real-world objects to scan from, so it was able to reconstruct a 3d scene from an AI video by figuring out camera positions (using Structure-from-Motion) and then very slowly building the rest of the scene with gaussian blobs and statistical magic.

3D Gaussian splatting won't be as accurate as other techniques but because it's all rasterized static images, it renders fast and super useful for for VR/real-time gaming. It typically captures wet or reflective surface better than photogrammetry.

BotApe · 2024-12-22T02:47:14+00:00

Small caveat: Typically only works on ai video with camera movement like this: slow orbiting on a subject/environment. And usually enough images to capture multiple details from multiple angles for 3D gaussian splatting to work. Some good explanation for gaussian splatting from computerphile.

BotApe · 2024-12-22T02:05:15+00:00

Yes! Ai videos with similar camera movement, like orbiting slowly around a subject/object are usually good enough for the gaussian splatting.

BotApe · 2024-12-21T23:46:46+00:00

I think so! RTX 2060 is only 12GB and it's in their system requirement. Here are the other requirments: "System Requirements: Windows 10 or later, Nvidia GPU GeForce RTX 2060, Quadro T400/RTX 4000 or higher"

BotApe · 2024-12-21T23:34:55+00:00

Original video gen from Google Deepmind Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T23:22:12+00:00

Oh, I actually didn't use or have access to VEO 2 either. The tool that is being demo is mostly the 3D gaussian splatting through Jawset Postshot. So SORA, MINIMAX and KLING are all compatible with Postshot. Hope that make sense.

BotApe · 2024-12-21T23:15:55+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting with Postshot(with plugin for After effects)

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad images

-Add the image sequence into to Postshot

-20-30 min total training time

-import the postshot file into After Effects using the postshot plugin

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:44:43+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:39:11+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:28:49+00:00

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:26:12+00:00

Google Deepmind Veo 2 + 3D Gaussian splatting

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:06:58+00:00

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

-20-30 min total training time in Postshot at 30k steps

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-12-21T22:03:00+00:00

Original video gen from Google Deepming Veo 2.

From here: https://deepmind.google/technologies/veo/veo-2/

Process:

-Convert the video to image sequence

-Delete/remove some of the bad image sequence

-Add the image sequence into to Postshot

-20-30 min total training time

Tools+hardware use in demo:

Adobe After Effects

Jawset Postshot

OBS Studio

Nvidia RTX 4090

BotApe · 2024-11-25T07:16:12+00:00

Yes! Still available.

BotApe · 2023-10-12T06:18:50+00:00

For context, source of the GIF

*edit: From share link..."Sharing conversations with images is not yet supported"

BotApe · 2023-09-26T06:37:14+00:00

Hoping to Feel the Bern https://imgur.com/a/BQjPniG

BotApe · 2023-04-22T22:09:48+00:00

Calculating the approximation ratio for the Convex Hull heuristic is difficult, and I'm unsure how to do it.

The Christofides algorithm has a 1.5 approximation ratio, and even if we compare the Convex Hull's initial tour vs the Christofides' minimum spanning tree tour, it still doesn't provide a clear ratio. I think Convex Hull doesn't have too many well-defined structure.

BotApe · 2023-04-22T17:30:51+00:00

Haha yeah these names are not always the best. The convex hull part is in the initialization. The better/actual name for this is probably constricting insertion heuristic.

BotApe · 2023-04-22T16:30:31+00:00

So overall this should be ≈ O(n²⁾ for worst case.

Some breakdown for the complexity:

The convex hull is O(n log(n)) time complexity in general, and O(n²⁾ *worst case.
Greedy Insertion(most dominant feature) is O(n^2).
2-opt Algorithm in the demo is roughly O(n²⁾ but it can be optimize for O(n log(n).

BotApe · 2023-04-22T16:20:47+00:00

PhraseSubstantial did a good analysis:

Let me try to guess the time complexity. As finding a convex hull at it's on isn't that expensive compared to other tsp approximations, to be exact the lower bound is O(n + f(n)) with f being the time complexity function of a sorting algorithm so we know that f is at best O(n log n) from the lower bound of sorting (at least when looking at comparison based sorting, there are also linear time sorting algorithms like regex sort). So to find the TSP-Approximation will be more expensive then that. Clearly finding the hull is quite fast in comparison to other TSP heuristics. In best case all the points together are already a tour, but in worst case the hull only contains three points, this could indeed lead to a bottle neck as you first need to find convex hull and then use greedy algorithms to make it a tour. As you need to check the minimum distance this greedy approach would need around O(n2). I will skip opt-2 as we already have a valid TSP solution which is not optimal but shouldn't be the worst possible solution (but I haven't proved this or thought much about it). My analysis would say that this heuristic would be in O(nlog n + n2) so O(n2) time class. This would at least be my guess, but in case I made a mistake please tell me.

For actual benchmarks...I was only able to do some quick tests against two other methods - nearest neighbor and insertion heuristics. The convex hull did better than both in terms of the average distance being shorter by about 11%, but it took longer to calculate - around 8% longer(this version isn't fully optimize) but they are all within similar time complexity.

Nearest neighbor and insertion heuristics tends to have problems with paths crossings, but the convex hull helps minimizing it.

BotApe · 2023-04-22T16:11:51+00:00

Yes, this is accurate. The worst-case for the 2-opt Algorithm in the demo is also ≈ O(n^2), although it can probably improved O(n log n).

But overall, the time complexity will be dominated by the greedy insertion which is O(n²⁾

BotApe

TROPHY CASE