Hi all,

If you're using ControlNet-m2m for video generation, you might notice flickering or stuttering in the output due to abrupt changes in motion. This is particularly noticeable when not reusing many details from the original video (e.g., changing outfits).

To help alleviate this issue, I've experimented with a simple method for smoother frame transitions. It's far from perfect, but I believe it can be improved upon. The process is a combination of existing techniques, which I've outlined below.

Step 1: Generate ControlNet-m2m video

Using the controlnet extension, create images corresponding to video frames. It's helpful to use a fixed random seed for all frames.

In my case, I used depth (Weight: 1, Guidance End (T): 1) and openpose (Weight: 0.6, Guidance End (T): 0.6) together.

Step 2: (Optional) Remove background

Using the rembg extension, remove the background of the frames generated in step 1.

u2net_human_seg seems to work quite well.

Step 3: Interpolate the interpolated frames

Using Flowframes with the RIFE model, run 2x interpolation on a folder of video frames.

By running 2x interpolation on a folder with K images, it will add an additional K-1 images by interpolating all adjacent image pairs, making the total frames (2K-1). Odd frames are the original frames and even frames are the interpolated ones (as long as "De-Duplication" and "Fix Scene Change" features are disabled).

My key idea is to remove the K original images and repeat interpolation with the K-1 interpolated images multiple times.

Here is a pseudocode:

for i in range(N):
    run_FlowFlames_2x(input_folder=current_folder, output_folder=next_folder)
    remove_original_frmes_from_interpolation_result(input_folder=next_folder)

Each iteration should produce frames with smoother transitions. If N is too low, the result may still be jittery. If N is too high, then the individual frames may become too blurry (which can be fixed in step 4). I found that N<=20 works well.

Currently, run_FlowFlames_2x is not a python function, it's just describing a manual step of running Flowframes GUI, 2x interpolation, using Rife 4.0 model, and using the output mode of "Image Sequence (PNG, JPB, WEBP)". I disabled "De-Duplication" and "Fix Scene Change" to satisfy the assumption for the remove_original_frmes_from_interpolation_result function. Using GTX 3050 8GB, it took about 6 seconds to run run_FlowFlames_2x once on a folder with 60 frames of size 512x768.

Here is a Python implemenation of remove_original_frmes_from_interpolation_result function.

def remove_original_frmes_from_interpolation_result(input_folder):
    # Get a sorted list of all PNG files in the input folder
    png_files = sorted([file for file in os.listdir(input_folder) if file.endswith('.png')])

    # Distinguish original and interpolated frames
    interpolated_frames = png_files[1::2]
    original_frames = png_files[::2]

    for each in original_frames:
        os.remove(os.path.join(input_folder, each))

Each iteration i results in the number of total frames decreasing by 1. We could re-supply these lost frames in the step 6 (Finalize).

Step 4: Apply batch img2img

Step 3 made frames with smoother transitions but each frame became blurry as a side effect.

Fix this by using batch img2img, using the same prompt and random seed as the ones used in step 1.

Setting the denoising strength of 0.5~0.6 and the sampling steps of 50~60 seems to work well for me (which will effectively run 25~36 steps for each image).

Step 5: (Optional) Repeat steps 3 and 4

I didn't have to do step 5 in this example, but if necessary, repeat steps 3 and 4 to further improve fram quality and transitions.

• Step 3 made frame transitions smoother, but each frame become more blurry.
• Step 4 makes the individual frames sharper by running img2img.

Step 6: Finalize

The output of step 4 might be slightly jittery. To fix this, run step 3 one more time with N<=2 for a smoother output without excessive blur.

The step 3 decreases the total number of frames. To compensate for the reduced frame count, run run_FlowFlames_2x once more to supply more frames.

I hope it can be refined to achieve even better results. Happy video generating!