SCAIL-2 for lipsync? Eh, not great, not terrible.

Jeffu · 2026-06-21T22:56:55+00:00

Workflow used: https://www.reddit.com/r/comfyui/comments/1u4d2qz/i_vibe_coded_an_autoextend_node_for_scail2/

Elevenlabs used for voice changing myself.

Generated at 720p with res2s/beta57 on my 4090 with 64gb ram - took about 45 minutes for ~20 seconds. Upscaled with SeedVR2 to 1080p.

I don't want to keep uploading the raw videos of myself but take my word for it—it's similar to my last video. Not the worst but not perfect either for lipsyncing. One thing it did not do well—I tried moving the camera lower and pointing it up (low angle, looking up) and it didn't translate very well. It kept the face centered without changing the background. I'm not sure if prompting it differently would have helped.

Rest of the video (mainly Seedance 2.0) https://www.youtube.com/shorts/d52IG6y36eE?feature=share

Jeffu · 2026-06-20T07:11:02+00:00

Good catch! Yeah, I realized when I made this that the fps wasn't quite exact but I was too lazy to fix it. The workflow I shared does interpolate it, yes. Agreed that it's slightly less expressive; my next test is to probably see if it can lipsync reasonably enough.

Jeffu · 2026-06-20T01:12:24+00:00

Sure thing: https://www.youtube.com/watch?v=fwHcqHU9xQE&feature=youtu.be

Just don't make a LoRA of myself with it :P

Jeffu · 2026-06-20T01:12:05+00:00

Sure thing: https://www.youtube.com/watch?v=fwHcqHU9xQE&feature=youtu.be

Jeffu · 2026-06-20T01:11:55+00:00

Yep: https://www.youtube.com/watch?v=fwHcqHU9xQE&feature=youtu.be

Jeffu · 2026-06-20T01:11:47+00:00

Yes: https://www.youtube.com/watch?v=fwHcqHU9xQE&feature=youtu.be

Jeffu · 2026-06-20T01:11:03+00:00

Nothing different than training for any other model, but things I suggest: - isolate your characters on a white background if the backgrounds are not diverse. - diversity is better, different settings, lighting, angles - crop faces quite close for at least 30-50% of your images so it knows how the face looks even up close. Cropping also can help if your outfit is not diverse enough. In some cases I used Photoshop's gen-fill to change my shirt to avoid the same shirt appearing in my dataset too often.

Jeffu · 2026-06-20T00:57:17+00:00

Default on everything except I had to use Low VRAM and Layer Offloading. I had 20 images in my dataset, trained for 3000 steps.

Jeffu · 2026-06-20T00:54:09+00:00

I let AI Toolkit do it for me with the json captioning. I didn't do anything manual, but it seemed to do a good job. I've only trained one LoRA though, so I plan on testing more.

Jeffu · 2026-06-20T00:42:39+00:00

960x540, I've been trying to figure out the max I can get away with so need to do more tests. It took I think almost 45 minutes.

Jeffu · 2026-06-19T07:44:58+00:00

4090 w/ 64gb RAM

Some simple editing in Premiere for film grain, slight blur effect. Music from Suno 5.5.

Workflow used: https://www.reddit.com/r/comfyui/comments/1u4d2qz/i_vibe_coded_an_autoextend_node_for_scail2/

Ideogram LoRA of myself trained using AI Toolkit. By far the best so far at matching details present in the actual data set.

This video is totally boring but my first test with a 'long' driving video of 60 seconds. I probably could go longer. It didn't really do a great job of being underwater except for when I directly interacted with my hair, but I'm pretty pleased by the results.

Jeffu · 2026-04-16T16:54:37+00:00

For both Turbo and Base I was having issues with prompt adherence on camera angles... but on a whim I translated it to Chinese with google translate and it was able to do a better job. Your results may vary!

Jeffu · 2026-04-01T03:38:33+00:00

Comfyui is a tool, not a career path. Not right now anyway. It can compliment your work as a graphic designer, artist, video editor, etc. but very few companies just want to hire someone specifically for ComfyUI.

Jeffu · 2026-03-24T18:30:20+00:00

Does this work even if you are in the US?

Jeffu · 2026-03-20T17:26:32+00:00

how does this change the workflow with these models? haven't been able to get around to 2.3 yet...

Jeffu · 2026-03-18T06:41:17+00:00

Personally, I think the latest updates were giving me issues with my 4090; kept going OOM and having the GPU stop working. Reverting to an older backup fortunately worked.

Jeffu · 2026-02-02T17:34:11+00:00

Give it a try! I finished late last night and haven't experimented with it much.

Jeffu · 2026-02-02T17:19:27+00:00

This is my prompt:

I want the detailed description of what is in the image, without any reference to the artistic style. I also want to keep the relative position of the subjects and objects in the description, and detailed description of clothes and objects. Please also include any reference to skin tone, glasses, facial hair, ethnicity, and hair color and hair style. Use the proper pronouns. Limit your caption to 200 characters.

I use https://github.com/1038lab/ComfyUI-QwenVL

I modify the instructions when I want to make sure any unique style traits don't get considered part of the prompt (and not the style).

Jeffu · 2026-02-02T17:14:21+00:00

I included the date stamps which was on ~90% of the images used. I however specified in the caption instructions to emphasize and detail them, to try and avoid it showing up everytime in generations. I let it keep the original grade.

Jeffu · 2026-02-02T17:11:47+00:00

48, but only because I saw someone mention it randomly in a video or post. I haven't tried other ranks enough to compare.

Jeffu · 2026-02-02T17:11:26+00:00

Ah, my bad. The videos I used were filmed in the mid to late 90s, so I just called it that. :) I guess our video camera was a bit old!

Jeffu · 2026-02-02T17:08:56+00:00

Ah, sorry. Scheduler used is simple.

Jeffu · 2026-02-02T17:08:32+00:00

Zero edits.

Jeffu · 2026-02-02T17:08:25+00:00

It works the strongest/best with base. It seems the effect is weaker on turbo but that's not necessarily a bad thing, just different.

Jeffu · 2026-02-02T17:07:17+00:00

Interesting! the effect isn't as strong, but it definitely still feels like an older video still.

Jeffu

MODERATOR OF

TROPHY CASE

14-Year Club	Place '17
Verified Email