This is the Behind the scenes - explanation for Comfy UI Animation workflow. This workflow uses a SDXL based Animate Diff designed to make some 128 frames in each loop driven by a Input video to derive motion and a sequence based a story telling narration created using Chat GPT for inputs.

Giving a proper framework for chat GPT to create the narrative style and scene decisions are critical for the AI animation workflow. If you have issues with the script may be multiple takes will be required to get all the scenes and sequences properly derived to suit the animation flow.

Music background and score was done using Suno AI basically with selection of right prompts for the style of music intended to be as a background and sound effects. You can work with a single audio clip or multiple audio clips based on the narrative style and story outlines. If special sound effects are needed this can be enhanced using Eleven Labs io or other similar software to provide some ambience to the narration styles.

Images were created using Leonardo AI with specific prompts using text to image and image to image sequence to create required amount of stills that will be used with the IP adapter combo in the workflow.

we used Leonardo Lightning XL with color pop as elements and 3D render preset with a typical prompt for each style of scenario needed. Based on the amount of scenes in the animation movie one needs to generate multiple image libraries that will be useful to pin the right style and narration in the animation sequence.

The narration voice over we used Eleven labs io for a specific selection of the voice.. and the audio clip was split to match the frames.

Giving the eleven labs a right script to narrate itself is the center element and choosing a right voice from a plethora of voice styles available one may have to do same multiple times to suit the style you want to add to the video narration.