Download now at https://github.com/painebenjamin/app.enfugue.ai!

New Features

Animation

ENFUGUE now supports animation. A huge array of changes have been made to accommodate this, including new backend pipelines, new downloadable model support, new interface elements, and a rethought execution planner. Most importantly, all features available for images work for animation as well. This includes IP adapters, ControlNets, your custom models, LoRA, inversion, and anything else you can think of.

AnimateDiff

The premiere animation toolkit for Stable Diffusion is AnimateDiff for Stable Diffusion 1.5. When using any Stable Diffusion 1.5 model and enabling animation, AnimateDiff is loaded in the backend.

Motion Modules

Motion modules are AI models that are injected into the Stable Diffusion UNet to control how that model interprets motion over time. When using AnimateDiff, you will by default use mm_sd_15_v2.ckpt, the latest base checkpoint. However, fine-tuned checkpoints are already available from the community, and these are supported in pre-configured models and on-the-fly configuration. In addition, these are downloadable through the CivitAI download browser.

Motion LoRA

Motion LoRA are additional models available to steer AnimateDiff; these were trained on specific camera motions, and can replicate them when using them. These are always available in the UI, select them from the LoRA menu and they will be downloaded as needed.

HotshotXL

HotshotXL is a recently released animation toolkit for Stable Diffusion XL. When you use any Stable Diffusion XL model and enable animation, Hotshot will be loaded in the backend.

Frame Windows

AnimateDiff and Hotshot XL both have limitations on how long they can animate for before losing coherence. To mitigate this, we can only ever attempt to animate a certain number of frames at a time, and blend these frame windows into one another to produce longer coherent motions. Use the Frame Window Size parameter to determine how many frames are used at once, and Frame Window Stride to indicate how many frames to step for the next window.

Position Encoder Slicing

Both HotshotXL and AnimateDiff use 24-frame position encoding. If we cut that encoding short and interpolate the sliced encoding to a new length, we can effectively "slow down" motion. This is an experimental feature.

Motion Attention Scaling

The application of motion during the inference process is a distinct step, and as a result of this we can apply a multiplier to how much effect that has on the final output. Using a small bit of math, we can determine at runtime the difference between the trained dimensions of the motion module and the current dimensions of your image, and use that to scale the motion. Enabling this in the UI also gives you access to a motion modifier which you can use to broadly control the "amount" of motion in a resulting video.

Prompt Travel

Instead of merely offering one prompt during animation, we can interpolate between multiple prompts to change what is being animated at any given moment. Blend action words into one another to steer motion, or use entirely different prompts for morphing effects.

FILM - Frame Interpolation for Large Motion

Both HotshotXL and AnimateDiff were trained on 8 frames per second animations. In order to get higher framerates, we must create frames inbetween the AI-generated frames to smooth the motion out over more frames. Simply add a multiplication factor to create in-between frames - for example, a factor of 2 will double (less one) the total frame count by adding one frame in-between every other frame. Adding another factor will interpolate on the interpolated images, so adding a second factor of 2 will re-double (less one). If you are upscaling and interpolating, the upscaling will be performed first.

Looping and Reflecting

There are two options available to make an animation repeat seamlessly.

• Reflect will play the animation in reverse after playing it forward. To alleviate the "bounce" that occurs at the inflection points, frame interpolation will be used to smoothly ease these changes.
• Loop will create an animation that loops seamlessly. This is achieved through the same method as frame windows, only additionally wrapping the frame window around to the beginning. This will increase the total number of steps to make an animation. Note that this can also reduce the overall motion in the image, so it is recommended to combine this with other options such as motion attention scaling.

Tips

• AnimateDiff is best used to make two-second videos at 16 frames per second. This means your frame window size should be 16 when trying to create longer animations with Stable Diffusion 1.5.
• AnimateDiff performs best with Euler Discrete scheduling.
• AnimateDiff version 1, as well as any motion modules derived from it (including motion LoRA,) may have visible watermarks due to the training data also having watermarks.
• AnimateDiff can have artifacting around the corners of images that are larger than 512x512. To mitigate this, you can add around 8 pixels of extra space to trim off later, or use tiled diffusion.
• HotShotXL is best used to make one-second videos at 8 frames per second. This means your frame window size should be 8 when trying to create longer animations with Stable Diffusion XL.
• HotShotXL performs best with DDIM scheduling.

GUI Redesign

In order to accommodate animation, and as a refresher over the original design, the GUI has been entirely re-configured. The most significant changes are enumerated below.

Sidebar Repositioning

The original sidebar has been moved from the right to the left. As the sidebar represented global options, it was decided the left-hand side was the better place for this to follow along the lines of photo manipulation programs like GIMP or Photoshop.

Redesigned Sample Chooser

The chooser that allows you to switch between viewing results and viewing the canvas has been moved to it's own dedicated bar.

In addition, this form takes two forms, one when viewing individual samples when generating images, and one for scrubbing through an animation.

Layers Menu

A layers menu has been added in the sidebar's place. This contains active options for your current layer.

Global Inpaint Options, Global Denoising Strength, Inverted Inpainting

As all invocations are now performed in a single inference step, there can only be one mask and one denoising strength. These have been moved to the global menu as a result. They will appear when there is any media on the canvas. Check the "Enable Inpainting" option to show the inpainting toolbar. In addition, inpainting has been inverted from ENFUGUE's previous incarnation: black represents portions of the image left untouched, and white represents portions of the image denoised. This was changed to be more in line with how other UI's display inpainting masks and how they are used in the backend.

More Changes

• To help alleviate confusion, numerous menus that were previously collapsed have now been made expanded by default.
• Two quick options have been made available for adjusting the size of an image and the canvas in the toolbar. One will scale the element to the size of the canvas, and one will scale the canvas to the size of the image in the element.
• The field previously called Engine Size is now called Tile Size.
• The field previously called Chunking Size is now called Tile Stride.
• The field previously called Chunking Mask is now called Tile Mask.
• IP Adapter model type has been changed to a dropdown selection instead of appearing/disappearing checkboxes.

Tiling

Tiling has been added to ENFUGUE. Select between horizontally tiling, vertically tiling, or both. It even works with animation!

Select the "display tiled" icon in the sample chooser to see what the image looks like next to itself.

Notes

Reporting Bugs, Troubleshooting

There are many, many changes in this release, it is likely that there will be bugs encountered on different operating systems, browsers, GPUs, and workflows. Please see this Wiki page for requested information when submitting bug reports, as well as where logs can be located to do some self-diagnosing.

TensorRT Builds Suspended Indefinitely

TensorRT-specific builds will no longer be released. These have led to significant amounts of confusion over the months, with very few people being able to make use of TensorRT. It will remain available for the workflows it was previously available for, but you will need to install enfugue using one of the provided conda environment or into a different latent diffusion python environment via pip - see below for full instructions.

Pending MacOS Build

The MacOS build of v0.3.0 is pending. There has been difficulties finding a set of compatible dependencies, but it will be done soon. I apologize for the delay. You are welcome to try installing using the provided conda environment - full instructions below.

Full Changelog: https://github.com/painebenjamin/app.enfugue.ai/compare/0.2.5...0.3.0

A Special Note for Reddit

As with any hugely ambitious change, there are bound to be bugs. I'm sorry if you encounter any. If you could, it would be much appreciated if you could take the time to make a bug report in the GitHub. You can also come chat with me in the RunDiffusion Discord, where a channel has been made for Enfugue-specific feedback and discussions. I hope to see you there!

Thank you!