We’re delving into the multifaceted capabilities of Stable Diffusion Automatic 11 11. We’ll explore the endless creative possibilities it offers through extensions like AnimateDiff, ReActor Faceswap, and ControlNet – IPAdapter model.
These incredible features have become integral to the animation landscape, providing a diverse range of applications for both seasoned creators and those just beginning their animation adventures.
Easing into Animation without Text Prompts
Stick around until the end, and you’ll see how to create animation videos with ease, all without the need for typing any text prompts in Automatic 11 11.
Let’s get started!
The Animate Diff Extension
At the beginning of this tutorial, we’ll start with the Animate Diff Extension. I’ll provide all the extension download links below:
- AnimateDiff for AUTOMATIC1111 Stable Diffusion WebUI
- Integrating AnimateDiff w/ CLI into AUTOMATIC1111 Stable Diffusion WebUI w/ ControlNet
- ReActor Faceswap
- IP-Adapter With Colab
In this AI-generated animation, it all begins with Text to Image results, and we use these generated images in the Image-to-Image tab to create short motion scenes. For this particular animation, I used a Cyberpunk style featuring two girls on the street.
A Smooth Workflow
To do this, you can select the image result in the Text 2 Image tab. Choose one of the images you’d like to work with and click “Send to Image-to-image tab.” There you go; the image appears in the Image-to-Image tab.
Now you can start experimenting with the Animated Diff extension. The initial frame will use this image, and no text prompt is needed. I’m going to set the width to 512 and height to 768 pixels. Ensure all settings are correctly configured, as other settings depend on your desired outcome.
In this case, I’ve set the denoising strength higher. Scroll down, and you’ll find the Animate Diff Extension. Click “Enable” and set the number of frames you want for the output result. Then click the “Generate” button.
Effortless Animation
As you can see in the Command Prompt window, the Animate Diff Extension is working on the GIF animation by merging images for each frame. Interestingly, in this animation, I applied the Zoom Out motion lora, but it’s only affecting the background.
In the second experiment, we’ll try using the “Travel Prompt in Animation.” As demonstrated in the GitHub example, it allows us to control the movement of a character’s mouth.
Smooth Character Movements
Let’s give it a shot. In the text prompt area, it’s straightforward to set up each animation on a new line, specifying the target frame number and the movement description. As you can see, the giant robotic character’s mouth moves in this animation result.
Here’s another example involving moving elements within an image. This time, I’m using an image with no characters, only a city view with an apocalyptic style. We’ll see how this extension creates animation for this type of image. We’re using Animate Frames 20 for this one, and all settings seem to be in order. Let’s click “Generate.” It’s excellent! I like how it incorporates fire and mist to create motion from this image.
Let’s try this building. Great! It truly conveys an apocalyptic feel. I know some people have asked about using AnimatedDiff to create story videos on YouTube. So here’s your starting point; I could create an apocalyptic story with this. Many channels talk about using various tools and logging into multiple free-trial plan websites just to create one animation. But for us Stable Diffusion users, we can do it all in one place! Plus, we have the freedom to create as many animations as we want.
ControlNet V2V
Moving on to the next feature of Animate Diff, we can use ControlNet V2V. Essentially, this means we can reference an image or video frames for animation control. For instance, you can make a character’s movements based on a source video.
In this example, I’m using a stock video footage of a boxer’s punching movement as the animation ControlNet video source. Simply drag and drop the video clip onto the video source panel; there’s no need to set the number of frames manually, as the extension will detect the number of frames in this video source.
In the ControlNet, I’m using “DW Open Pose – ControlNet Is More Important” option, IP Adapter, and “Lineart – ControlNet Balance” option. The result, as you can see, isn’t a frame-by-frame copy from the source video. It changes the style into more of a 3D animation, including the character’s face changes.
Enhancing Faces with ReActor Faceswap
So how can we maintain the character’s face as we want it and generate it with more of a 3D animated style? This time, we’ll include the ReActor Faceswap extension in the process. I’ll use a short clip of a Korean girl as an example.
For the ControlNet, we’ll use the same three models enabled, with no changes from the previous generation process. We’ll scroll down the page and enable ReActor, then drag and drop a face image onto the source image panel. In this case, I’m using an AI girl, Nancy’s image as the source.
Then, click “Gender detection: Female only” on both source and target. Everything seems to be in order. Click “Generate.”
The Final Result
After some time, I came back to check the Command Prompt, and it’s all done. The Faceswap Extension was in the process, and then Animate Diff merged all the generated frames into a single animation. There you go, pretty good this time. The character’s movement looks smooth. There’s one frame where the clothing color changed, but that’s not a concern; we did all this using only Stable Diffusion without other video editing tools. And we created a 7-second GIF animation. No complaints here, unlike some other tools. Ha!
Enjoy the Creativity
Let’s click it to view it in a larger size. Nice! I like how it captures the sunset light, even though the character keeps moving.
That’s Animate Diff running with ControlNET IP Adapter and ReActor Faceswap. You can check out all the generated animations at the end of this video. I’ll see you in the next one. Have a great day!