In today’s comprehensive tutorial, we embark on an intriguing journey, crafting an animation workflow from scratch using the robust Comfy UI. Our mission is to navigate the intricacies of this remarkable tool, employing key nodes, such as Animate Diff, Control Net, and Video Helpers, to create seamlessly flicker-free animations.
You may have witnessed some of my recent YouTube shorts and Community posts, where I demonstrated the art of crafting flicker-free animations. Flickering has historically been a daunting challenge when working with Stable Diffusion for animation. However, thanks to the prowess of Animate Diff and Control Net, we now wield potent tools that ensure our animations remain beautifully smooth.
The Comfy UI Configuration:
To commence our artistic journey, we’ll launch Comfy UI, clearing the canvas to make way for our masterpiece. We’ve streamlined this process by pre-installing the essential custom nodes, and I’ve listed them here for your convenience. These nodes are effortlessly installed through the Comfy UI manager, as detailed in my previous videos.
Constructing the Workflow:
Our workflow begins to take shape with the introduction of the Animate Diff Loader, a pivotal node essential for crafting animations. Ensuring a seamless experience, the Animate Diff model is pre-downloaded and installed as per the instructions provided in previous tutorials.
Building the Foundation:
Next, we’ll lay the groundwork with context-setting options, meticulously organizing our nodes within a group to maintain order and clarity.
The Checkpoint:
Crucial to our workflow is the Checkpoint Loader, specifically the “Checkpoint Loader Sample with Noise Select.” We must connect it to our checkpoint models – for this tutorial, I’ve selected the “revAnimate” model paired with the “beta schedule select” set to “sqrt linear (Animate Diff).”
Streamlined Performance:
In this workflow, we opt for individual VAE loading instead of the VAE model bundled with the checkpoint. The MSE 84000 VAE didn’t meet our expectations, leading us to the Berry Mix VAE, tailored for animations.
Framing Our Vision:
Video footage takes center stage as we implement the “Load Video” from the VHS custom node within the Video Helpers suite. This node permits the selection of a video to orchestrate the movement in our AI-generated animations.
Setting the Stage:
It’s crucial to prepare our images for animation. The “Upscale Image” node springs into action, ready to transform video frames into the desired output dimensions and quality settings. These image frames will form the heart of our animation, seamlessly synchronized with Animate Diff.
The Power of Prompts:
Managing text prompts is essential for an artful animation experience. Employing the “Batch Prompt Schedule” node, we skillfully choreograph our animations, dictating frame numbers and corresponding actions or transitions. In this tutorial, we transition from sunlight to rain and lightning, an exquisite example of the creative potential of this tool.
Navigating Text Prompts:
Underneath the surface, a world of negative prompts exists, masterfully managed within the “Clip Text Encode.” These negative prompts coexist with our control net model, ushering in endless possibilities for animation manipulation.
The Control Net’s Control:
Central to our workflow is the “Control Net Advanced” custom node, elevating our animation quality by virtually eliminating flickering. We’ve opted for the line art model, corresponding to the Beta schedule “linear animate diff.” This pivotal node reshapes our animation’s character, smoothing out any irregularities.
Sealing the Deal:
Our control net group culminates with the “Line Art Pre-Processor,” setting the stage for pristine control over our animations. The control net’s strength is tailored to 0.5, the sweet spot for our specific use case.
Synchronizing the Symphony:
We bundle our control net group, ensuring a streamlined and efficient workspace.
The Sampler’s Touch:
The Sampler node assumes center stage, a critical final step before the grand reveal. Connecting with the positive and negative prompts, the Animate Diff Loader model takes the limelight.
Making Magic Happen:
The “K Sampler” brings us to the brink of animation magic. We set a fixed seed number, connecting the latent to the VAE Decode. Crucially, we don’t connect the checkpoint loader model here – doing so would strip our animation of its charm.
Rendering the Masterpiece:
Our animation workflow is ingeniously wrapped up with a custom node for the final output. Instead of the conventional “Save Image” node, we introduce the “Video Combine” node, enabling us to save the output in MP4 format.
A Key Tip:
For those seeking to encode the output in MP4, the prerequisite is the installation of FFmpeg. Keep in mind that the frame rate should not exceed 24, the highest setting supported by Animate Diff. For GIF output, no extra nodes are necessary.
The Grand Finale:
With the curtain raised, our animation workflow springs to life. Each frame seamlessly glides into place, and the magic unfolds frame by frame. While our workflow may appear intricate, it paves the way for a new realm of animation possibilities with Comfy UI.
A Lesson in Patience:
A true artist appreciates the journey as much as the destination. In this unedited video, we maintain the real-time experience to demonstrate the duration of such an animation process.
Navigating the terrain of AI tools, it’s imperative to stay vigilant. The Command Prompt window keeps us informed of the process details, offering a lifeline for troubleshooting and fine-tuning our creations.
Comfy UI stands as a testament to the power of artistic expression, a canvas where imagination meets AI-driven creativity. The art of crafting AI animations is no longer an enigma, thanks to this comprehensive workflow that brings visions to life.
Prepare to be captivated, for the possibilities within Comfy UI are as boundless as your creativity. The world of AI-driven animations is yours to explore and conquer, one frame at a time.
Conclusion: Unleashing the Creative Power of Comfy UI
In conclusion, our journey through the intricate world of AI-driven animations has revealed the transformative potential of Comfy UI. With a keen eye for detail and a palette of custom nodes, we’ve crafted an animation workflow that not only simplifies the process but also ensures the highest quality results.
Flicker-free animations, once an elusive goal, are now well within reach thanks to the remarkable synergy between Animate Diff and Control Net. Comfy UI has emerged as a catalyst for creative expression, promising smooth and visually stunning animations.
As we’ve explored this workflow, we’ve decoded the intricacies of various nodes and settings, optimizing every step to achieve our artistic vision. Whether you’re seeking to breathe life into characters, engage in captivating storytelling, or simply embrace the endless possibilities of AI-driven creativity, Comfy UI offers an artistic playground where imagination knows no bounds.
Throughout this journey, we’ve uncovered the importance of thoughtful prompts and precise control net configurations, ultimately guiding the narrative of our animations. With each frame meticulously rendered, we’ve witnessed the magic of AI animations unfolding in real-time, reminding us that art and technology are powerful allies in the realm of creative expression.
As we’ve persevered through the creative process, we’ve highlighted the significance of patience and vigilance. The Command Prompt window, often an overlooked companion, plays a crucial role in addressing potential hiccups and ensuring the flawless execution of our animations.
Comfy UI, as our trusty companion, has reshaped the landscape of AI-driven animations. With unwavering dedication to artistry, it has provided us with a canvas to bring our visions to life, frame by frame.
In the world of AI-driven animations, Comfy UI is our gateway to boundless creativity. It beckons artists, storytellers, and creators to explore the uncharted territories of imagination. It offers the promise of flicker-free animations and breathtaking results.
So, whether you’re crafting animations, telling stories, or simply pushing the boundaries of what’s possible with AI, Comfy UI stands as your steadfast ally. With it, the world of AI animations becomes a canvas where you wield the brush, and your creativity knows no limits. It’s a journey worth embarking on, frame by frame, as we redefine the very essence of visual storytelling through the artistry of Comfy UI.
Resources for this tutorial :
The full workflow can be find in Our Pateron (for all supporters only)
Custom Nodes You Need:
ComfyUI-Impact-Pack : https://github.com/ltdrdata/ComfyUI-Impact-Pack
ComfyUI-VideoHelperSuite : https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite
ComfyUI-AnimateDiff-Evolved : https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved
ComfyUI-Advanced-ControlNet : https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet
ComfyUI_UltimateSDUpscale : https://github.com/ssitu/ComfyUI_UltimateSDUpscale