Automatic1111 Animatediff Controlnet - How to make AI animations Stable Diffusion

To make incredible AI animations, combine Animatediff and ControlNet. This video covers the installation process as well as some easy little tricks that can produce some really cool results.
Project Resources + Prompts: goshnii.gumroad.com/
Animatediff for Beginners: • AnimateDiff and (Autom...
Animatediff Models: huggingface.co/guoyww/animate...
ControlNet Models: huggingface.co/webui/ControlN...
Helloypund2d on Civitai: civitai.com/models/134442/hel...
Animatediff Tutorials: • Automatic 1111 Tutorials
Installing Deforum and ControlNet STEP-BY-STEP (Automatic 1111): • Installing Deforum and...
#animatediff #aianimation #gifs #stablediffusion #text2video

[@pedro_a_martins]

This is awesome! Thank you so much!

[@goshniiAI]

You are most welcome, and I appreciate your kind words!

[@serinadelmar6012]

Same, Thank you! ❤

[@goshniiAI]

@@serinadelmar6012 I'm glad to hear from you.♥

[@DaltOniXProductions]

This was a very helpful reminder thanks! I need some coffee lol

[@goshniiAI]

Lol...I'm grateful that you stopped by and glad that was useful

[@aiximagination]

Greate video. Thank you!

[@goshniiAI]

I appreciate your kind words very much!

[@T.L-TV]

thank you for this tutorial

[@goshniiAI]

Thank you for the feedback, and you're welcome.

[@romeofthesouth]

This is excellent, I've been trying to get consistency for a while now. I will try this later. I have Forge so hope it renders faster than 8 hours 😭

[@goshniiAI]

I'm glad you found the video helpful! Using Forge can speed up the process compared to an 8-hour render, which can be exhausting! lol

[@apidyahex9213]

Just waiting for checkpoints to download to give it a go, but so far this is the best video I've seen that teaches animation in stable diff. keep them coming you are awesome

[@goshniiAI]

Happy animating! Your words of encouragement mean a lot, and I am glad that I could assist you. Thank you for your feedback

[@apidyahex9213]

@@goshniiAI I'm yet to try it fully but honestly your description and steps with links and highlighting are so easy to follow. I'm coming to your channel first for all my info and tutorials now! . thanks for spending time making the video.

[@goshniiAI]

@@apidyahex9213 It's nice to know everything connects with you and is simple to follow. thank you for tunning in again

[@apidyahex9213]

finally got round to trying and got this error....EinopsError: Error while processing rearrange-reduction pattern "(b f) d c -> (b d) f c". Input tensor shape: torch.Size([2, 6144, 320]). Additional info: {'f': 16}. Shape mismatch, can't divide axis of length 2 in chunks of 16. any ideas? thanks@@goshniiAI

[@apidyahex9213]

tried everything and after that error even text to image gen doesn't work unless i restart stable diff thanks for the vid but no luck with any animation for me still .

[@MisterCozyMelodies]

thanks, again!!

[@goshniiAI]

I'm glad to help, and I appreciate your comments.

[@olafguerrero740]

Thank you for creating this tutorial. Quick question did you say it took 8 hours to see the final results? Can you recommend some options to create lower rez examples to make sure controlnet is using the reference folder first before you do your final output?

[@goshniiAI]

You are welcome! Yes, the final high-resolution output took approximately 8 hours. To create lower-resolution examples to ensure Controlnet is properly using the reference folder before the final render, try reducing the image size or rendering a shorter sequence first. This allows you to quickly verify the process and make changes as needed before committing to the final render. I hope this helps!

[@ArtWreckAI]

thank you for this tutorial. BTW the PNG info gave me "parameters: none"

[@goshniiAI]

You're welcome, and I appreciate your feedback. If the PNG info shows "parameters: none," you might want to verify your settings. Or you can find the image that was used in the project resources

[@Joe-ce6cc]

i have a question, why do you need to import the png sequence batch ? since you already got the video as reference for openpose ?

[@goshniiAI]

Hello, the video reference was required for Animatediff, while Batch Frames guided controlnet. I didn't use the video for Controlnet because the image sequence helped me save GPU processing time. However, there could always be multiple ways to achieve the same result.

[@ChaseEverything]

Hi for some reason I got to the stage where you hit the explosion and then generate and it didn't generate with the pose even though I can see the mask for it. :(

[@goshniiAI]

You are correct, there is a problem with controlnet and animatediff merging in A1111, which I also encountered. I'll discuss the solution in an upcoming upload in order to get this working. kzbin.info/www/bejne/pJ3OZ2eYeq53a5Y

[@ucphamvan1711]

Hello, may I ask how did you write such a long reminder and still make it work? When I write a reminder longer than 75 words it gives me the error RuntimeError: CUDA error: device-side assert triggered.

[@goshniiAI]

Hello there, It's probably caused by memory limits on your GPU, especially if processing is done with CUDA. To use less memory, you could divide your reminders into smaller sections or make your output size a little smaller.

[@ucphamvan1711]

@@goshniiAI i using 3090ti 24g vram. Can you help me optimize?

[@goshniiAI]

@@ucphamvan1711 I believe you are well-equipped for high performance. You have a powerful GPU. I will not recommend overclocking, as it usually has negative long-term effects on any GPU. Consider using a lower resolution for beginning drafts and then upscale the final for animations. Also Keep track of your VRAM usage while rendering. If it reaches your 24-GB limit, you may need to adjust the settings or consider using cloud resources with more VRAM for complex projects.

[@ucphamvan1711]

@@goshniiAI Thank you, I try again

[@barisvkabalak]

when I use animatediff, it batches 32 different images. How can I fix this?

[@goshniiAI]

At times AnimateDiff performs random animations. Kindly check the settings to ensure you're specifying the correct processing inputs, as well as the model you're using for AnimDiff's generations.

[@villevase]

Hi, Can you tell me why AnimateDiff is changing the photo on half way? Thanks!

[@goshniiAI]

Thank you for reaching out if anything was unclear along the way. To clarify, the technique was not to achieve similarity with the reference image, but rather to use the open pose motion from the reference to direct the animation And I agree that Animatediff sometimes creates surprises and uncertainties in the process. Hopefully, based on my recent research, I'll be able to upload a process to get around this soon.

[@villevase]

Thanks for clarifying. Great tutorial!@@goshniiAI

[@goshniiAI]

@@villevase You are very welcome! Happy creating.

[@RankingIsHell]

I believe that because he is using controlnet the image doesn't change halfway because Controlnet takes over but if you aren't using Controlnet then try not going over 75 characters. It has worked for me as long as you don't go over 75 characters as I had the same problem too. Others have said on reddit the same thing.

[@goshniiAI]

I appreciate all of the extra information and practical advice. It'll be useful to a lot of us. Let's keep exploring and discovering together! @@RankingIsHell

[@ChaseEverything]

I was working on it but then I got this EinopsError: Error while processing rearrange-reduction pattern "(b f) d c -> (b d) f c". Input tensor shape: torch.Size([2, 6144, 320]). Additional info: {'f': 16}. Shape mismatch, can't divide axis of length 2 in chunks of 16

[@goshniiAI]

The error could be a problem with the dimensions not being divisible by 16. You may need to reshape or edit your input data just to make sure that it matches the intended format.

[@dragongaiden1992]

Friend, what graphics to use, I can't use hires?

[@goshniiAI]

Hey friend, a GPU with at least 8GB of VRAM, though 12GB or more is better. Some common options include the NVIDIA RTX 3060, 3070, or 3080 series.

[@ChaseEverything]

@@goshniiAI Hi there brother, what GPU did you use for this video and what speeds were you getting for Generation?

[@Postfxx]

hi its been 3 days i am trying this animate diff + controlnet unit is not working in batch. but if i disable animate diff control net is working fine* please help me with be very much thank ful.

[@goshniiAI]

I'm sorry to hear that, it must be frustrating. 1. I recommend carefully reviewing your parameters before generating your propmt 2. Run a stable diffusion update for A1111. 3. Check A1111's extension updates. 4. Check the control net settings under Automatic 1111 Settings. 5. Please check for FFMPEG if you do not already have it installed. Sometimes, adjusting a few parameters can make all the difference. I hope any of these are useful.

[@Postfxx]

thank u for ur reply.. really appriciate your feedback and your work.checking with that only hoping to get it correct.

[@goshniiAI]

You are welcome, For reference, these are my controlnet settings. tinyurl.com/2rjynv3b @@Postfxx

[@Postfxx]

@@goshniiAI how to comment send screenshot on this.. its getting deleted. link getting deleted...

[@Postfxx]

please help.. really need this to work..

[@HopsinThaGoat]

your honest opinion does this replace deforum

[@goshniiAI]

That's a great question. However, in my opinion, the decision comes down to our individual targets since both have positive benefits. The most important thing is to choose the tool that best fits our project goals.