Easy Image to Video with AnimateDiff (in ComfyUI)

Easily add some life to pictures and images with this Tutorial. The Magic trio: AnimateDiff, IP Adapter and ControlNet. Explore the use of CN Tile and Sparse Control Scriblle, using AnimateLCM for fast generation.
💻 AnimateLCM: tinyurl.com/y4...
🕸️ Discord: / discord
#animatediff #comfyui #stablediffusion
============================================================
💪 Support this channel with a Super Thanks or a ko-fi! ko-fi.com/koal...
☕ Amazing ComfyUI workflows: tinyurl.com/y9...
🚨 Use Runpod and access powerful GPUs for best ComfyUI experience at a fraction of the price. tinyurl.com/58... 🤗
☁️ Starting in ComfyUI? Run it on the cloud without installation, very easy! ☁️
👉 RunDiffusion: tinyurl.com/yp... 👉15% off first month with code 'koala15'
👉 ThinkDiffusion: tinyurl.com/4n...
🤑🤑🤑 FREE! Check my runnable workflows in OpenArt.ai: tinyurl.com/3j...
============================================================
CREDITS
===========================================================
🎵Music
Song: Bike Box, Artist: Kia, Music by: CreatorMix.com
It's Our Time, Music by CreatorMix.com
✂️ Edited with Canva, and ClipChamp. I record the stuff in powerpoint.
========================================================
© 2024 Koala Nation
#comfyui #animatediff #stablediffusion

[@koalanation]

If you prefer it without the ai voice, check out with original voice (in Spanish) kzbin.info/www/bejne/faSyhaF9mr12eck

[@tech3653]

any toturoal for easy translation of voice offline using ai ?

[@Kratos30000]

Which GPU do you need for these kind of animations?

[@DanielThiele]

honestly, its really good for ai voice. no hablo espanol senior. muchas gracias. :D

[@koalanation]

@@DanielThiele 🤣🤣🤣

[@chrisfletcher2646]

I've been frustrated with AI voiceovers for a lot of stuff i watch, but now that I know you did this as a non-native language it makes sense and I'm grateful as well. Muchas muchas gracias!!!!!!!

[@SemorezX]

awesome works , thank u so much

[@SeanOdyssey]

Thankyow

[@user-Cyrine]

Love your videos so much! Can you make a tutorial video on FlexClip’s AI tools? Really looking forward to that!

[@koalanation]

Thanks for the idea!

[@LuigiEspositoGraphic]

it works well, but the details are extremely lower than the original image, how can I fix it?

[@koalanation]

You may want to increase the tile settings, but yeah, the method do changes things. Try also to use a checkpoint corresponding to the style of the original image (realistic, cartoon, anime...), to get better adherence. Those are some ideas....obviously, image upscaling or a second AD pass may also help

[@YING180]

thank you for your video, that's very helpful

[@VanessaSmith-Vain88]

Can you set up the whole thing for us to use it?

[@lildrill]

🤣😂

[@VanessaSmith-Vain88]

Yeah, that was really easy, piece of a cake 🤣

[@koalanation]

Yes, it was 🤣

[@boo3321]

very easy tutorial only took me HOURS to do it, I'm curious how to make people walk or move with comfyui.

[@koalanation]

Well, I cut quite a bit to show only the main steps, otherwise the video is mostly rendering... For moving people, controlNet with a reference video of someone walking is probably the way. With motion director should also be possible, I believe, but I need to find the time to try and see if results are 👍

[@ristom1]

Kickass video man!!! Im trying to learn cool AI like this for music viduals this is 10/10 cool. Gonna also do blender renders as bases and use ai to make them trippy. Have any tutorials for video to video?

[@koalanation]

Check out the morphing and audioreactive videos. Using masks is more complicated but gives you more power to play with

[@ristom1]

@@koalanation thank you!!!

[@user-cb4jx8og2k]

great video you skipped some steps but still detailed. Question, do we not need to change the text prompt for each randomized pic > Also why did you use Load video path node for an image ?

[@koalanation]

Hi! In principle you do not need to change it, but you can, of course. Take into account the 'tile' control net is rather strong and you cannot do big transformations. The Load video node allows you to use http addresses, but the Image load node not (at least did not work for me). That is why I use it for the randomized image.

[@estebanmoraga3126]

Thans for the tutorial! Question: Is it possible to feed Comfy with a reference video for it to animate the image using said video as reference? Like, say I have an image of a character, and I give Comfy a video of someone skateboarding, is there a method with which I could get comfy to animate the character skateboarding based on the video? Cheers and thanks in advance!

[@koalanation]

Yes! You can use a reference video and use controlnets such as openpose, depth, lineart, etc, to guide the composition of each frame. There are many videos and tutorials about it.

[@estebanmoraga3126]

@@koalanation Thanks for replying! The most I've been able to find are tutorials on animating a referenced image using prompts or generating a video using another video as reference also using prompts, have yet to find one where they animate a reference image based on a reference video, guess I just have to look harder tho!

[@koalanation]

Check out: kzbin.info/www/bejne/joCYloGAZr1lqKs. Take into account this is rather complex with all the samplers and so on. Here: kzbin.info/www/bejne/gZKXdoGaa5iJeNE, I think it is more clear, but take into account the IP Adapter node it does not work like in the video anymore.

[@policani]

Sparse Control Scribble is also difficult to search for. I have no results for all three words, and three results for Control Scribble.

[@koalanation]

The models are here: huggingface.co/guoyww/animatediff/tree/main

[@marcdevinci893]

I carefully followed and really want to get this going but getting a KSampler error. 'Given groups=1, weight of size [320, 5, 3, 3], expected input[32, 4, 96, 64] to have 5 channels, but got 4 channels instead'

[@koalanation]

Try to change or bypass, one by one, the controlnet and animatediff nodes and see if the workflow runs.

[@hamster_poodle]

hello! Does SparseControl work with AnimateDiff LCM properly? not V3?

[@koalanation]

Hi! With the V3 lora adapter works. I am not sure if that is the way it was intended, but it does something. I have tried to use the RGB sparse but I do not manage to get it work nicely...you can also switch to version 3 and fine tune results, but obviously generations will take longer

[@rachinc]

4:54 I got to this part of the tutorial, my workflow is at 88% ksampler and then the word "reconnecting" came over the screen. terminal: [AnimateDiffEvo] - INFO - Using motion module AnimateLCM_sd15_t2v.ckpt:v2. Unloading models for lowram load. UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' (I have a 128gb ram computer)

[@koalanation]

Hi! That looks a bit odd having 128 gb and stopping at 88%..sometimes comfyui crashes when the CPU is overloaded...try to test with less frames or smaller ones to see if that is the case.

[@rachinc]

@@koalanation when a video loader box is present you can go to Select_every_nth = if you put 1 its going to generate every frame of the video. If you choose 2 its going to generate every other frame of the video.... Since you dont have that box, what is the equivalent in your workflow?

[@rachinc]

@@koalanation i tried 16 frames and all it gave me was an all black box

[@dschony]

It was a little problematic to install all this modules and nodes. The webui crashed and I had to update it, recover the venv and also to reinstall comfyui's dependencies ... it took hours. Nothing for newbies.

[@dschony]

Ok. Found out, that compared to the time it takes for generation, this little time to fix the environment is nothing. But I like the tutorial ;)

[@user-bc9fe4hq2b]

​@@dschony Hey) Which GPU do you use and how much time does it take to generate 1-2 sec video?

[@dschony]

@@user-bc9fe4hq2b GPU: NVIDIA GeForce RTX 3060 / 8GB VRAM. It takes about 1 hour for 2 sec with a frame rate of 30, or 1 min per picture. It depends of the used models and nodes, of the settings (steps) and more.

[@dschony]

Well, I found, that it's better not to use the stable diffusion WebUI with the ComfyUI extension, but to use a separate standalone installaion of ComfyUI with it's own environment.

[@koalanation]

Good you could find a work around. Sometimes all custom nodes and models can be tricky in comfyui with all updates and constant changes. Thanks for providing such good advice to others!

[@Shahriar.H]

ModelPatcherAndInjector.patch_model() got an unexpected keyword argument 'lowvram_model_memory' I'm getting the above error on the KSampler before VAE encode node. How do I fix this? Edit: I'm using Stability Matrix to run ComfyUI if that's a relevant information.

[@koalanation]

Hi, There was an issue raised in the AD github, they said it should have been fixed. Try to update both the AnimateDiff Evolved nodes and ComfyUI. I do not know how is that done in Stability Matrix...in ComfyUI, normally I do it via the Manager.

[@HOT4C1DR41N]

I couldn't make it work :( I get this error every time: Error occurred when executing ADE_ApplyAnimateDiffModel: 'MotionModelPatcher' object has no attribute 'model_keys'

[@koalanation]

Seems odd...are you using AnimateLCM_t2v? Maybe try with other model to see if it runs, or use the gen 1 AnimateDiff Loader

[@katonbunshin5935]

I have the same

[@koalanation]

Use the model at: civitai.com/models/452153/animatelcm and make sure nodes and comfy is up to date.

[@katonbunshin5935]

@@koalanation oh... i wrote solution here but i dont know why is it not added... so... in my situation, there was problem when i was updating AnimateDiff from Manager. To fix it remove AnimateDiff from custom nodes and get AnimateDiff from repo, then place it in Custom nodes - works for me

[@koalanation]

Ok! I have not see it neither...anyway, sometimes during updates this things happen

[@jaydenvincent2007]

when I click Queue Prompt is says "TypeError: this is undefined" and nothing happen. I have all required nodes/models, and comfyui updated/restarted. can you please help?

[@koalanation]

Hi! I have never encountered this error...googling it refers to an issue with MixLab nodes...not sure if that would be your case. Maybe try to disable or uninstall custom nodes to see if there is one affecting ComfyUI.

[@bordignonjunior]

Geeez this takes long to run. which gpu do you have? amazing tutorial !!!

[@koalanation]

Hi! Thanks! I am using a RTX4090/3090 or A5000 via Runpod, which generates the video rather fast. You can try to decrease the number of frames and also the resolution of the images. Try to do interpolation with 3 frames instead of 2.

[@kargulo]

I have 4060 with 16G and 50 % is taking 15 min , that is first creating , I hope next one will be faster :)

[@SiverStrkeO1]

Great video! I'm new to all that and im wondering of there is a way to keep the details. I'm trying to use a city skyline as img to video, and there for example, a lot of windows are getting removed.

[@koalanation]

That seems difficult with this method If the windows are small. Reducing the scale factor may work. Otherwise some trick with masks and controlnets may work but I have not really tried it with sparsectrl

[@doctorrisk]

Is it possible to input 2 pictures and have AI make a video transitioning from one to the other?

[@koalanation]

Yes, you can transition using masks. Check out my morphing and audioreactive videos to get an idea

[@MarcusBankz68]

I'm getting an error with IPAdapterUnifiedLoader, says clipvision model not found. I've downloaded a few versions and put them in my clip_vision folder but still getting the error. Is there a specific one for this node?

[@koalanation]

Sometimes with IP adapter is confusing...try to use the IP adapter model and clipvision separately (without using the unified loader) following the instructions of the IP adapter repo. I like plus and VIT-G. github.com/cubiq/ComfyUI_IPAdapter_plus?tab=readme-ov-file

[@frankliamanass9948]

It all worked and animates the image but every time it comes out very bright and faded. Any suggestion on how to fix it?

[@frankliamanass9948]

It appears the results in the tutorial are also faded and over brightened but at the end when you show examples they look fine. Did you find a fix or was it in your post processing?

[@koalanation]

Depending on the source of image, settings, etc, the image might be too dark or too bright, as you say. There are nodes that do that. I like Image Filter adjustments. But I think it is better to use a regular video editor, it is faster and easier to use.

[@elifmiami]

I was wondering how did you bring node number on the box ?

[@koalanation]

If you go to the Manager, on the left column you will see the option 'Badge'. There you can set the number of the node to appear over the node.

[@elifmiami]

@@koalanation thank you !

[@rachinc]

4:53 those blur effects you put over your video make it hard to see what you're doing.

[@koalanation]

Yep, I got too enthusiastic with the video effects when editing the video...promise not to overdo next time

[@vl7823]

hey i'm getting this error "Could not allocate tensor with 828375040 bytes. There is not enough GPU video memory available!" I have an AMD Rx6800Xt 16gb vram, any workaround or fix? Thanks

[@koalanation]

Hey! Not sure what the messages are with AMD, but maybe you can try first reducing the size of the latents and/or reducing the batch size. Looks like some limitation with the VRAM.

[@user-bc9fe4hq2b]

hey! Were you able to fix and launch it?

[@joonienyc]

hey buddy , how did u copy second Ksampler with all lines connected duplicated ? at time line 4:40

[@koalanation]

Copy normally with ctrl+c, then paste with ctrl+shift+v.

[@joonienyc]

@@koalanation ty my man

[@kizentheslayer]

where do i save teh animate lcm model to?

[@koalanation]

models/animatediff_models

[@ManuelViedo]

"easy"

[@koalanation]

🤣🤣🤣

[@generalawareness101]

yeah, no to SD1.5 anything.

[@AB-wf8ek]

Did SD1.5 hurt your feelings?