ComfyUI AnimateDiff Prompt Travel: Runpod.io Cloud GPUs Tutorial

This video is a complete start to finish guide on getting ComfyUI setup with the addition of the ComfyUI-Manager and AnimateDiff with Prompt Travel on runpod.io
I am not sponsored by runpod.io at all. Here is the referral link associated with my account:
runpod.io?ref=liuedypr
The repos used are located here:
github.com/comfyanonymous/Com...
github.com/ltdrdata/ComfyUI-M...
The workflow file can be downloaded from here:
drive.google.com/file/d/1bG9-...
The model (checkpoint) downloaded for this tutorial series is here:
civitai.com/models/83867?mode...
The VAE used can be downloaded from:
huggingface.co/stabilityai/sd...
The motion_modules can be found on here:
civitai.com/models/108836/ani...
Socials:
x.com/c0nsumption_
/ consumeem
/ c0nsumption_

[@elowine]

Just a heads up. The ComfyUI used by Runpod seems a bit outdated and it (for me) didn't recognize the Manager installation. Executing "git pull" inside the ComfyUI folder fixed this for me.

[@8561]

I literally just spent 2 hours trying to figure out what the problem was! It looked like a bug on the frontend with Comfy but ofc updating did the trick.

[@c0nsumption]

I mention it at some point in the vid! :) Thanks for dropping the comment. I will be updating this tutorial very very soon

[@AlphaBotCalls]

this comment saved my ass

[@8561]

Another heads up! Not sure who needs to hear this (probably a lot of people) but here's what you need to do in runpod if insightface (via IPAdapter face models) isn't working: In your virtual environment, run apt-get update apt-get install -y build-essential pip install onnxruntime onnxruntime-gpu insightface This comment will save asses for sure @@AlphaBotCalls

[@aisestudio]

waiting, it was working a moth ago how in this tutorial, but not now? sadness @@c0nsumption

[@KennyGarland]

Enjoy all your vids, clearly explains what to do and helps get things up and running.

[@elowine]

Many thanks for this! 🥰 I managed to get a pod to work last week by combining a few workflows but that was such a mess. This workflow is so much cleaner and doesn't run into all the low disk errors I got.

[@c0nsumption]

Awesome. Happy to help 👏🏽

[@luclaura1308]

Wonderful tutorial, thank you!

[@samwalker4442]

my bro, that was awesome....thanks man 🙌

[@seancondev3321]

Brilliant, have been thinking about this

[@aisestudio]

Thank you for this information!

[@zaneaustin22]

Thanks for the fantastic video

[@petermolvaier]

wow, you are amazing, i fallowed the instructions and it is working; i aprecite a lot your energy and time you spend sharing al this infos; for people without programming background all this things looks like Chinese. cheers

[@c0nsumption]

Thanks dude 🙏🏽 Going to be making some more Runpod tutorials to show you guys the level of versatility you actually have there. Considering doing live streams 🤔

[@petermolvaier]

@@c0nsumption It would be grate to have live streams; i know you mentioned at a point in your video that you are thinking of making a channel on discord. If you have time, i encourage you to do it cause you are starting to build a community.

[@blockchaindomain2226]

dude you're the shit! I Dont think ill be renting GPUs but I love all your videos!

[@TheDreamingMachineFilms]

Amazing! Thank you so much!

[@yuxiwan8917]

Just wonderful!

[@findingforeverness8569]

much appreciated!!

[@EpochEmerge]

Thank you so much! Saw your premiere and instantly prepaid for runpod 😅This is pure gold. Is there a way to automatically test new prompts with different models? Now i manually form a queue, but I’d like to make a list of prompts and a list of models, basically XYZ for nodes

[@c0nsumption]

Give me a couple days and I’ll drop a video on this. Bit of a loaded question

[@JimDiMeo]

Yo this is an amazing video... one small ( HUGE ) thing.... it's much better to purchase Network Storage and do everything you just did... THAT way you can move around to different GPUs on the fly and bring all that configuration, checkpoints, models, etc with you.... If you do not use Network Storage then you will have to go through your entire process in this video each time you want to purchase a new GPU...Just a heads up man... keep the videos coming. Network Storage costs you like ($0.07/GB/Month, $0.05/GB/Month for 1TB or more) when you are not using a GPU pod..

[@c0nsumption]

Awesome. Better to keep it simple though. Not many technical savvy people jumping in. Trying to get as many people possible using this new tech so I have to give bite sized chunks or else they get scared. Even having the video be 20 min is a huge deterrent lol. I will keep this in mind though

[@BusyAi]

Totally understand... it just an additional 2 steps in the beginning.. saves you loads of time later... @@c0nsumption

[@koalanation]

Can you use network storage with community cloud?

[@Gorto68]

@@c0nsumption Great tutorial -- novice with Runpod, but would love to know how to save the workspace before terminating the pod. Using community GPUs.

[@elihusolano5993]

a bit late but yes, I have been doing that since sunday@@koalanation

[@openstorystudio]

An amazing instructional video! Question - how do you load your own images, videos .... from the desktop to the pod

[@c0nsumption]

Thanks 🙏🏽 You can just drag and drop them into the Jupyter file browser where you want them stored. I’d recommend making an input folder in the same directory that I had you make your venv. So the directory holding it would look like: -ComfyUI -venv -inputs

[@piclezwd]

Great video thanks! Question, why in the video you mentioned to not open (click) on the port 3000?!

[@HistoryIsAbsurd]

thanks alot eh well worth the sub

[@veryA72]

First time i tried this , i made a half-ass-follow-thru .... and my ComfyUI-Manager did not work as it should. Awesome tutorial

[@devoiddesign]

Quick question related to Runpod. I have been only using this method to connect with my runpod. THANK YOU! I wanted to share the static url they provide with a colleague but the Runpod will not keep or install VHS_VideoCombine... Any thoughts?

[@Mr.Dude2two]

Amazing video! Would love more simple runpod tutorials like this. Sometimes my ComfyUI seems to disconnect from the Jupyter terminal and it will show images generating in Jupyter, but nothing appearing in ComfyUI. Did I do any of the steps wrong you think?

[@c0nsumption]

🤔 Could just be server lag. Or some kind of bug with the server itself.

[@user-sf5uo3zf7k]

Hello! Your ComfyUI tutorials are great and I always learn by watching your videos. I have one request, if possible, could you please tell me how to use FaceDetailer (Stable Diffusion=Adetailer) with animatediff in ComfyUI? I'm still having difficulty finding a good workflow for using FaceDetailer with the animatediff of ComfyUI.

[@Panchocr888]

Wow! Thanks so much for this detailed guide! Could you please share the animdiff worflow? the json file i mean, thanks in advance!

[@phxfilmmaker]

Would setting up on AWS (maybe EC2 instance with G5) follow similar workflow?

[@dimas4251]

Amazing tutotial !!!! Keep them coming i will be checking you channel a lot from now on. Can you share the workflow file ? I tried to download it but the link goes to a diagram png image

[@c0nsumption]

You can use the png file :)

[@krosmo]

Thank you so much for this. As somebody who is an absolute beginner in all this, your video was extremely helpful. I'm wondering, is it sometimes worth it to never completely terminate your pod? I would like to use runpod extensively, with various types of workflows (various nodes,models, etc). But I do not want to have to set everything up everytime. Because it's relatively cheap, I might just consider leaving the pod turned off, but not completely terminate it. Or do you have another suggestion? Thanks again! subbed!

[@c0nsumption]

There are way to actually offload that stuff to the cloud and connect your cloud storage to Runpod instead. So technically you won’t always have to set up. I’ll be covering things like this extensively. Been researching Runpod for a couple weeks now

[@steinlo]

@@c0nsumption yeah I second this. Would love a video that delves into a runpod workflow that start everything up again after terminating, without too much work/scripting.

[@eprohoda]

Super. good channel- see ya-

[@c0nsumption]

Thanks sir 🙏🏽

[@bilalpenbegullu2851]

I was looking for the god in the wrong place, he was here all along.

[@c0nsumption]

😂 you’re too kind

[@omerduhan7408]

Thank you for the video. I can't see base model when I try to choose on the interface of comfyui, it is saying null. I have wget model from civitai but it is not showing up.

[@krlkai4222]

I followed everything perfectly up until 18:11 and got an error when queuing the prompt: Error occurred when executing ADE_AnimateDiffLoaderWithContext: ('Motion model motionmodule15v2.ckpt is not compatible with SD1.5-based model.', MotionCompatibilityError('Expected biggest down_block to be 3, but was 2 - motionmodule15v2.ckpt is not a valid AnimateDiff-SD1.5 motion module!')) Is this because the motion module is missing an extension?

[@christopherniv1755]

THanks for the video. can you do it for kaggle ?

[@neerhuman]

Great tutorial! Just a note: the link to the workflow file doesn't point to a json but to a png image of the workflow

[@c0nsumption]

Should work the similar in the mean time. Thanks for giving me the heads up, I’ll fix it in an hour! Just got home 😳

[@soapyowl3042]

@@c0nsumption yes the workflow would help me out a lot! Thank you so much for the tutorial btw it helped me quite a lot!

[@soapyowl3042]

oh nevermind you can just drag and drop the png file haha

[@doududouduFR]

Greetings. thanks for your work. When I activate load checkpoint on comfyui, it shows null, base models are not shown, what can I do?

[@c0nsumption]

Sorry for delayed response. I will be refilming this to update the tutorial. You must ensure that your checkpoint fully downloaded when doing wget and that it was named properly 🙏🏽

[@busraergin3679]

yeeyyy

[@elihusolano5993]

Could you posibly make a tutoriol on how to run it serverless?

[@AlexAtiq]

Is it possible to initiate a Runpod VM with limited GPU resources, and once the setup is done / user interface (UI) is configured, can you migrate it to a pod with higher GPU capabilities?

[@c0nsumption]

I will be doing a deep dive very soon brother. I am launching my channel again as of tomorrow. Runpod will be a huge topic. One experiment we will be doing is hooking up Runpod to an external storage system to see if we can then accomplish what you’re saying: - set up instance - connect to storage medium - run workflow. Forgive my tardiness, I have been in deep research with a couple businesses and am bringing Cloud Computing, AI, Spatial Computing, and much much more as the primary focus of my channel. Tomorrow will be an announcement and updates video 🙏🏽

[@meredithhurston]

Is there an update on this workflow? It isn't working for me.

[@digidope]

Does it save everything you did like new models and ffmpeg setup when you stop the pod? So you don't need to do all that everyday. Do they charge for keeping the data/pop while it's not running?

[@c0nsumption]

With questions like this: Please watch video first. This is what the video is about. This is touched on multiple times in different sections.

[@c0nsumption]

16:30 till just about the end

[@pavi013]

Good tutorial, i was trying to get facefusion work with the same steps but im unable to connect with runpod's ip:port. I also added the facefusion's default port.

[@c0nsumption]

Hmmm, interesting. I’m currently tied up but I’ve been researching Runpod extensively. Videos and live streams coming :) Been delayed cause of many interviews and connections, forgive me 🙏🏽

[@pavi013]

No worries! i found docker template for facefusion, so it's working for now at least.@@c0nsumption

[@rpwogus]

Help needed: I don’t see comfyui template on runpod. Looks like it’s gone or no longer available. What should I do, or as an alternative which template could I use instead of the comfyui template? Thanks in advance for any help!

[@c0nsumption]

Technically you would just need to git clone ComfyUI in and proceed with installation instructions. I’ll be updating this soon 🙏🏽

[@rpwogus]

@@c0nsumption thank you!

[@devoiddesign]

Hello, Thank you for well timed and detailed tutorials! Unfortunately, at the last step of course I received an error stating my motion module is not compatible? Error occurred when executing ADE_AnimateDiffLoaderWithContext: ('Motion model motionmodule15v2.ckpt is not compatible with SD1.5-based model.', MotionCompatibilityError('Expected biggest down_block to be 3, but was 2 - motionmodule15v2.ckpt is not a valid AnimateDiff-SD1.5 motion module!')) File "/workspace/ComfyUI/execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/workspace/ComfyUI/execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/workspace/ComfyUI/execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/nodes.py", line 112, in load_mm_and_inject_params mm = load_motion_module(model_name, motion_lora, model=model, motion_model_settings=motion_model_settings) File "/workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/motion_module.py", line 230, in load_motion_module raise ValueError(f"Motion model {model_name} is not compatible with SD1.5-based model.", e)

[@devoiddesign]

NVM I downloaded the latest beta for that motion module's SDXL version by mistake.

[@wagmi614]

thanks bro, mind adding timestamps and chapters? easier to come back to the specific parts

[@c0nsumption]

Yeah, Il figure out how tomorrow. Super tired after Muay Thai 🧍🏽‍♂️

[@alexmedvec4571]

Ok.The problem is some of my dependencies in Friz nodes disappear randomly after I shut down the GPU. I think that I fixed the problem after 3rd time building the image generator. the problem was likely that after going back to workstation after shutting down GPU for the day, I didn't activate venv again. It probably corrupted the files this way. You should have said that we had to do it every time we access our workstation.

[@mireusted499]

How much VRam would I need to run Steerable motion?

[@c0nsumption]

I will be dropping a video on this very soon actually, stay tuned. Haven’t tested on new updates!

[@jittthooce]

why am I getting a "No space left on device" error when installing pytorch?

[@c0nsumption]

Your pod doesn’t have enough storage space. You need to adjust the amount of storage they are giving you. I’m gonna need to update this.

[@user-yn1ir6hf4d]

Hey nice artwork! Just want to ask that are you interested in our new ai generation product? Maybe I can send you beta version to you~

[@c0nsumption]

Contact me on X: @c0nsumption_

[@chauhandivyansh9416]

I dont know why, but the comfy ui manager button doesnt appear in the UI.

[@c0nsumption]

‘git pull’ in the ComfyUI root folder. Needs to update so that it can be compatible with the manager

[@chauhandivyansh9416]

@@c0nsumption Thanks a lot man it helped.

[@SheRoMan]

Hi i have that error in my local Comyui "[AnimateDiffEvo] - WARNING - ffmpeg could not be found. Outputs that require it have been disabled" how i can fix it please :)

[@c0nsumption]

Watch from 16:30 forward. Make sure to finish the video, there are important details on it

[@SheRoMan]

yes i tryed what did you do in the video but this error happend apt : The term 'apt' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + apt install ffmpeg + ~~~ + CategoryInfo : ObjectNotFound: (apt:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException @@c0nsumption

[@SheRoMan]

I'm talking about confyui in my PC not in cloud @@c0nsumption

[@SheRoMan]

Should i download or insatll something cus i'm a noob in this @@c0nsumption

[@c0nsumption]

@@SheRoMan chatgpt is saying that’s a powershell error and not possible on a Linux server

[@9yrolder376]

i cannot install. this is a scam i bet how can i get money back.... it just stops installing for no reason and i cant do anything

[@c0nsumption]

I already said you need to increase your drive size.

[@9yrolder376]

@@c0nsumption when i terminate the pod i will lose all these hours of downloading stuff? wow.

[@9yrolder376]

i cannot install new checkpoints, i install them and they never show up on comfy ui........

[@RoysCastle]

please help! Now this error has started to appear at the apply ip adapter stage: ERROR:root:Traceback (most recent call last): File "/workspace/ComfyUI/execution.py", line 155, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/workspace/ComfyUI/execution.py", line 85, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/workspace/ComfyUI/execution.py", line 78, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/workspace/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 654, in apply_ipadapter self.ipadapter = IPAdapter( File "/workspace/ComfyUI/custom_nodes/ComfyUI_IPAdapter_plus/IPAdapterPlus.py", line 279, in __init__ self.image_proj_model.load_state_dict(ipadapter_model["image_proj"]) File "/workspace/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}: \t{}'.format( RuntimeError: Error(s) in loading state_dict for Resampler: size mismatch for proj_in.weight: copying a param with shape torch.Size([768, 1280]) from checkpoint, the shape in current model is torch.Size([768, 1024]). how to fix it?

[@3ky3ky]

Error occurred when executing ADE_AnimateDiffLoaderWithContext: Motion module 'motionmodule15v2.ckpt' is intended for SDXL models, but the provided model is type SD1.5. File "/workspace/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/workspace/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/workspace/ComfyUI/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/nodes_gen1.py", line 138, in load_mm_and_inject_params motion_model = load_motion_module_gen1(model_name, model, motion_lora=motion_lora, motion_model_settings=motion_model_settings) File "/workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/model_injection.py", line 424, in load_motion_module_gen1 raise MotionCompatibilityError(f"Motion module '{mm_info.mm_name}' is intended for {mm_info.sd_type} models, " \

[@3ky3ky]

nvm ... downloaded bad version, fixed and working 20-04

[@9yrolder376]

couldnt install requirements.txt, gave me this error " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━ 516.0/731.7 MB 183.0 MB/s eta 0:00:02ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device" what a scam this website????

[@c0nsumption]

You need to make your drive larger, not enough storage!

[@c0nsumption]

Delete your pod! Make sure to increase storage config to something more useable 🙏🏽

[@yuxiwan8917]

Just wonderful!