NEW Image2Video for ComfyUI. How to use CogVideoX.

[@sebastiankamph]

Workflow and guide here www.patreon.com/posts/112417217

[@LouisGedo]

👋 hi

[@thenextension9160]

thanks, went ahead and subscribed on patreon. your worflow works well and you packaged everything up nicely.

[@CuoreJTD]

CogVideoDecode: - Exception when validating inner node: tuple index out of range any idea how to fix it ?

[@thenextension9160]

@@CuoreJTD I think that’s means old nodes against newer model not sure if they are compatible yet

[@bystrovphoto]

Prompt outputs failed validation CogVideoDecode: - Exception when validating inner node: tuple index out of range

[@CuoreJTD]

Same error (

[@Afshin-Zarechian]

Hi, the DPM scheduler seems to be missing for me? Can I just use DPM++?

[@daliprane1711]

Did you try? Same here

[@Afshin-Zarechian]

@@daliprane1711 Yeah I used DPM++, not sure if this is optimal

[@Valeria.Delmar]

Thank you for your work, your videos are very educational and interesting.

[@ripleyhrgiger4669]

Thank you for the video and workflow! Can't wait to try it out!

[@Radarhacke]

Thank You. Is this normal on a 16 GB VRAM Card to need over an hour for one video? In addition, the t_tile length and overlap setting is missing in my CogVideo Sampler. Please help. Edit: I have a Pinokio Install of Cog Video Studio. A 50 Steps Generation takes only a few minutes on the same machine (Model: 5b).

[@Geffers58]

Fantastic stuff, thanks for the guide. Gonna need a bigger hard drive soon with all these multi-gig models!

[@sebastiankamph]

I tend to delete models to make room for new ones on a daily basis 😅

[@GenoG]

Cool concept and well explained! Not sure what I would use it for, but good to know. Also, I always learn some other thing when I watch your videos, so thanks for explaining things so clearly! 😀

[@sebastiankamph]

Thanks for watching! Have a great day Geno 🌟

[@psznt]

Hi! Thanks for the video. I tried it out but get this error on my Mac: Expected query, key, and value to have the same dtype, but got query.dtype: float key.dtype: float and value.dtype: c10::Half instead. Could you help fix it?

[@KifKroker]

great, works perfectly !

[@tobsho]

how do i fix this error? DownloadAndLoadCogVideoModel Error no file named diffusion_pytorch_model.bin found in directory

[@hug_studio]

I'm stuck on this too, did you manage to solve it?

[@sonic55193]

This is taking forever on my 3080ti. 12gb VRAM. It's stuck on cogvideosampler. 2% for 15 mins, I don't think 12gb VRAM is enough to run this unless you're willing to wait whole day.

[@97BuckeyeGuy]

In the "(Down)load CogVideo Model" Node, change the "enable_sequential_cpu_offload" to true and try again. That helped me get it to run on my 12GB RTX 4070Ti.

[@sonic55193]

@@97BuckeyeGuy I'll try that.. Thank you.

[@ericbournat]

@@97BuckeyeGuy thanks

[@Larimuss]

It's probably not processing just run out of memory.

[@nocommend9520]

Error no file named diffusion_pytorch_model.bin found in directory C:\Users\PT Computer\Desktop\From PS To motion PS\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\models\CogVideo\CogVideoX-5b-I2V. im having a problem with this one did anybody solve it can u suggest me anything to contenue Please can you help me out? I've fresh installed ComfyUI and all the dependencies + the custom nodes & models as per the video. Still I keep getting this error.

[@kymatekk]

Any tips on how to save time when rendering? I even put the steps to 25 instead of 50 but it still takes me 30 minutes to generate the 24 frames. I run on a 4090 laptop (16gb vram, about 30% less fast than Desktop 4090)...

[@Nelwyn]

Try setting fp8_transformer to fastmode. That helped a lot for me.

[@abdullahazzam664]

Hey Sebastian, I keep getting this error "DownloadAndLoadCogVideoModel 'CogVideoXTransformer3DModel' object has no attribute 'context_embedder'" Please can you help me out? I've fresh installed ComfyUI and all the dependencies + the custom nodes & models as per the video. Still I keep getting this error.

[@daniellobo2290]

I also keep getting that same error

[@EllesarAragorn]

i am getting : CogVideoDecode: - Exception when validating inner node: tuple index out of range any idea how to fix it ?

[@CuoreJTD]

Same error (( Do not know how to fix

[@Valarxx]

Thank you for this video, very helpful!

[@mr.entezaee]

How and where can I download this manual model? CogVideoX-5-i2v My internet is automatically weak for me

[@autonomoussoup]

This is amazing! Thank you so much~

[@Pascal_Tesla]

Realy cool. thanks.

[@vasilybodnar168]

In my case, I had no error but every time output was weird - the picture consisted of tiles or mosaics... Tried to change some settings - doesn't help. So.. I discovered that there is another node - CogVideo GGUF Model and it can use I2V gguf. So I gave it a try, and it's worked.

[@97BuckeyeGuy]

Nice find!

[@royjones5790]

Where was this? I can't seem to identify it. I got something that says (Down)load CogVideo GGUF Model as a node. Is that it? Just a node to force a download? Can't find the file independently?

[@vasilybodnar168]

@@royjones5790 yes, this node. At first run it will automatically download all files.

[@Larimuss]

Nice thanks. So it loads the video process into ram and CPU I guess.

[@thewebstylist]

I get stoked about these tittles to watch until realize ComfyUI is those steps and layout that throws my brain way off

[@kaitokai2441]

Hello, can someone help me? When press queue its not downloading the model, it says Error no file named diffusion_pytorch_model.bin found in directory and then the folder where the model is supposed to be.

[@haydnstrauss]

It seems there's actually no way to randomize the seed in the latest version of ComfyUI?

[@agxai]

I have an assignment for you buddy! do a tutorial on " Importing & Combining Workflows" from multiple .json files!

[@r481l07]

Did you encounter this issue during the production process? 'CogVideoSampler User specified an unsupported autocast device_type 'mps'.

[@MiracleMan-ol6gd]

How is this rivalling Runway and the like? The output is almost stationary.

[@maharshistudios]

Worked on my Titan 24gvram. How to proceed is yet to come as it still needs to reduce to low vramm

[@gateopssss]

Is anyone else experiencing an issue with the blocky video and it turns into blue/purple color? I just waited an hour for this thing to render just to see the blocky mess it turned into. And it doesn't work with any prompt, any image, nothing. I just followed the entire video, used the given workload with the darth vader picture and it's always the same with every picture and every prompt, blocky colored mess! What am i doing wrong?

[@knot-head]

same

[@gateopssss]

@@knot-head So far i've only experienced the issue with this workload, but theres another one with cogvideo fun 5b i2v, different workload, different model but i had no issues apart from unsatisfying results which was a mess on every prompt.

[@knot-head]

I got it working, so it seems it does have to be num_frames 49 to avoid the bad result, But, I have to enable_sequential_cpu_offload and enable_vae_tiling to avoid other errors. This makes it use a fraction of my vram unfortunately and takes forever. It seems to barely overrun tho, maybe if the 4060ti 16GB was headless it would make it, idk

[@knot-head]

@@gateopssss got it working but very slow, I am going to try some other workflows and or nodes, I'm especially interested in getting the temporal tiling working for endless loops .

[@gateopssss]

@@knot-head The num of frames was the issue? Damn thats so stupid.. I had a situation with the cpu offloading to be put on false, and i had it generate a blocky mess (24 frames) for 4 hours! When i enabled cpu offloading, it took only 20 minutes. If you find another workflow please hit me up!

[@zeltfor]

Is it lenght limited or can you extend it after the initial generation?

[@travelingchris86]

You could extract the last frame and run it again, haven’t tried it but might work.

[@Teamsao90210]

Is there any way to expand on this to be able to stitch multiple clips together and pass multiple prompts in (1 per clip) so we can have tons of small clips that we can make a longer video out of?

[@knot-head]

Not a good one, you can run it twice and then play one in reverse so they go together seamlessly. I'm about to try prompting for panning the camera left, and then one panning the camera right to try to get a decent result from that. With the one in reverse it should create one long pan in the same direction, hopefully it would work to create one long zoom in also.

[@daliprane1711]

@@knot-head smart haha

[@edwassermann8368]

cool. thank you!

[@sebastiankamph]

You bet!

[@Cu-gp4fy]

noticed u using Thinkdiffusionn would that help if my computers complet rubbish ?

[@parthwagh3607]

Thank you so much for all your informative videos. They helped us a lot. I have some special request. Could you please create a workflow that can (1) generate image of person using flux model, (2) then that image will go to correct all deformities such as bad hands and bad eyes, (3) after this, this image will go for face enhancement, (4) then more details to the skin and hair will be added to make it realistic skin and more natural human being, not typical ai generated image. (5) And finally, we will upscale the processed image. All above will be done in single workflow. We can also do batch processing in it. also, we can add functionality providing multiple images of single character including whole body to create consistent character. and this would be done without lora, just using multiple images not single image. Thank you.

[@CerebralChuckles]

Anyone know why I cannot see the manager in my UI?

[@psrenatorusso]

Mine also had disappeared. I deleted the Manager's folder, on custom nodes, and installed the Manager again (downloaded the .bat to install it on a portable Comfyui).

[@jimdelsol1941]

Hi ! I don't know if anyone had that problem before, but my first generation is working fine, and then all my subsequent generations are just black. Sometimes it work again for one time after rebooting my computer, once I got lucky and it worked again after changing the scheduler but everything black again afterwards. Any Idea why it might do that ? I have a RTX3090 and 32gb of Ram.

[@LEONARDO-z-b2o]

everything work fine but with img2video mode only appears me the "envato" font with grains, any suggestions? very strange

[@manleycreations6466]

Will it handle 3:2 and 2:3 or can it only do landscape?

[@SemorezX]

unfortunately ; cant change resolution

[@knot-head]

the 5b-I2V only does 720x480 but 5b and 2b will run at other resolutions but the results were not as good. Also notice that the width and height are reversed on the sampler node for some reason, but need to match the resize image node,

[@NOORIEAI]

Thanks

[@themessengerofgood]

is it possible to make a loop video in 30-40 seconds?

[@ragoutvideo1]

CogVideoSampler Given groups=1, weight of size [1920, 16, 2, 2], expected input[20, 32, 60, 90] to have 16 channels, but got 32 channels instead getting this error. also have a little less options in the CogVideo Sampler

[@VermontJayson]

how do i auto save the output?

[@64kernel]

Definitely needs optimization. It says 10 hours on my 12 GB 3060.

[@Gaganpulsi]

Only optimization you really need is 4090 lol

[@Gaganpulsi]

Tbh 8 of them

[@Neuromindart]

I am using an 8gig card but it keeps saying out of memory by 400 mb. Is there anything I can tweak so that I can still run it?

[@harveyheap9111]

I've gotten stuck just downloading the FP8 model, I get an error, certificate verify failed: unable to get local issuer certificate, can anyone help?

[@marcopineda9507]

Question: What's the VRAM of your GPU so it can process Flux

[@fooboomoo]

there are many quantized versions out there so something for everyone

[@luketodd6545]

Im getting the following error on a mbp m2 max - can anyone help? CogVideoSampler Expected query, key, and value to have the same dtype, but got query.dtype: float key.dtype: float and value.dtype: c10::Half instead.

[@NWO_ILLUMINATUS]

Where does the Thudm cogvideox 5b 12v file save?

[@TheLichoShow]

It work faster using "--fast", did you tried it ?

[@EvilSpyBoy]

I just tried the workflow a few times and it was getting stuck at the CogVideo Sampler frame generation part. I was going to ask for help but I just changed the load CogVideo Model fp8_transformer from disabled to enabled (which it says is for nvidia gpus). Not sure if that resolved my problem or if it was just something like the seed in the CogVideo Sampler... but it appears to be trying to run now (yep just changed from 7/50 to 8/50 so it's not hung... yet)

[@sebastiankamph]

Happy to hear you found something that worked for you! (it seems to at least)

[@CircadianHighway]

Hi - What graphic card will this need to seamlessly run on Win 11?

[@BasketOfPuppies]

A 4090 or 3090. It really looks like this one needs 24Gig of VRAM to be happy from all the comments of folks with less VRAM.

[@artisinalbread4378]

I'm beating my head against a wall here. I cannot use any image to video CogX workflow. They are all broken in different ways for some reason on my pc. Right now this workflow gives me this error: Prompt outputs failed validation CogVideoDecode: - Exception when validating inner node: tuple index out of range And the worst part is I have no idea who to ask for help.

[@KrakenCMT]

I just tried this too and had the same error. I tried updating ComfyUI, but that didn't work, since it was already updated. Still a good idea to do that tho. After a bit of troubleshooting, I ended up having to replace these 4 nodes - CogVideo Sampler, the CogVideo model download, the video encode and decode nodes. You'll have to copy the previous node values to the appropriate spots. My guess is what happened was that the nodes have changed in a way that possibly broke the workflow somewhat. This got it to work and queue up and run, but I got some weird artifacts. It's probably my just settings.

[@developmentmotion4765]

hello, just wanna ask, why my 5b-i2v not appear, there is only 2b and 5b? tyvm

[@sebastiankamph]

Update to latest

[@jdavidmusic]

Getting about 5-6 mins on my 4090. However, I cannot increase the num_frames passed 49 or I get an output that show a bunch of tiny tiles, bright chroma, and one frame of it is almost the original image. I tried increasing tile sample height/width size... decreasing frame rate... decreasing steps... everything exhibits the same colorful tiny tiles...

[@jdavidmusic]

It seems the only thing causing the error is setting the num_frames above 52

[@octopuss3893]

you can't change the number of frames because it runs at 49. while waiting for it to get faster you can lower the step to 10

[@jdavidmusic]

@@octopuss3893 Well I was able to raise the num_frames to 52... after that it gets the tiling. Was hoping to adjust the frame rate to like 15, but increase the num_frames to 60 or 75, then use Davinci Resolve to boost the frame rate with interpolated frames to 24 or 30. At 740x480 it's already NTSC resolution from 25 years ago... but to only have 2 second clip... well that really limits its uses. I mean choosy beggars and all aside...

[@porsepa]

@@jdavidmusic I got the same result, thought I was the only one getting this error and was about to go crazy... like someone suggested in another comment, you can use the last frame to keep the sequence going but c’mon, that’s so impractical lol

[@Sedjwin_]

Failed due to trying to allocate 56GB of cuda memory. I must be missing something? my 12gb card should be enough for your default settings?

[@johnfraser21]

I too get a failed error do to not enough vram and I'm using a rtx 3080 12gb model.

[@johnfraser21]

I tried a fresh install and no change. I think 12 gig cards just aint enough unfortunately.

[@TheBagOfHolding]

​@johnfraser21 maybe you need to change to mid vram in the config

[@binarybrian]

Yeah, I'm not sure how 6GB cards are supposed to work. Maybe once the 2b-I2V comes out. In the meantime I ticked the "enable_sequential_cpu_offload" to get things rolling on my 16GB 4060ti. 25 minutes to run the job but that's fine...

[@ratside9485]

You can also run it with Pinokio and everything will be installed automatically for you. (CogStudio)

[@DaveTheAIMad]

The workflow may need updating, denoise max is now 1 and dpm is not regonised. so far the result has not been as good as mochi... however mochi wont do image to video only text.

[@christiandeleo7737]

I have an Nvidia rtx 3060 12gb 32gb ram and after 30 mins is still at 3%! 😅 had many errors, now is saying temp tiling and context schedule disabled.. 😂😂 first run.. let’s see if it works

[@MaximPechony]

hey can someone help me ? , i am trying to figure out how to make higher res videos i know this model is limited to 720p but how can i upscale the video? are there any wrkflows for this model? i need realistic kind of 3d simulations and this model gives reallt good results. i am open to using different models if there is a higher resolution cap. hardware is no issue

[@sebastiankamph]

Simple way to upscale videos is topaz video. You can also run your video through an upscaling model and ksampler with a low denoising (you'd have to build that workflow yourself).

[@vicentepallamare2608]

It's simply stuck on 6gb 3060 :( Any option to make it work - even waiting longer?

[@ricky_pigeon]

someone said in another comment: Larimuss: In the "(Down)load CogVideo Model" Node, change the "enable_sequential_cpu_offload" to true and try again. That helped me get it to run on my 12GB RTX 4070Ti.

[@vicentepallamare2608]

@@ricky_pigeon will re-test thanks. I've also tried a setting for Nvidia that will offload to the system ram. Slow sure but hopefully no oom....

[@levi_melon369]

Sorry to say that, i feels like i gotta to put the mic into your mouth, so that mic can receive your speech clearly

[@egilsnorri4667]

git clone and install missing nodes fails to import for me.

[@SemorezX]

must clone all folders ; tolenizer ; transformer ; vae and ...

[@lawrencekevin8807]

I had to swap in CogVideoXFun Sampler, and then I was able to run the process.

[@sebastiankamph]

Great to hear you got it running!

[@vicentepallamare2608]

what specs?

[@M--S]

These are no "videos", only fotos with a moving background! Who needs this?

[@sebastiankamph]

Why don't you prompt it for what you want then? Like a person walking for example.

[@joethompson9124]

?? you can get a lot of movement, just depends on your prompt

[@M--S]

@@joethompson9124 Have a link to some examples? (The tutorial should provide this!)

[@M--S]

@@sebastiankamph What I saw up to now, did not convince me! Do you have a link to some examples?

[@thenextension9160]

you need to prompt

[@BlueEagle669]

I want to see what type of video he's making out of this thumb nail...

[@anianait]

outputs black ...

[@vincentnestler1805]

Thanks!

[@sebastiankamph]

Very kind, thank you for your support, you absolute mvp you ☺️💫

[@vincentnestler1805]

@@sebastiankamph your video is perfectly done. Thank you! I am encountering a couple of errors. on my Mac (m3 max pro - 120 gb unified ram) I get the following error "CogVideoSampler User specified an unsupported autocast device_type 'mps'". Thoughts on that? I am double checking the error on my Windows machine (4090 with 16gb vram, gaming laptop) Any guidance much appreciated.

[@sebastiankamph]

@@vincentnestler1805 I'm sorry but I don't own a mac so can't test. Do let me know if it ran on your pc. Check the requirements.txt in the custom node. For windows you can: pip install -r requirements.txt Maybe something similar for Mac.

[@vincentnestler1805]

@@sebastiankamph update - On my desktop gaming rig with 4090 - ran great. On my lapton... It hangs at about 8%. Might not meet the min spec. On the mac... I will keep digging. Thanks again for a great video. Very helpful.

[@DecentralisedGames]

The AI Martin Shkreli.

[@sebastiankamph]

That's kind of rude

[@johnfraser21]

@@sebastiankamph Yeah I don't know what this troll is talking about. Keep on doing what you do my guy, you're great!

[@DecentralisedGames]

@@sebastiankamph Haha, I love ya dude, been following for ages, you'll be right with my jokes

[@sinuva]

would be nice to do looping videos...

[@ricky_pigeon]

ones its done put it in flowframes...

[@chenlin322]

it's really slow. I'm only 3060 6G😂.I don't expect it to work out🤣

[@vicentepallamare2608]

did you produce anything - mine is simply stuck :( And GPU load goes to 0 so I reckon it just stops ... no oom message tho

[@content1]

Really? Sorry for being honest, but the sample looks terrible, I am not even going to try. see you in the next video.

[@abdullahazzam664]

ANYONE HELP!!!

[@obscuremusictabs5927]

first

[@sebastiankamph]

You got it! 🌟

[@lucifer9814]

Not a single person on youtube speaks about how to install this, it's all about how to use it. At this point I feel like they just bait you into watching their videos to get their watch time. I am fed up of going through tons and tons of videos just to find the same content. Even here, apparently if you don't sign up for their patreon you can't get the content !!! This is absolute bullfkingshit

[@knot-head]

The workflow is not behind the paywall, just scroll down.

[@lucifer9814]

@@knot-head thanks

[@phu320]

that install is way too complicated for me. why does everybody seem to prefer incredibly complex installations?

[@groboclone]

I personally like that there's that barrier to entry, it means those of us willing to put in the work to use a lot of this cutting edge stuff have an advantage over those who aren't. Anyway, I think people at the forefront of open source AI are interested in advancing the technology, not in creating frictionless user experiences for laypeople so as to have mass appeal. That'll come in 2/3 years when this is an iPhone app

[@thenextension9160]

its actually not that hard. this stuff is bleeding edge technology it is not beginner friendly. if you want in on this, you need to put in some work and spend time learning. you can do it.

[@derrickpang4304]

Comfy Ui is not that complicated if you put in the hours to get used to it. The amazing part of comfy is it allows different technologies to be incorporated into one single platform without much modifications.

[@AB-wf8ek]

If you want easier installation, there are plenty of video generators that only require a credit card number to run.

[@mr.entezaee]

If I want to download the model manually, then in which folder should I put it? google_t5-v1_1-xxl_encoderonly

[@martovitch]

models/clip

[@blade6568]

Got this issue "Given groups=1, weight of size [1920, 16, 2, 2], expected input[26, 32, 60, 90] to have 16 channels, but got 32 channels instead" any fix?

[@stevietee3878]

Same here, did you find a fix ?

[@Ayawiskas69]

@@stevietee3878 I have the same problem!

[@SemorezX]

must make new folder in model and clone all files and folders ; like tolenizer ; transformer ; vae and ...

[@blade6568]

@@SemorezX hmm interesting

[@RobertWatts1957]

I had this error, because I had chosen the wrong model. I forgot to pick the I2V one.

[@Martin-o1x4c]

im stuck, can't install bigger models like the fp8_e4m3fm needed, it says "Install failed: google-t5/t5-v1_1-xxl_encoderonly-fp8_e4m3fn Bad Request"

[@CuoreJTD]

CogVideoDecode: - Exception when validating inner node: tuple index out of range any idea how to fix it ?

[@abdullahazzam664]

Hey Sebastian, I keep getting this error "DownloadAndLoadCogVideoModel 'CogVideoXTransformer3DModel' object has no attribute 'context_embedder'" Please can you help me out? I've fresh installed ComfyUI and all the dependencies + the custom nodes & models as per the video. Still I keep getting this error.

[@shaibanihangoro5927]

Thanks

[@abdullahazzam664]

Hey Sebastian, I keep getting this error "DownloadAndLoadCogVideoModel 'CogVideoXTransformer3DModel' object has no attribute 'context_embedder'" Please can you help me out? I've fresh installed ComfyUI and all the dependencies + the custom nodes & models as per the video. Still I keep getting this error.