Stable Diffusion ComfyUI Create AI Influencer Dance Video With Mimic Motion (2024 Guide)

[@kalakala4803]

Just tested it. Looks good, I created with anime and 3D characters.

[@TheFutureThinker]

Great 👍 you did it.

[@pxrposewithnopurpose5801]

mimic motion not showing the result. so basically it shows me the reference image then my pose reference and then my pose. but doesn't show the reference image being animated

[@kalakala4803]

🎉🎉nice , I think I can move on from AnimateDiff to this 😂😂😂

[@TheFutureThinker]

Yes! Let's go!

[@AsiaFreeTVNet]

😄 我來了！Ben 嘩, 見到真係 KZbin 好多白痴留言。D人係咪真係咁L蠢 問D問題...好心佢哋啦, 認真了解就明白架啦👍

[@crazyleafdesignweb]

This is very cool framework. Really game changer.

[@giorgosbitsounis]

the installer guide should be step by step. It has been very difficult for me and I can't install it😶😶

[@kalakala4803]

the first day on ComfyUI is hard. Don't know where to click install.

[@wagnerfreitas3261]

great video man, thx

[@fsamobby]

just cut the model/backgrounf out of the scene using tracking feature in davinci or your preferred video editor then use AI to modify model then export it again in davinci so your background would be fine and sharp with minimum aberrations

[@christophermoonlightproduction]

@@fsamobby I did that but it just gives me a flashing, glitchy background.

[@InnovateFutures]

Nice! Thanks! Bye AnimateDiff 👋👋

[@TheFutureThinker]

Have fun

[@QuarkDoge]

On this video progress takes ~1 sec for a step. On my 8gb GPU - 1 min (!) for a step. How much VRAM do you have?

[@TheFutureThinker]

24 GB. I experienced that in AnimateDiff before with 8GB VRam , I know how that feel. Maybe try cloud gpu or updagrade ur hardware before trying.

[@arjuntrehan48]

@@TheFutureThinker how will it work on a 4060Ti 16GB will it be enough??

[@TheFutureThinker]

@@arjuntrehan48 I don't have a 4060Ti , can't comment on it.

[@GG_Edify]

@@TheFutureThinker How wiill it work on MacBook Pro 2023?

[@Lightof-Truth]

great tutorial! thanks!

[@christophermoonlightproduction]

I got it set up and it's working okay but I have two problems. One is there's a lot of distortion so I think I need to find the right sizing to optimize my results. The other is that I'm getting a row of my sorce video, my charger, the open pose animation, and the final result, all as one output. How can I just get the final output video without the other stuff? Thank you.

[@AxiomaticAssumptions]

The future is wild

[@arjuntrehan48]

for real :skull:

[@pxrposewithnopurpose5801]

Wow thats crazy

[@JolieKim2795]

Is it possible for it to look as realistic as a real person if combined with the most realistic model possible, rather than semi-realistic like this? I want to adapt it to create AI girls dancing ?^^

[@anonymousgamer6665]

How to get rid of the stick figures and the stock image from the final output video?

[@CyberPhonkMusic]

Thanks King

[@TheFutureThinker]

no please, I am not. Tencent is the hero, who make use of SVD with a better way. Rather than just generate camera panning motion. Should be thanks Tencent.

[@nonmaisX]

@@TheFutureThinker Tencent is god and you are Jesus 🤣

[@sirjcbcbsh]

My result was blurry, could you give me some advise on how to generate higher resolution videos. I got a rtx4060 with 16vram

[@Aireelsmaker-k8n]

Sir i getting this error. Error occurred when executing DownloadAndLoadMimicMotionModel: Error no file named config.json found in directory C:\Users\akash\Documents\ComfyUI_windows_portable\ComfyUI\models\diffusers\stable-video-diffusion-img2vid-xt-1-

[@HassanAli-o1j1c]

Hey bro i have a image and a video and i want to replace a person in a video with person from image also i want to generate consistent poses for person from image as know it is creating long hairs in some frames and short in some frames and i want to match the lighting for person can you guide me how i can achieve this?

[@nadershalash]

I use run comfyai how to make it make all video lent my inputs is 10 sec he oly make first 2 sec and stop any solution for this , thanks in advanced

[@onlyyoucanstopevil9024]

Thx for new info 👍👍👍

[@TheFutureThinker]

Welcome👍

[@defiantvisuals7072]

I get the following error: Error occurred when executing MimicMotionSampler: "compute_index_ranges_weights" not implemented for 'Half' Anyone know a solution?

[@rawnclark]

I get error "sizes of tensors must match except in dimension 2"

[@Vashthareaper]

for anyone struggling with this, install Kijai mimicmotion wrapper instead, its easy on gpu

[@TheFutureThinker]

Lol, its Kijai wrapper in this video tutorial. Are you missing something?

[@TheFutureThinker]

And , " its easy on gpu?" So you are running ComfyUI with GPU before? Feels like you are happy that you find something new

[@Vashthareaper]

@@TheFutureThinker yes but this original repo gives too many errors to too many people for some reason.

[@TheFutureThinker]

@@Vashthareaper I understand, and the LivePortrait node that they did. Missing some features , looks like they are rushing to release some new custom node recently.

[@RaulRodriguez-hk4xr]

my question is this have to be install in local drive because for what i see its very heavy to storage all this model in a local drive ?

[@TheFutureThinker]

Yes , install locally. Its large file, so it takes time to download

[@RaulRodriguez-hk4xr]

@@TheFutureThinker how do you manage with so many model them, do you recommend any other way to install this ?

[@TheFutureThinker]

@@RaulRodriguez-hk4xr er... Woah.. that is very very tough. I brought a 4TB hard disk only running AI stuff.

[@godrimuru9631]

i am considering to became a patreon but i have a question. if i join the patreon do i get also the comfyui templet or only a tutorial video? thanks for the video btw it is amazing work

[@JolieKim2795]

How long for the 10sec clips to be made ?

[@TheFutureThinker]

Depends on your hardware

[@singhsgurpreet]

it gives error as :- Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate` any idea?

[@TheFutureThinker]

Low cpu memory usage? You use CPU to run this?

[@UnsolvedMystery51]

Can this ai model handle multiple characters?

[@TheFutureThinker]

No , its born to create single dance animation

[@paulcrist1762]

Thanks for making this, it was easy to follow for even a noob with an old machine like me :) I did have a question though, and I'm sorry if it's already been asked. My first attempt at MimicMotion was bad, a dancing blob basically. So I started looking for tutorials on how to use it since I'm sure it was something I did wrong. I cam across a video on installing MimicMotion and when I checked the folders I saw that in the Models/MimicMotion folder there's only 1 file, MimicMotionMergedUnet_1-0-fp16 In that other video which was for just plain MimicMotion with no merge, he was downloading several models for MimicMotion. Do we also need to download and install those models, are they not needed, or are they in a different folder already? Thanks again for the great video

[@paulcrist1762]

I'm continuing to experiment and this went over my head. I took a picture of the first frame of the video with a screen capture then resized it to have very close to the same number of pixels and used it as my reference picture to test things. I still get a dancing blog and I also got this error message: Warning torch.load doesn't support weights_only on this pytorch version, loading unsafely. mimicmotion I can't find a node to turn off "weights only" I went searching on google an found this well written explanation that went over my head: This warning typically appears when you're using PyTorch 2.4 or newer and trying to load a model with torch.load() while specifying weights_only=True. Here's what it means and how to address it: What's happening: PyTorch 2.4 introduced a security feature: In versions 2.4 and later, torch.load() defaults to a safer loading mechanism that avoids potential vulnerabilities associated with pickle deserialization. weights_only=True is deprecated: The weights_only parameter was intended to load only the model weights and not the entire model state dict. However, this feature has been deprecated in favor of a more secure approach. How to fix: Update your code: Remove the weights_only=True argument from your torch.load() call. Load the full state dict: Load the entire model state dict using torch.load(). If you only need the weights, extract them from the loaded state dict. If you still need to load weights only (less common): Consider the risks: Loading weights only can be less secure as it may bypass important checks. Use a custom loading function: You might need to write a custom loading function that extracts the weights from the state dict manually. Important: Always load models from trusted sources. Loading models from untrusted sources can pose security risks. Stay updated with PyTorch releases. PyTorch updates often include important security improvements. I should mention that I'm using Comfyui Portable and your Freebie work load. I'm mostly a noob with this stuff & with Python and everything related but I do have a high IQ and I learn pretty fast. I changed some of the parameters in the nodes while experimenting but changed them all back again. So does this mean that I have to downgrade my PyTorch because I can't seem to find it in ComfyUI Portable to check the version I'm using.

[@Hackinside]

What about a fighting scene where there are two characters?

[@TheFutureThinker]

So far I have try the jump over action(from the stock footage in Artlist) For fight scene like Raid 2 the Karambit part, I will try it this weekend.

[@Hackinside]

@@TheFutureThinker I'm looking forward to the video about it.

[@hououinkyouma8233]

does it work for anime/ 2d char ?

[@TheFutureThinker]

Yup, tested

[@pixelwizardai]

What is gpu requirements for this to run as I have 8gb vram only.

[@UnsolvedMystery51]

Better to have 16 GB or more.

[@pixelwizardai]

@@UnsolvedMystery51 Thankyou very much for the information.

[@SeaHuang-w6j]

Only when patreon is unlocked can workflow be download?

[@TheFutureThinker]

example workflow is right in the Github. if you really have watch this video you will know. Don't have to join my Patreon if you don't want to.

[@MrPer4illo]

Nice. How long does it take to generate a 5 sec video?

[@kalakala4803]

@@MrPer4illo depends your gpu, and overall of your hardware setting. You have to test.

[@WhatTheAI-y1i]

having problems with mimicmotion getposes

[@nirdeshshrestha9056]

What's the gpu needed m

[@TheFutureThinker]

I prefer VRAM 16GB +

[@petEdit-h9l]

Is it better than viggle

[@TheEconoVision]

You should make an app of this.

[@TheFutureThinker]

Thanks, but its hard to maintain for diffusion AI niche, I rather do LLM model with better business model.

[@voxyloids8723]

@@TheFutureThinker will be a tutorial on this )))) ?

[@TheFutureThinker]

@@voxyloids8723 I am sorry, which subject? LLM ?

[@UnsolvedMystery51]

Should be easy to create such app right? See many Discord doing it

[@bgmspot7242]

Anyone please tell im new to this, Is it better than animate anyone and viggle ai ?

[@TheFutureThinker]

animate anyone kind of robotic movement sometimes... that's what I experienced.

[@bgmspot7242]

@@TheFutureThinker Thank you for responding. Is it better than viggle AI?

[@TheFutureThinker]

@@bgmspot7242 well, AI and tools, is like a car. depends how you set and drive it. viggle saw their animation for anime character right? no comment how other company do

[@voxyloids8723]

WOW ! How did you install it!? !?!??!

[@voxyloids8723]

Oh you install from Kijai GREAT ! I had some issues with AIFSH repo. Will try kijai now

[@TheFutureThinker]

​@@voxyloids8723 hey, yes. Actually I have installed both. But Kjai custom node have better indicator, and define each features in the framework in separate custom node. So I like it , as always use his created custom node. AF version is working also. But the first time it run, it feel like freeze, but actually it is downloading the SVD model in the backend. Thats what I experienced.

[@sudabadri7051]

Great tips both 👌

[@ryansenger937]

@@TheFutureThinker have you created a comfyui for this? Would I be able to download it from your Patreon?

[@paologalati4060]

What ide?

[@TheFutureThinker]

ComfyUI

[@Vashthareaper]

nothing really works as they show in the examples, too much deformation even after following exact settings and steps, in their example videos there is not a single deformation

[@TheFutureThinker]

Research demo cherry pick. Thats why we have to try on ourselves.

[@dragongaiden1992]

Friend, how much does it cost to access your workflows? And the other thing is, what graphics card did you use? I wouldn't want to pay for something that won't work on my computer.

[@TheFutureThinker]

Ahbibi, maybe you can join our Discord group to understand first. Or play around with the example workflow, which I pointed in the video. i dont really need whatever Patreon joon in if you are not sure.

[@reaperhammer]

This is cool, but the vram requirements are insane lol, V100 for 16 frames according to github... 😢

[@crazyleafdesignweb]

😉 when everyone is running on Comfyui, and you are setting up for server config?

[@InnovateFutures]

Bro. fact check before you comment V100 is the server training requirement.

[@reaperhammer]

Hmm... well the github page makes mention of the v100, nothing about that being the training requirements as far as I understand it... I kinda wish these projects would make it more clear on the system requirements... and performance expectations

[@InnovateFutures]

@@reaperhammer ok, so you are setting up the python script version.

[@sorijin]

Patreon workflows zzz

[@TheFutureThinker]

right on the description, for ya freebie ;)

[@sudabadri7051]

Damn these guys wanna complain even before reading the description 😅

[@kalakala4803]

Drama maker🤣🤣🤣

[@TheFutureThinker]

@@kalakala4803