Next level of AI Animation. ControlNet and DeForum in Stable Diffusion.

[@MarylandDevin]

When I change my Preprocessor setting under CN1 to Canny (or anything) I have no other options in the Model drop down next to it. It stays None, and there are no other model options even after hitting refresh button next to it. Is this something I need to download separate?

[@MarylandDevin]

Yes, I had to down load them. If you're having the same issue

[@Geekatplay]

yes, you need download models and preprocessors

[@chrisreid2021]

Hey seems like a pretty common problem from reading the comments but any clue why this is not working for me? I'm familiar with controlnet and deforum however when I try to use them together like in the tutorial I do not get any motion on the output, it's like the control net settings are not applying. I have double checked everything multiple times, Cadence is on 2 so it's not that. When I check my output folder it shows it has correctly separated all the frames for the input vid so I have no idea why it's not applying to the frames for the output. Control net and everything and is working fine everywhere else so I'm super stumped. I'm also using 2 controlnet exactly the same as shown in the video and they are both enabled and set on ControlNet is more important, they are just simply doing nothing

[@naiadsnest]

I had the same problem. In my case, it turned out there was some conflict with the Additional Networks and Dynamic Prompts extensions. I disabled them in the web UI (Extensions -> Installed -> uncheck the boxes of sd-dynamic-prompts and sd-webui-additional-networks and click 'Apply and restart UI' orange button) and ControlNet started working without any issues. If you also have those extensions installed, you might be facing the same problem. I hope this helps! :)

[@Geekatplay]

Use batch processing. You can take video, split on frames and use folder with each frame as batch folder in ControlNet, Styles ...etc

[@belshazer]

I had the same issue and then I changed under "keyframe" tab the animation mode from "2D" to "Video Input". this worked for me using all the other steps in this video. Lots to still learn on how to better control it. Good luck!

[@MatichekYoutube]

@@belshazer ya, had similar problem - ControlNet was not activating unless I did Video Input - but after 1 month I guess they updated the code

[@LostBoysGaming]

Love your videos my friend but please use darkmode for your interface, the white screen is blinding.

[@Toshiboyai]

when i load the control net preprocessor no model types come up, how come?

[@zagzagel2626]

I don't know what's wrong but for me controlnet doesn't have any effect to anything. Any ideas?

[@Geekatplay]

be sure you click "enable" check box

[@AnotherPlace]

same controlnet is not affecting the output, it's enabled tho

[@kinokotensai416]

@@Geekatplay I also faced the same issue. I checked Enable for Deforum > ControlNet > CN Model1, CN Model2, but the generated video seems to have ControlNet applied only to the first frame, and it doesn't appear to be functioning thereafter. Is there any solution to this problem? By the way, ControlNet is functioning correctly in img2img.

[@BenSake]

@@kinokotensai416 I have the same exact problem, only the first frames gets affected by Controlnet, the video turns out with a static pose.

[@chrisreid2021]

I was just typing up a post for this, I have the exact same issue, I know everything is set up right and controlnet works everywhere else

[@נתנאלרותם-פ3ה]

thanks! Can you kindly share the settings file? for this, and all other vidoes?

[@MatichekYoutube]

If controlnet is not working, you probably have errors cause of other extensions - I fixed this - so I disabled all other extensions - besides Control net and Deforum. And then it works ok.

[@Geekatplay]

thank you for suggestion

[@unreal_unit]

What you think, Deforum+ControlNet make better animation. Or WarpFusoin?

[@Geekatplay]

Personal opinion at this moment warpfusion providing better end result, however it is heavily unbalanced by how difficult it is use for average creators and it is requirement of colab

[@cmdelavega]

Thank you for the tutorial! It seems like control net is taking any effect. I have a video of a dancer that I'm specifying for the video path in controlnet but the final output is static - didn't follow any of the controlnet models (depth, canny or openpose). Any suggestions would be helpful. Thanks!

[@Geekatplay]

Thanks for sharing!

[@MarylandDevin]

lol his response. But seriously, im having similar issue. Did you have any luck?

[@spaceMatushka]

Привет, подскажи пожалуйста, почему у меня в настройках controlnet в графе "Directory for detected maps" указано просто "detected maps", а в графе "extra path to scan for controlnet" пусто? Хотя устанавливал также как на видео.

[@ad2m989]

You are the most sincere person. Thank you ❤I hope you put more to use Thank you, Mr

[@Geekatplay]

Thank you, I will

[@marotaoYTB]

Hello, i installed deforum extension but it dont appear in the tab, you know why?

[@Geekatplay]

be sure extensions is enabled in "etensions" tab (check box marked), if you have many extensions, then tab maybe hidden off browser screen. And also try to restart server.

[@sordidloam]

I would love to see a video on how to interpolate between a set of already rendered images. I’m having some issues figuring out how to do this. For instance, do I still need a prompt? If so do I need to have the strength down so that it doesn’t overly affect the quality of the images? I really just need the images to blend/ morph into the next.

[@Geekatplay]

great suggestion,

[@EwokS-r2r]

i couldn't get the '2d/3d' animation modes to use my controlnet input video, but using 'video input' animation mode worked. i dont think i missed any steps...any tips?

[@matty_verse]

Hi @Vladimir, great video. I was wondering if you could explain how you used the 2D animation mode, but with the video initialization of your face. I didn't catch how you were able to do this at 21:40. I can only get video initialization to work with the video animation mode and get an error after the first frame when combining 2D Animation Mode + Video initialization. Are you doing this in the openpose image path?

[@matty_verse]

Additionally, for the CN pre-processor resolution, if I'm using a non-1:1 aspect ratio, should I put the CN pre-processor resolution as the biggest dimension (i.e. 1280 when using 1280x720 under run)?

[@Geekatplay]

i will going to make video with detailed information about all settings

[@EwokS-r2r]

i have the same issue, did you get it to work?

[@NeoXSaibot]

good video, how can I get this vibrants colors ? thanks

[@Geekatplay]

use in prompt : "vibrant colors"

[@NeoXSaibot]

@@Geekatplay thanks

[@ChrisMattar]

hey vlad i keep getting this error, any idea why? I copied everything like you Error: ''NoneType' object is not iterable'. Before reporting, please check your schedules/ init values. This only happens if controlnet is enabled.

[@BenSake]

Thanks, very informative video. I have an issue with this, controlnet openpose succesfuly extracts input frames from video however only the first frame gets processed by deforum. After that, instead of taking the second frame from controlnet input frames, deforum just keeps on going and using the first frame for the whole video. Any idea what is the issue?

[@Geekatplay]

never got that problem, be sure you dont have any images set as reference in init tab

[@BenSake]

@@Geekatplay I managed to get it working by setting deforum strength schedule to 0 but then I get very inconsistent frames.. Does Controlnet really work for you with strength values of 0.6 or more? I have the same issue on google colab and on mac. Maybe it's different with windows..

[@SofíaÁlvarezRocha-c3b]

It's posible to make this with a video? I don't want to generate the video I want to esit6the video I already have with this

[@vzdroidplayz]

Is there a way to animate one existing image into another existing image?

[@Geekatplay]

yes, i have couple videos coming up

[@vzdroidplayz]

@@Geekatplay thank you, I kind of figured it out using interpolation

[@joelcolon3688]

i followed the instructions on the video, but when i try to generate image, I get this error, Error: 'Video file format 'r: ew folder eudap' not supported. Supported formats are: ['mov', 'mpeg', 'mp4', 'm4v', 'avi', 'mpg', 'webm']'. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \. Full error message is in your terminal/ cli. I made sure I was using / istead of \ not sure what schedules/ init values are, I could not find that in the video. Can you please help. Thanks.

[@Geekatplay]

you have space in name : "new folder"

[@DEEPAKJOSHI-oo9bz]

I followed the exact same steps, still when i am generating the video, its creating a whole black image seq along a rectangle (and the shape is morphing). can anyone help me in this?

[@Geekatplay]

be sure you dont have any spaces in names

[@DEEPAKJOSHI-oo9bz]

@@Geekatplay sorry but which names? If you are talking about the folder name, its "man_dance".

[@kemikemi756]

I've never used stable diffusion but I love your videos for information

[@Geekatplay]

thank you for your support!

[@TheMrkeksomatic]

i can't choose/ insert a model for the preprocessor.. any ideas? thank you!

[@Geekatplay]

be sure you download yml and model files, it is my other video, how to install control net

[@TheMrkeksomatic]

@@Geekatplay unfortunately can not find the link you used in the other video to get the yml and model files

[@ifoundthistoday]

thank you so much .. exactly what I needed to start !!

[@Geekatplay]

thank you!!!!

[@HobbyBobby.]

awsome tutorial but i am having trouble making the video input work, i go step by step as u explain, but it seems it only generates from the prompt even if i set it to Control net is more important

[@Geekatplay]

be sure no spaces in path and names

[@jonparravicini-g5n]

you are the #1

[@Geekatplay]

thank you, now we need tell that to KZbin ;)

[@vinifabretti7977]

This Controlnet tab didn't show in my Automatic1111 Deforum tab. I don't know what I am doing wrong here.

[@Geekatplay]

it is hard to tell without seeing problem, here possible issues. you may have too many extensions installed and tab with control net hidden (overflow), you may need restart you SD server. You may not correctly installed ControlNet, not all files copied into system location. You may have some browser addon that effecting correct load of the controlnet

[@Enricii]

Great! I was messing with deforum + controlnet just yesterday. I find it the best compromise between easiness of installation and good results Vs SD-CN animation, text2vid and warp fusion (this is difficult to install imho). Do you agree?

[@Geekatplay]

it is easier to use with Deforum

[@livclark9871]

Why is my controlnet not working, the image is very different from my original video

[@Geekatplay]

be sure you "enable" your control net, checkbox.

[@michail_777]

I'm surprised.I've done masks through hybrid video and it's not bad either.But this,two,or maybe 5 videos at a time that can be processed.This is super!!!! There is such an instrument "Depth Warping & FOV" I set MiDaS/Zoe weight 0,4. Also there is "Optical flow" in the tab "Coherence", I believe that flicker can be reduced. And the coherence is much better with these tools. But I did it with "hybrid video". I'll have to try it here, too. Thanks again.

[@Geekatplay]

i need play more with masks, going to work with greenscreen and masks

[@hairy7653]

i cannot get controlnet to do anything in deforum, ive tried everything that i can see in the vid...great vid btw

[@hairy7653]

i got it working, i had to use 2 control nets not one!

[@Geekatplay]

yea, sometimes it is bugging on me, reload UI will help to show all fields, but UI interface buggy and may have some problems

[@daetojekf5973]

@@hairy7653 how did you deal with this problem?

[@evdagiyik2635]

how did you solve the problem it gives a result outside of your control network i did everything as in the video

[@hamzadakhlaoui857]

hello, I just recently discovered this world and I like your tutorials too much, thank you for the content you make. I wanted to ask you at the beginning of the video at minute 3.36 you talk about the right path of ff-mpeg must write what appears to you ?

[@Geekatplay]

path it is where ffmpeg app was installed, be sure it is using no spaces path name. example: c:/instaldirectory/myapppath/ffmpeg.exe

[@tonatiuhsilva1859]

Wow, amazing video, thank you so much for explaining it, I have a question, how can I use two models in controlnet (using different images)the first model needs to be applied starting in frame 0 (which worked ok) but the second model I need to be applied in frame 120 for example use the following: CN model 1: Weight schedule 0:(1) Starting Control Step schedule 0:(0.0) Ending Control Step schedule 0:(1.0) ControlNet Input Video/ Image Path C:\Users\Tona\Pictures\to edit\image_1.jpg CN model 2: Weight schedule 120:(1) Starting Control Step schedule 120:(0.0) Ending Control Step schedule 120:(1.0) ControlNet Input Video/ Image Path C:\Users\Tona\Pictures\to edit\image_2.jpg but I get the error: Error: 'local variable 'value_is_number' referenced before assignment'. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \. Full error message is in your terminal/ cli. It seems like the Weight schedule need to be the same in CN model 1 and 2 = 0:(1)

[@mynameisChesto]

Hope you've figured this out, but to my knowledge, you can't start a string from a higher value than zero; that is, your keyframe can't look like 120:(0) It has to start from zero and then keyframe, so it looks like this 0:(0), 120:(1.0) Hope that helps!

[@MrSlamdung]

Im using deforum trough auto1111 on Runpod, but the UI doesnt recognizes the path for my Init files :( Great video dude

[@Geekatplay]

be sure no spaces in the path, it will break it

[@anthonyhsu9509]

Great video!! When I combined Deforum with ControlNet, this came up: Error: 'expected str, bytes or os.PathLike object, not NoneType'. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / instead of \. Full error message is in your terminal/ cli. Has anyone had the same issue?

[@Geekatplay]

dont use spaces in naming, path and file names

[@DAAN_MUSIC]

Use / instead of \ even for local Paths to the source video for controlNET

[@Ascended23]

This can mean a lot of things. One case where I got this in the beginning is when I forgot to check the box to generate control images.

[@Nazubal]

Vlad, when you say installed, drop the e sound from the last e in "installed". Say it like installd. Google how its pronounced for further verification

[@Geekatplay]

thank you for your help. i always appetite people who helping !!!!

[@michail_777]

Владимир здравствуйте.Пишу по русски потому,что переводчик парой так переводит что смысл слов изменяется Я делаю всё как вы показываете в видео.В принципе делать много не надо.Прописать промт и вставить видео в CN,а далее это настройки для того что-бы маска держалась хорошо.У меня в CN всё отображается хорошо.Даже 2 строки: одна для видео,вторая для видео с маской.Вторая строка,как написано,экспериментальная.Но у меня генерация идёт,но генерирует просто то,что в промте написано.Такое ощущение что Deforum не видит видео,которое вставлено в CN. Можно было бы сказать то,что я неправильно прописал путь,к примеру C:\AI,а не так C:/AI,но тогда выдавал бы ошибку.Путь прописан правильно C:/AI/...mp4 . Может быть у вас установлена какая-то особая модель к CN в Deforum?Просто в Deforum есть функции которые работают только с определённой моделью. Как я понял иногда надо перезагружать для того что-бы строку показало,но у меня с этим проблем нет. Но я всё-ровно перезагружал, до 4 видео вставлял,результата нет. У вас бывает такое что видео вставляешь,а "Deforum" не видит видео? Если да,то это дело в программе,возможно то,что всё это ещё экспериментально.Если нет,то непонятно почему у вас работает,а у меня нет. Спасибо за понимание.

[@Geekatplay]

я записал несколько уроков на русском, они выйдут очень скоро. я выставлю линк к ним позже здесь.

[@Geekatplay]

kzbin.info/www/bejne/aJS6eaGEmLGkrLs

[@kemikemi756]

I read an article on my phone that stable diffusion has hackers is this any true ?

[@Geekatplay]

some models(check point) that used in Stable Diffusion may have hack code. That allow to execute python script and hack your computer. Some extentions that check for that code is out (anti hacking) as well new format (safetensors) was introduced to prevent that, for older models "pickle" or verification code was added. Be careful where u getting your checkpoint models from. Just like anything on internet from news to videos ;)

[@EddieGoldenberg]

Thank you Vladimir! I love your videos and slava ukraine!

[@Geekatplay]

thank you. Slava Ukraine!

[@twilightfilms9436]

You Yanet explained how to import the video, in a tutorial called “video”……

[@Geekatplay]

ok

[@jamminmandmband]

I have followed all the instructions the best I could. But here is what is happening. The video is just the generated prompt repeated over and over again. What is happening? kzbin.info/www/bejne/o5KUaIeEpbaCkMU The resulting video does not resemble the original in the slightest.

[@Geekatplay]

you need add more weight to original image in control net, also be sure it is enabled

[@姚钦文]

hallo，i follow you steps with settings for deforum and controlnet, but last running i get this error，I wonder how to fix it： Reading ControlNet *static* base frame at E:\sd_webui_new\sd-webui-aki-v4.1\outputs/img2img-images\Deforum-Testing\controlnet_1_inputframes\000000000.jpg Error running process: E:\sd_webui_new\sd-webui-aki-v4.1\extensions\multidiffusion-upscaler-for-automatic1111\scripts\tilediffusion.py Traceback (most recent call last): File "E:\sd_webui_new\sd-webui-aki-v4.1\modules\scripts.py", line 418, in process script.process(p, *script_args) TypeError: Script.process() missing 21 required positional arguments: 'enabled', 'method', 'overwrite_size', 'keep_input_size', 'image_width', 'image_height', 'tile_width', 'tile_height', 'overlap', 'tile_batch_size', 'upscaler_name', 'scale_factor', 'noise_inverse', 'noise_inverse_steps', 'noise_inverse_retouch', 'noise_inverse_renoise_strength', 'noise_inverse_renoise_kernel', 'control_tensor_cpu', 'enable_bbox_control', 'draw_background', and 'causal_layers' 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。 Error running process: E:\sd_webui_new\sd-webui-aki-v4.1\extensions\multidiffusion-upscaler-for-automatic1111\scripts\vae_optimize.py Traceback (most recent call last): File "E:\sd_webui_new\sd-webui-aki-v4.1\modules\scripts.py", line 418, in process script.process(p, *script_args) TypeError: Script.process() missing 7 required positional arguments: 'enabled', 'encoder_tile_size', 'decoder_tile_size', 'vae_to_gpu', 'fast_decoder', 'fast_encoder', and 'color_fix' 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。 Error running process_batch: E:\sd_webui_new\sd-webui-aki-v4.1\extensions\sd-webui-additional-networks\scripts\additional_networks.py Traceback (most recent call last): File "E:\sd_webui_new\sd-webui-aki-v4.1\modules\scripts.py", line 436, in process_batch script.process_batch(p, *script_args, **kwargs) File "E:\sd_webui_new\sd-webui-aki-v4.1\extensions\sd-webui-additional-networks\scripts\additional_networks.py", line 189, in process_batch if not args[0]: IndexError: tuple index out of range 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。 Error running postprocess_batch: E:\sd_webui_new\sd-webui-aki-v4.1\extensions\multidiffusion-upscaler-for-automatic1111\scripts\tilediffusion.py Traceback (most recent call last): File "E:\sd_webui_new\sd-webui-aki-v4.1\modules\scripts.py", line 454, in postprocess_batch script.postprocess_batch(p, *script_args, images=images, **kwargs) TypeError: Script.postprocess_batch() missing 1 required positional argument: 'enabled' 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。 Error running postprocess: E:\sd_webui_new\sd-webui-aki-v4.1\extensions\multidiffusion-upscaler-for-automatic1111\scripts\tilediffusion.py Traceback (most recent call last): File "E:\sd_webui_new\sd-webui-aki-v4.1\modules\scripts.py", line 445, in postprocess script.postprocess(p, processed, *script_args) TypeError: Script.postprocess() missing 1 required positional argument: 'enabled' 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。 Error running postprocess: E:\sd_webui_new\sd-webui-aki-v4.1\extensions\multidiffusion-upscaler-for-automatic1111\scripts\vae_optimize.py Traceback (most recent call last): File "E:\sd_webui_new\sd-webui-aki-v4.1\modules\scripts.py", line 445, in postprocess script.postprocess(p, processed, *script_args) TypeError: Script.postprocess() missing 1 required positional argument: 'enabled' 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。

[@Geekatplay]

try update, i had some problem with resent update, it is working with Deforum but give errors with text2image. Now trying to debug where is error