How to use ControlNet. ControlNet for Stable Diffusion Tutorial.

[@sebastiankamph]

Download Prompt styles: www.patreon.com/posts/sebs-hilis-79649068 Please support me on Patreon for early access videos. It will also help me keep creating these guides: www.patreon.com/sebastiankamph

[@ianhmoll]

Can't find the free button to download. It needs to donate on Patreon to download?

[@ianhmoll]

@@sebastiankamph find it. Thank you :)

[@alecDeloyed]

I recommend you cascadeur to make model animation, it uses AI that makes more fluent animation.

[@Distty]

Thank you very much.

[@89gxzhj9v]

​@sebastiankamph bro help me every time I try to make images by using control net , the say out of Cuda memory even when I make every sitting on low quality!!!

[@kingkhalagar2836]

Good video, but I'm surprised you didn't cover the new stuff in ControlNet like Open Pose Hands + Face. Those seem like by far the biggest points of interest now, as you can easily get near perfect hands and faces pointed exactly how you want. Would love a video focusing on it, since I've been tinkering with it and it works, but I don't know if I am actually using everything as well as I could be haha

[@Elwaves2925]

I haven't updated yet and didn't even realise that was a thing in ControlNet 1.1. I agree that it would be great to cover that as it seems like a great feature. Maybe he didn't want the video to be too long, so decided to cover install and the basics?

[@kernsanders3973]

It's mind boggeling basically no one is talking about ControlNet1.1 with all these new additions. The Lineart one as well is mind blowing, ALL my old line art I ran through it and can generate the exact line art image filled in with any style I wish. It's mind blowing. This should be top news. Also they just released their new TEST version where you can control lighting, its insane how good it is. I can specify exactly where the light source is located, the intesity and what color. As well what in the picture should be illuminated to what degree from the light. Example making a sword glow from all the light bouncing off it. This stuff is a massive game change for anyone independent

[@blackvx]

​@Kern Sanders I'll check this out, thank you!

[@shadowdemonaer]

@@kernsanders3973 Thank you for speaking about this in more detail. I am very excited about this. I can't wait to make my images look right!! I have just honestly been settling for whatever pops out and just using whichever images suck the least, but this makes me very happy. This changes the game completely!!

[@hiddendrifts]

@@kernsanders3973 i've never given controlnet a try since my version seems to be unable to find the controlnet models properly fsr. but by the sounds of this, it's worth the effort to do some tweaking and get it working properly

[@Stand_By_For_Mind_Control]

Man you have been THE most helpful person towards getting me started here. Thanks a lot for the very easy to understand tutorials. This is all a lot harder than I thought it was going to be.

[@sebastiankamph]

Happy to help, and glad that you find my guides useful! I'm sure you'll get the hang of it 🌟

[@RiiahTV]

im 1month in and ive been getting jiggy, how are you 3 months in brother

[@elaxel1469]

I get easily frustrated with this complicated steps... but I want to THANK YOU for the clear way you give the explanations, and how calm you talk!

[@sebastiankamph]

Happy to help, glad you're enjoying the content!

[@Cylonick]

I'm only here for the jokes.

[@brainwithani5693]

The jokes are a big plus.

[@NEETAZ]

​@@brainwithani5693and multiplication

[@crax83]

I'm here for the amazing Swedish accent! And the jokes 😁

[@channelschannel]

you are the best! i spent lots of time trying to fix controlnet and your video saved my time

[@1hitkill973]

Mine wasn't 1:1 result of the OpenPose. Maybe because I didn't have any models for the ControlNet. But I absolutely love this tutorial. It opened up new doors for me to explore. Thanks a lot.

[@klassetool3517]

i dont have any models, do you know how to get them

[@stevoshilling4409]

"Quality of life" updates 🤣🤣🤣 Half of us in the digital industry are literal walking zombies mate, what quality of life?

[@nighthaven6244]

Ok it... doesn't work. I put all the same, except maybe the checkpoint because i don't know where to find it but i tried anyway multiple checkpoint so i guess that doesn't come from here, I didn't entered style yet but as i uderstand it's just automatic prompt so nothing to do with controlNet... I am lost, it generate a picture but the position of the model is completely random and doesn't follow at all the picture on controlnet with all the settings... :(

[@fixelheimer3726]

Hi, when I saw the video title, I hoped you would cover the new preprocessors and what works best and especially with what model to achieve XYZ.

[@sebastiankamph]

I felt the video would be too long, so mainly covering the basics and then doing follow-up videos that are more indepth. That was the idea at least 😅

[@coda514]

Great content as always. Very useful information. By the way, two antennas met on a roof, fell in love and got married. The ceremony wasn't much, but the reception was excellent.

[@sebastiankamph]

Boom, nailed it! 💫

[@Mack_au]

Controlnet has no impact on images. There are a lot of complications for no reward.

[@DizzyDJW]

I wish I could see what he was doing, but I was flash bombed at least 12 times during this video. I have no eyesight left.

[@Ocodo]

Great video. The new ControlNet looks very promising.

[@TheSchwarzKater]

Yeah but as soon as the scene is more complex, like two people hugging, it's really hard to not get multiple arms and legs. I've yet to figure out how to do it properly.

[@spartanx169x]

Its now been 6 months, have you made any progress?

[@playmaka2007]

Come for the dad jokes, stay for good AI content.

[@sebastiankamph]

We all love a good dad joke to brighten the day

[@StyxGamingProject]

I believe what Sebastian forgot to mention is that OpenPose is for SD v1.5, if you will try to use it with v2.0 or 2.1 models, it will not operate, so make sure to use models trained for 1.5!

[@Crealter]

Not Working with SDXL Model :(

[@FrankieRockett]

Despite him saying in the video very very clearly at 9:28 that all of the presets are available for "Free" - guess what? They are NOT FREE. They are in fact all locked away behind a pay wall. Not nice, not fair and not necessary. If you state something you are offering is free then either give it free of take down the video and it's dishonest statement.

[@sebastiankamph]

They used to be free. I changed it to be a part of my Patreon offering to pay the bills. Sorry to hear that you're upset by this. I can't take down the whole video for an old statement that is wrong sadly.

[@20xd6]

earlier you showed how to add VAE to the UI at the top. Is there a way to add 'clip' at the top of the UI as well? For another video idea, I think it would be cool if you went over each controlnet model and how to use them. Openpose is fairly straightforward, but the other ones are a lot harder to use. I don't even know what half of them are supposed to do. (I'm looking at you, Hed, Seg, and Mlsd!)

[@bambinoesu]

Still don't have the hardware to run stable diffusion smoothly 😑

[@CoconutPete]

As an SD noob I think there's a need for making tutorials customized to specific types of SD such as A1111 tutorials, ComfyUI tutorials, Forge, etc. The biggest confusing factor (for me) is trying to figure out which additional files are required for certain things to work. For example, I tried to install the SSD-1B "thingy" (Lora?) but noticed if I selected it in the "Add network to prompt" drop down it made my renders look horrible. If choosing "none" the renders looked great. Just a random example but an opportunity for content ideas for the experts lol

[@MrNamanix]

Hey Sebastian, Quick question. In your interface I see buttons 16:9, 4:3, 1:1. How do you get those? It would be amazing to be able to switch between those modes with buttons.

[@sebastiankamph]

Aspect ratio, search in extensions tab

[@towlodar]

Can someone recommend an awesome tutorial for Mac? I would greatly appreciate it.

[@tradinglikeanoob]

openpose models showing "none " plz help

[@johndrippercodes]

amazing

[@ethnobeat-u4n]

Big plus, ok best wishes from your neighbour country

[@kernsanders3973]

Thanks surprised no one is talking about ControlNet1.1. The additional Line art controlnet is a massive game change for ALL line artists everywhere. This news should be everywhere but barely hear anyone talking about it. Never in history did we have a A.I. able to take a line art image and perfectly fill it in to any style what so ever, until now, yet hear no one talking about it. Also ControlNet has just released a TEST version of their lighting control, so you can take any image and control the complete lighting composition for that piece. You can specify where the light source is located as well what in the picture is being illuminated. The tool is freakn insane. I'm honestly surprised barely anyone is mentioning it.

[@ChrisCapel]

Whoa, the lighting control sounds crazy. Do you have a link to some kind of video? Is it available yet?

[@MrSongib]

2:30 There are some options for smaller files either from Civitai or Huggingface.

[@elyssafleur]

When I tried controlNet, it's like ControlNet is not activated at all. Whever if it openpose or canny... I tried three different checkpoint but I get the same result. Does anyone has any idea why and could help me?

[@moibe182]

Wow, I'm just starting with Stable Diffusion and this is just mind blowing, we had all this tools without knowing it, for me it's like discovering fire, and all that knowledge is thanks to your tutorial!

[@sebastiankamph]

Happy to hear that. Welcome aboard! 😊

[@rastanchacha]

I have to say, the dad jokes made me subscribe

[@sebastiankamph]

Glad to have you aboard!

[@MikevomMars]

Jeez, I am so sick with the speed of changes - every week new features, some explained, some not, while the GUI gets more and more clustered. It's a horrible and quite chaotic situation right now!

[@IceMetalPunk]

No mention of the model improvements in 1.1? Most of them are just "we improved the data set and model training for better quality", but there's a few standouts: (1) The segmentation model now supports COCO protocol, which means an extra 182 colors/categories. (2) OpenPose can now track face orientation, and can also track hands better with more detailed OpenPose skeletons. (3) Lineart model and preprocessor are entirely new, and can convert lineart into images, including hand-drawn lineart (like a more structured "scribble"). (4) Shuffle model and preprcessor are entirely new; they basically distort the input image enough to keep the style but lose the details, then the model was trained on those; so it's basically ControlNet-based style transfer. (5) Instruct-Pix2Pix is brand new as well, and is trained with a mix of normal prompts and instruction prompts, so it can do instruction-based edits entirely in ControlNet, applied to any base model.

[@rbscli]

ip2p is really interesting, I've been playing with it and it kind of amazes me every time. Have to check the new improvement to poses with face and hands as well.

[@MysticMango09]

In my case, i dunno why the yaml files didnt download automatically.

[@thegreatujo]

I can't download the styles. You say in the videos they are ffree but when I click on it it asks for patreon membership (paid)

[@huevonesunltd]

Do you know whats up with controlnet and SDXL ? the models that are for SDXL i got don't even work

[@darshank8748]

A but weird you haven’t already done a Stable Diffusion XL video

[@sebastiankamph]

Since it's just a beta inside locked systems (and some api access), I felt it was more interesting to do it when they release it open source to everyone. But it's been taking longer than expected.

[@darshank8748]

@@sebastiankamph Ok, thanks for the answer. They are taking their time. According to Emad they will release the model with some fine tuned version because this was one of the gripes of the first version of SD

[@BetterHumanz]

Hi, I have tried to figure this out but can't. Under ControlNet, there is no "ControlNet Unit 0" I only have the "Single Image" option..any help would be much appreciated

[@wrayday7149]

Wild. Went to check my folder and the large files weren't there... guess that explains why it never worked. I feel like the person in those "THERE HAS GOT TO BE A BETTER WAY" commercials.

[@AkshatKumarSrivastava]

Hey everyone I hope all of you are doing well. Can anyone help me with this error : RuntimeError: "LayerNormKernelImpl" not implemented for 'Half' ? I recently download and installed stable diffusion and everytime I click the generate button the above error is displayed

[@lanoi3d]

It's hilarious how some people say AI art doesn't take any effort. Boy how little they know! This makes conventional 3d modelling programs and advanced photoshop seem like a walk in the park...

[@acbattery]

Very good Tutorial. I created a pose similar to a "Handstand" Position. However, i tried lots of different models and prompts but all results are more than bad. Why does it seems almost impossible to get an image of a person doing a handstand although ControlNet has an very exact pose for it?

[@miketeok]

I don’t usually subscribe a channel, but I saw your prompt style is free, I subscribed instantly, this is the only way I can show my respect as a cheapskate

[@ObaedaKorani]

Thank you so much my friend

[@slowfire1]

Nice tutorial, but still can't get stable diffusion to create the image i want, even when using ControlNet, why is this? All i want is to create an image from the point of view of a person being pinned to a wall, similar to how VI pinns Caitlyn to a wall in arcane, in that one scene.

[@fullstackwithsantosh]

I noted that a1111 automatically downloads preprocessors. I wonder if it will download the openpose models as well if I have selected them from the dropdown menu.

[@tothetopvisuals]

ControlNet 1.0 was already working great for me. But I had to adjust some settings to avoid some issues. 1.1 eliminated those issues and everything work so much better.

[@LeeDesignStudioYT]

aside from the great instructions/tutorial...thanks for making me lol ;D

[@toddz2994]

this video is so helpful! thx

[@sebastiankamph]

Glad it was helpful!

[@scottgust9709]

Why do you prefer Mikubill over Automatic??

[@nick19950]

i have a cuestion, Is stable diffusion the same as automatic 1111? what´s the diference between a both? someone explain it for me, please

[@tiu7]

I followed all the steps correctly but the control net doesn't show up in "settings" HELP PLEASE

[@lizng5509]

Thank you for sharing!! It seems the T2ia color and style models don't work on this Control-net version.

[@FLEXTORGAMINGERA]

All of good stuff and i cant even run on rtx 2060 6gb 😢

[@Anonymouse007]

at 9:29, my webui don't have styles, how do I get it? edit, okay I found the styles and installed but the output not that good like yours.

[@DittMods]

Does LoRa not work when using controlnet? I get the same face with LoRa and without LoRa, when I only want her face from LoRa and her pose from openpose

[@disordiamercyy3665]

Love your content but you talk like the sloth from this animal movie :D Still love your content. Flash.

[@wrillywonka1320]

control net doesnt work. i tried but if says i have to put no half vae or something. i even reinstalled it. i dont know how anyone is using this

[@RegularAdventurer711]

Thanks for your great videos! I hoped you could have unlocked the content

[@artofgarduno]

Does this for in stable diffusion easy 3.0

[@Robstercraw]

I get Warning: Failed to install mediapipe, some preprocessors may not work.

[@Ceireavne]

You know why I say mucho to my spanish friends? It means a lot to them.

[@sebastiankamph]

Excellent! 🌟🌟🌟

[@user-jk9zr3sc5h]

Ah the extension doesnt show up if you dont have the models- interesting

[@rizkyekydev]

It's in localhost right? Is it heavy ? what GPU are you using ?

[@sebastiankamph]

Rtx 3080. Works with any gpu 4+gb vram

[@CoconutPete]

I enjoy the jokes

[@oliviera1983]

please more zoom in videos. Thank you! :D

[@godswillomonkhodion6393]

What's a minimum system I can run the SDXL on?

[@Oxes]

does anyone know why my stable diffusion disables extensions? isn't there a webui version where every extension is ready ? *** "Disable all extensions" option was set, will not load any extensions *** that's what preventing from loading extensions anyone know why?

[@musicandhappinessbyjo795]

Hello, I have been using control net open pose(full). The general structure seems to be accurate but the position of finger get to totally ruined for some reason. Also at times I get different position as well. I am using the realistic vision, 16:9 ratio resolution. What is that I am getting wrong.

[@YammyMarcel]

When i try to download control net using the URL, it doesn't work and tell me,"AssertionError: Extension directory already exists: D:\Stable-Diffusion\stable-diffusion-webui\extensions\sd-webui-control net" but it's just an empty folder. When I delete the folder to redownload it, it tells me,"GitCommandError: Cmd('git') failed due to: exit code(128) cmdline: git fetch -v -- origin stderr: 'fatal: detected dubious ownership in repository at 'D:/Stable-Diffusion/stable-diffusion-webui/tmp/sd-webui-controlnet''. is there any way to fix this?

[@CMDRRustyDog]

I notice here you rendered in 1024x1024. It's it better to stick to 512 or768 as the models are trained for those resolutions? This is what I've heard. Is it then ok to go higher without suffering a quality loss? Thanks for the great content btw, subbed.

[@MugiwaraRuffy]

At some point, StableDiffusion will just a) randomly refuseto obey to the openpose settings at all b) does not even generate anythin anymore because of a very cryptic error: "devices' argument must be DML" Very frustrating Also, when it actually works, the poses are very weird sometimes. But when you just can not try again, because SD doestn let you, it's VERY annoying. Using A1111 btw

[@caseyj789456]

Vdo interessatn maus qu'est qu'il marmonne ce petit vieux..

[@MADVILLAIN669]

Hey Im following your guide, but I dont have a list of models after clicking 'enable', it is simply not there and says 'none' and no other options available

[@sebastiankamph]

Did you download the models and put them in the folder specified?

[@MADVILLAIN669]

@@sebastiankamph oh thats right! I was skipping that part because I had already used your excellent installation video, so I thought that was unneeded, thanks!

[@bryanbigler2261]

How did you increase the weight at 9:27?

[@gomezabel8880]

I love your tutorial. I just beef up my computer and installed stable diffusion a few days ago so I guess I'm a newb. I was able able to install controlnet however :( installed stable diffusion Auto1111, and about 12 extensions, but .....I'm starting to believe the extension for controlnet is removed from the Github repository maybe recently, am I going nuts? Are you seeing it the extension on Github?

[@avetmimarlk9934]

Before installing the Controlnet, how do you install Stable Diffusion? There is a chance to use it on stablediffusionweb but it's not screen that I see here nd it always gives "error". How do I get this? Can anybody help me?

[@Silver_Nomad]

It's not even 2.0, and it already gets a lot of new preprocessors and models. Interesting, why all the models have dropped in size so much? Models for 1.0 were 5,71 gigs in size, while models for 1.1 are just 1,45.

[@reasonablyargued9003]

I just updated from 1.0 to 1.1 today and can no longer run scripts (specifically, Prompt Matrix) with controlnet. I reviewed the readme and confirmed the "allow other scripts to control this extension" box is checked under settings. All the controlnet models work fine otherwise, it's only when scripts are enabled that CN is bypassed. Anyone else having this issue?

[@TeamPhlegmatisch]

@Sebastian Kamph I get an error message when running with sdxl model. I updated to the latest version, but it only works with SD models. I get "mat1 and mat2 shapes cannot be multiplied" Any idea why?

[@rproctor83]

Why isn't the tile model working as described? Whenever trying to use I simply get enlarged blurry version, it never attempts to redraw tiles. No matter settings. Updated everything. Followed multiple guides to the T. Double triple checked everything.

[@jolopukkii]

Thanks for the vid! I managed to install Control Net and few extentions, BUT Stable Diffusions window continues to show the default interface plus the Control Net interface at the bottom.. And the ControlNet doesnt work..

[@EYYYYYJustin]

the list of models next to processor is empty

[@greenjacketguy3759]

Thanks you I've been trying to find Control net 1.1 videos, Can you do a video covering the new Controlnet features/processors in 1.1 . They added quite a few and the Gitpage is kind of confusing since it's mostly in Tech speak. The various Line new art features, IP2P, Content Shuffle and Various soft edges model are new

[@SuperGari2]

cool, but how come I cannot see any list of Model after checking on Enable? any idea?

[@MADVILLAIN669]

hey I have the same problem, no list of models

[@mattlegge8538]

dude, Thanks heaps for your tutorials, I have a question I can't find an answer to: How can I batch generate pose images from a bunch of frames? I have done it accidentally in stable diffusion, but I can't replicate it and it's driving me nuts.

[@treydaddyman]

Thanks for the tutorial!

[@fernandomasotto]

thanks Sebastian for the info. I updated Automatic1111 controlnet to 1.1, but i still can use my old fp16 safetensor models, right? Or do i need to update to 1.1 versions?

[@Epicfuzz]

the github for controlnet is not loading anymore

[@sebastiankamph]

Works for me

[@lyonstyle]

beginner friendly as always, good job on this video Sebastian!

[@grillodon]

But there is no longer a preview of the annotator? Especially for the canny?

[@AtheistExpert]

does this work on 2.1 checkpoints? (noticed theres a 10gig file for the 2.1's) and what can you speak to those in terms of being... better?

[@BauldyBoys]

Anyone know if 1.1 supports colour palette models? I haven't been able to find one and the old version is breaking my gens.

[@rebelape4257]

im using collab how do i install the models, i have open pose editor but the only options are listed in the processor tab . if i select open pose i dont have any options under the models tab

[@mimirtells]

9:25 how did you updated the weight that easily? Is there a shortcut or a macro?

[@Younoob633]

Should we use .pth files or .safetensors files (smaller size) as shown on other KZbin channels like Albert Bozesan?

[@WhoIsJohnblack]

Sir I am totally lost, I saw your a1111 install video, I understand is the ui for stable diffusion. If I installed that, do I need to add controlnet thingy majig or what lol?

[@sebastiankamph]

ControlNet is an extension for stable diffusion. This video adds it to your a1111. So if you want to use it, you need to add ControlNet

[@WhoIsJohnblack]

@@sebastiankamph got it. What about if I see still images online of an image with the pose I would like to use for a character I draw? Would I add it to controlnet as a model or something? Im sorry i am absolutely new to all this

[@andresz1606]

Idea for ultra premium, VIP GoldMembers only video: how to merge openpose with charturner LoRA for fine-tuned characters turnarounds. That would be great.

[@MsDario66]

Hi, I followed your tutorial to the letter but I can't get the same result, I always get women's faces appearing. What am I doing wrong? Thank you and congratulations the tutorials are very interesting and effective.

[@Jinsomedia]

Hi, did you fix this finally? If yes, do you remember how? I have the same issue