How to use ControlNet. ControlNet for Stable Diffusion Tutorial.

ControlNet 1.1 for Stable diffusion is out. Let's look it over and I'll show you how it works from installation to image.
How to install Stable Diffusion • How to Install Stable ...
app.posemy.art
github.com/Mikubill/sd-webui-...
huggingface.co/lllyasviel/Con...
Prompt styles here:
/ sebs-hilis-79649068
Support me on Patreon to get access to unique perks! / sebastiankamph
Chat with me in our community discord: / discord
Ultimate Stable diffusion guide • Stable diffusion tutor...
My Weekly AI Art Challenges • Let's AI Paint - Weekl...
My Stable diffusion workflow to Perfect Images • Revealing my Workflow ...
ControlNet tutorial and install guide • NEW ControlNet for Sta...
Famous Scenes Remade by ControlNet AI • Famous Scenes Remade b...
LIVE Pose in Stable Diffusion • LIVE Pose in Stable Di...
Control Lights in Stable Diffusion • Control Light in AI Im...
Inpainting Tutorial - Stable Diffusion • Inpainting Tutorial - ...
The Rise of AI Art: A Creative Revolution • The Rise of AI Art - A...
7 Secrets to writing with ChatGPT (Don't tell your boss!) • 7 Secrets in ChatGPT (...
Ultimate Animation guide in Stable diffusion • Stable diffusion anima...
Dreambooth tutorial for Stable diffusion • Dreambooth tutorial fo...
5 tricks you're not using in Stable diffusion • Top 5 Stable diffusion...
Avoid these 7 mistakes in Stable diffusion • Don't make these 7 mis...
How to ChatGPT. ChatGPT explained in 1 minute • How to ChatGPT? Chat G...
This is Adobe Firefly. AI For Professionals • This Is Adobe Firefly....
Adobe Firefly Tutorial • Adobe Firefly Tutorial...
ChatGPT Playlist • ChatGPT
CHAPTERS
0:00 New ControlNet 1.1
0:30 Installation
3:19 How to use ControlNet
4:09 Find your input with PoseMyArt
7:00 How to use ControlNet continued
12:04 What's new in 1.1?

[@sebastiankamph]

Download Prompt styles: www.patreon.com/posts/sebs-hilis-79649068 Please support me on Patreon for early access videos. It will also help me keep creating these guides: www.patreon.com/sebastiankamph

[@ianhmoll]

Can't find the free button to download. It needs to donate on Patreon to download?

[@ianhmoll]

@@sebastiankamph find it. Thank you :)

[@alecDeloyed]

I recommend you cascadeur to make model animation, it uses AI that makes more fluent animation.

[@Distty]

Thank you very much.

[@89gxzhj9v]

​@sebastiankamph bro help me every time I try to make images by using control net , the say out of Cuda memory even when I make every sitting on low quality!!!

[@Stand_By_For_Mind_Control]

Man you have been THE most helpful person towards getting me started here. Thanks a lot for the very easy to understand tutorials. This is all a lot harder than I thought it was going to be.

[@sebastiankamph]

Happy to help, and glad that you find my guides useful! I'm sure you'll get the hang of it 🌟

[@RiiahTV]

im 1month in and ive been getting jiggy, how are you 3 months in brother

[@elaxel1469]

I get easily frustrated with this complicated steps... but I want to THANK YOU for the clear way you give the explanations, and how calm you talk!

[@sebastiankamph]

Happy to help, glad you're enjoying the content!

[@timmi3701]

Thanks for the updates, always looking forward to when you do them.

[@lyonstyle]

beginner friendly as always, good job on this video Sebastian!

[@digitalboomer]

Brilliant video, the styles are great, your sponsor is a keeper (those pose models will come in handy for storyboards for sure!). Thanks!

[@sebastiankamph]

Glad you enjoyed it!

[@moibe182]

Wow, I'm just starting with Stable Diffusion and this is just mind blowing, we had all this tools without knowing it, for me it's like discovering fire, and all that knowledge is thanks to your tutorial!

[@sebastiankamph]

Happy to hear that. Welcome aboard! 😊

[@stanislavsidorovich6242]

you are the best! i spent lots of time trying to fix controlnet and your video saved my time

[@coda514]

Great content as always. Very useful information. By the way, two antennas met on a roof, fell in love and got married. The ceremony wasn't much, but the reception was excellent.

[@sebastiankamph]

Boom, nailed it! 💫

[@MrArrmageddon]

Thanks really nice and helpful video. I did not realize control net got so much better.

[@miketeok]

I don’t usually subscribe a channel, but I saw your prompt style is free, I subscribed instantly, this is the only way I can show my respect as a cheapskate

[@Ocodo]

Great video. The new ControlNet looks very promising.

[@MaisnerProductions]

great video on the update. thanks!

[@fixelheimer3726]

Hi, when I saw the video title, I hoped you would cover the new preprocessors and what works best and especially with what model to achieve XYZ.

[@sebastiankamph]

I felt the video would be too long, so mainly covering the basics and then doing follow-up videos that are more indepth. That was the idea at least 😅

[@1hitkill973]

Mine wasn't 1:1 result of the OpenPose. Maybe because I didn't have any models for the ControlNet. But I absolutely love this tutorial. It opened up new doors for me to explore. Thanks a lot.

[@klassetool3517]

i dont have any models, do you know how to get them

[@BrogramFilmss]

Man, glad i found your channel, great videos and very helpful, thank you!

[@sebastiankamph]

Glad they're helping! 😘

[@NEETAZ]

Oogabooga little booger

[@optimoos]

very very cool Sebastian!

[@sebastiankamph]

Thank you very much! 😊😘

[@rickarroyo]

Very nice! I'll try :) Thanks

[@tothetopvisuals]

ControlNet 1.0 was already working great for me. But I had to adjust some settings to avoid some issues. 1.1 eliminated those issues and everything work so much better.

[@kernsanders3973]

Thanks surprised no one is talking about ControlNet1.1. The additional Line art controlnet is a massive game change for ALL line artists everywhere. This news should be everywhere but barely hear anyone talking about it. Never in history did we have a A.I. able to take a line art image and perfectly fill it in to any style what so ever, until now, yet hear no one talking about it. Also ControlNet has just released a TEST version of their lighting control, so you can take any image and control the complete lighting composition for that piece. You can specify where the light source is located as well what in the picture is being illuminated. The tool is freakn insane. I'm honestly surprised barely anyone is mentioning it.

[@ChrisCapel]

Whoa, the lighting control sounds crazy. Do you have a link to some kind of video? Is it available yet?

[@ObaedaKorani]

Thank you so much my friend

[@Repossessionn]

Great video as always Sabastian!

[@sebastiankamph]

Thank you!

[@NEETAZ]

Gugugaga i need protection

[@kingkhalagar2836]

Good video, but I'm surprised you didn't cover the new stuff in ControlNet like Open Pose Hands + Face. Those seem like by far the biggest points of interest now, as you can easily get near perfect hands and faces pointed exactly how you want. Would love a video focusing on it, since I've been tinkering with it and it works, but I don't know if I am actually using everything as well as I could be haha

[@Elwaves2925]

I haven't updated yet and didn't even realise that was a thing in ControlNet 1.1. I agree that it would be great to cover that as it seems like a great feature. Maybe he didn't want the video to be too long, so decided to cover install and the basics?

[@kernsanders3973]

It's mind boggeling basically no one is talking about ControlNet1.1 with all these new additions. The Lineart one as well is mind blowing, ALL my old line art I ran through it and can generate the exact line art image filled in with any style I wish. It's mind blowing. This should be top news. Also they just released their new TEST version where you can control lighting, its insane how good it is. I can specify exactly where the light source is located, the intesity and what color. As well what in the picture should be illuminated to what degree from the light. Example making a sword glow from all the light bouncing off it. This stuff is a massive game change for anyone independent

[@blackvx]

​@Kern Sanders I'll check this out, thank you!

[@shadowdemonaer]

@@kernsanders3973 Thank you for speaking about this in more detail. I am very excited about this. I can't wait to make my images look right!! I have just honestly been settling for whatever pops out and just using whichever images suck the least, but this makes me very happy. This changes the game completely!!

[@hiddendrifts]

@@kernsanders3973 i've never given controlnet a try since my version seems to be unable to find the controlnet models properly fsr. but by the sounds of this, it's worth the effort to do some tweaking and get it working properly

Really awesome tutorial, thanks a lot, it helped me!

[@sebastiankamph]

Glad it helped!

[@Glebean]

thanks sebastian youre amazing

[@StyxGamingProject]

I believe what Sebastian forgot to mention is that OpenPose is for SD v1.5, if you will try to use it with v2.0 or 2.1 models, it will not operate, so make sure to use models trained for 1.5!

[@Steamrick]

I was really happy to find that an Openpose option had been added to PoseMyArt. I hope that they keep at it and expand on the feature to match new controlnet options.

[@JohnVanderbeck]

Needs IK in my opinion. But I'm not sure how much it matters as long as ControlNet can't use foreshortened poses it will continue to fall apart in any but the most basic of cases. You still get a better result not using OpenPose and instead using sketch input.

[@PoseMyArt]

@@JohnVanderbeck Hi John! I've added IK to PMA, please check it out and let me know what you think! (: Also, you can use the normal export in PMA to get better results with foreshortened poses! (:

[@NEETAZ]

​@@JohnVanderbeckin my opinion I would ask you to Say Funny cat

[@JohnVanderbeck]

@@PoseMyArt Amazing!

[@JohnVanderbeck]

@@PoseMyArt So I see the IK option in the settings but don't see how it actually WORKS. Normally you would have hand and feet targets you could move and rotate which would in turn drive the IK chain, but I don't see that here. I'll keep playing with it though because if I can get it to work it will be a game changer.

[@Pauluz_The_Web_Gnome]

This video is very helpful! Thanks Seb...Cya

[@sebastiankamph]

You're very welcome! 🌟

[@mcarthcart414]

Roy from PoseMyArt also added a nifty hands option to the OpenPose export. It works great!

[@user-gl9tr6eq7e]

Thanks a lot for your videos mate.

[@sebastiankamph]

Happy to help!

[@bradcasper4823]

great video

[@user-dj5ss3ls8p]

Posemy is amazing!! Thanks

[@blackvx]

Thank you👍

[@20xd6]

earlier you showed how to add VAE to the UI at the top. Is there a way to add 'clip' at the top of the UI as well? For another video idea, I think it would be cool if you went over each controlnet model and how to use them. Openpose is fairly straightforward, but the other ones are a lot harder to use. I don't even know what half of them are supposed to do. (I'm looking at you, Hed, Seg, and Mlsd!)

[@MrSongib]

2:30 There are some options for smaller files either from Civitai or Huggingface.

[@Tomasin19]

Subscribed!

[@Kryptonic83]

thanks!

[@cereal.consumer]

Nice video. Subscribedo.

[@sebastiankamph]

Happy you liked it. Welcome aboard!

[@fernandomasotto]

thanks Sebastian for the info. I updated Automatic1111 controlnet to 1.1, but i still can use my old fp16 safetensor models, right? Or do i need to update to 1.1 versions?

[@johndrippercodes]

amazing

[@chaerazard]

Thanks!

[@sebastiankamph]

Thank you very much for your support 😘😊💫

[@InnerAtmos]

I love the puns, never stop!

[@sebastiankamph]

I got you! 😊

[@MrNamanix]

Hey Sebastian, Quick question. In your interface I see buttons 16:9, 4:3, 1:1. How do you get those? It would be amazing to be able to switch between those modes with buttons.

[@sebastiankamph]

Aspect ratio, search in extensions tab

[@Cylonick]

I'm only here for the jokes.

[@brainwithani5693]

The jokes are a big plus.

[@NEETAZ]

​@@brainwithani5693and multiplication

[@crax83]

I'm here for the amazing Swedish accent! And the jokes 😁

[@user-bm4bf1nw5u]

Big plus, ok best wishes from your neighbour country

[@CMDRRustyDog]

I notice here you rendered in 1024x1024. It's it better to stick to 512 or768 as the models are trained for those resolutions? This is what I've heard. Is it then ok to go higher without suffering a quality loss? Thanks for the great content btw, subbed.

[@nic-ori]

Thanks.

[@rupeshrsingh3109]

you are best here

[@sebastiankamph]

Thank you! 😊

[@twilightfilms9436]

Thanks for the tutorial! Do you have any idea how to control the direction of the eyes in img2img using batch sequencing?

[@NEETAZ]

Plz kile suskire and sare the vedio

[@gomezabel8880]

I love your tutorial. I just beef up my computer and installed stable diffusion a few days ago so I guess I'm a newb. I was able able to install controlnet however :( installed stable diffusion Auto1111, and about 12 extensions, but .....I'm starting to believe the extension for controlnet is removed from the Github repository maybe recently, am I going nuts? Are you seeing it the extension on Github?

[@surij8376]

Can you please do a video on removing backgrounds and explaining the settings. Thanks!

[@mattlegge8538]

dude, Thanks heaps for your tutorials, I have a question I can't find an answer to: How can I batch generate pose images from a bunch of frames? I have done it accidentally in stable diffusion, but I can't replicate it and it's driving me nuts.

[@EvilGamerCrowley]

9:25 how did you updated the weight that easily? Is there a shortcut or a macro?

[@AtheistExpert]

does this work on 2.1 checkpoints? (noticed theres a 10gig file for the 2.1's) and what can you speak to those in terms of being... better?

[@MartinBeck-vp1wg]

not convinced with controlnet. thanks for the video anyway

[@playmaka2007]

Come for the dad jokes, stay for good AI content.

[@sebastiankamph]

We all love a good dad joke to brighten the day

[@musicandhappinessbyjo795]

Hello, I have been using control net open pose(full). The general structure seems to be accurate but the position of finger get to totally ruined for some reason. Also at times I get different position as well. I am using the realistic vision, 16:9 ratio resolution. What is that I am getting wrong.

[@Younoob633]

Should we use .pth files or .safetensors files (smaller size) as shown on other KZbin channels like Albert Bozesan?

[@IceMetalPunk]

No mention of the model improvements in 1.1? Most of them are just "we improved the data set and model training for better quality", but there's a few standouts: (1) The segmentation model now supports COCO protocol, which means an extra 182 colors/categories. (2) OpenPose can now track face orientation, and can also track hands better with more detailed OpenPose skeletons. (3) Lineart model and preprocessor are entirely new, and can convert lineart into images, including hand-drawn lineart (like a more structured "scribble"). (4) Shuffle model and preprcessor are entirely new; they basically distort the input image enough to keep the style but lose the details, then the model was trained on those; so it's basically ControlNet-based style transfer. (5) Instruct-Pix2Pix is brand new as well, and is trained with a mix of normal prompts and instruction prompts, so it can do instruction-based edits entirely in ControlNet, applied to any base model.

[@rbscli]

ip2p is really interesting, I've been playing with it and it kind of amazes me every time. Have to check the new improvement to poses with face and hands as well.

[@TeamPhlegmatisch]

@Sebastian Kamph I get an error message when running with sdxl model. I updated to the latest version, but it only works with SD models. I get "mat1 and mat2 shapes cannot be multiplied" Any idea why?

[@Ceireavne]

You know why I say mucho to my spanish friends? It means a lot to them.

[@sebastiankamph]

Excellent! 🌟🌟🌟

[@rebelape4257]

im using collab how do i install the models, i have open pose editor but the only options are listed in the processor tab . if i select open pose i dont have any options under the models tab

[@TheSchwarzKater]

Yeah but as soon as the scene is more complex, like two people hugging, it's really hard to not get multiple arms and legs. I've yet to figure out how to do it properly.

[@spartanx169x]

Its now been 6 months, have you made any progress?

[@greenjacketguy3759]

Thanks you I've been trying to find Control net 1.1 videos, Can you do a video covering the new Controlnet features/processors in 1.1 . They added quite a few and the Gitpage is kind of confusing since it's mostly in Tech speak. The various Line new art features, IP2P, Content Shuffle and Various soft edges model are new

[@jolopukkii]

Thanks for the vid! I managed to install Control Net and few extentions, BUT Stable Diffusions window continues to show the default interface plus the Control Net interface at the bottom.. And the ControlNet doesnt work..

[@Nutronic]

What was the keyboard shortcut for the weights in the woman waving?

[@toddz2994]

this video is so helpful! thx

[@sebastiankamph]

Glad it was helpful!

[@leveloup]

Thanks for your great videos! I hoped you could have unlocked the content

[@hateterrorists]

Can you use control net to control the things that aren't people? Can I use it to force buildings or other scene elements to have particular shapes or orientations?

[@sebastiankamph]

Yes! Use canny or depth models for example

[@nighthaven6244]

Ok it... doesn't work. I put all the same, except maybe the checkpoint because i don't know where to find it but i tried anyway multiple checkpoint so i guess that doesn't come from here, I didn't entered style yet but as i uderstand it's just automatic prompt so nothing to do with controlNet... I am lost, it generate a picture but the position of the model is completely random and doesn't follow at all the picture on controlnet with all the settings... :(

[@rastanchacha]

I have to say, the dad jokes made me subscribe

[@sebastiankamph]

Glad to have you aboard!

[@oleksiidubrov4661]

Great

[@BauldyBoys]

Anyone know if 1.1 supports colour palette models? I haven't been able to find one and the old version is breaking my gens.

[@CoconutPete]

As an SD noob I think there's a need for making tutorials customized to specific types of SD such as A1111 tutorials, ComfyUI tutorials, Forge, etc. The biggest confusing factor (for me) is trying to figure out which additional files are required for certain things to work. For example, I tried to install the SSD-1B "thingy" (Lora?) but noticed if I selected it in the "Add network to prompt" drop down it made my renders look horrible. If choosing "none" the renders looked great. Just a random example but an opportunity for content ideas for the experts lol

[@planebreach]

All i want to know is if it has tileable functionality. I really need those tileable depth maps and normal maps.

[@lizng5509]

Thank you for sharing!! It seems the T2ia color and style models don't work on this Control-net version.

[@nick19950]

i have a cuestion, Is stable diffusion the same as automatic 1111? what´s the diference between a both? someone explain it for me, please

[@reye9162]

i downloaded the same extension you show in the video and that is listed in the description, but none of the models show. there's only one which i downloaded and placed in the folder because i thought i didn't have any. they're all there, in the controlnet/model folder, but none work nevermind. apparently, there's nothing there now that i checked again. everything simply disappeared and the .yaml files in Illyasviel's huggingface files don't appear in the models either once downloaded

[@kbizzy111]

is there a way to use cli instead of the gradio app

[@Oxes]

does anyone know why my stable diffusion disables extensions? isn't there a webui version where every extension is ready ? *** "Disable all extensions" option was set, will not load any extensions *** that's what preventing from loading extensions anyone know why?

[@rizkyekydev]

It's in localhost right? Is it heavy ? what GPU are you using ?

[@sebastiankamph]

Rtx 3080. Works with any gpu 4+gb vram

[@bubarules]

Hey Sebastian, any idea why after installing it and restarting the webui I can't still not see the control net area? Do you need an specific stable diffusion version to get it to run?

[@sebastiankamph]

Make sure you're running the latest one. Do a git pull with cmd inside your folder.

[@bubarules]

@Sebastian Kamph weird, yeah I did the git pull

[@rproctor83]

Why isn't the tile model working as described? Whenever trying to use I simply get enlarged blurry version, it never attempts to redraw tiles. No matter settings. Updated everything. Followed multiple guides to the T. Double triple checked everything.

[@reasonablyargued9003]

I just updated from 1.0 to 1.1 today and can no longer run scripts (specifically, Prompt Matrix) with controlnet. I reviewed the readme and confirmed the "allow other scripts to control this extension" box is checked under settings. All the controlnet models work fine otherwise, it's only when scripts are enabled that CN is bypassed. Anyone else having this issue?

[@sneakerhead_paradise95]

do we have to downlaod the files every time we restart it in cmd its downloading again

[@DizzyDJW]

I wish I could see what he was doing, but I was flash bombed at least 12 times during this video. I have no eyesight left.

[@acbattery]

Very good Tutorial. I created a pose similar to a "Handstand" Position. However, i tried lots of different models and prompts but all results are more than bad. Why does it seems almost impossible to get an image of a person doing a handstand although ControlNet has an very exact pose for it?

[@avetmimarlk9934]

Before installing the Controlnet, how do you install Stable Diffusion? There is a chance to use it on stablediffusionweb but it's not screen that I see here nd it always gives "error". How do I get this? Can anybody help me?

[@artofgarduno]

Does this for in stable diffusion easy 3.0

[@PejabatSuki]

Does LoRa not work when using controlnet? I get the same face with LoRa and without LoRa, when I only want her face from LoRa and her pose from openpose

[@Nathan-ib2rs]

Great succinct video... and you had me at "big plus" lol... I have lost my preview of control net when playing with settings. Do you know how to get it back? Any tips appreciated

[@Max-sq4li]

Can you explain clearly what you mean

[@Nathan-ib2rs]

@Max-sq4li sure thing. I was doing some batching with img2img and controlnet...I followed a tutorial on a few settings. One was to turn the preview off ...not save image. (when you run your image with canny for eg... it shows the image and the black and white canny image.) I no longer get the canny image :( just the final output. I can't find which setting.

[@sebastiankamph]

You probably checked this in Settings -> Controlnet -> Do not append detectmap to output

[@Nathan-ib2rs]

@@sebastiankamph that's probably it, I'll give it a shot. Thank you!

[@IshioRamo]

openpose and mediaface should fuse together and make more advanced preprocessors/models. mediapipe is way better to keep facial expressions and we still need something for holding objects like weapons, purses, umbrellas, etc.

[@sebastiankamph]

💯

[@siddharth9242]

Nice guide and all, but i gotta ask man, are you swedish by any chance ?

[@sebastiankamph]

Sure am. Did my lovely accent and stunning Nordic looks give it away?

[@Sevrmark]

Actually, the yaml files are downloaded with controlnet. They were there before I added the pth files

[@marvelheroes6639]

Hey mate I am thinking of making manga / comic book ! So is it possible to make my characters pose like that with control net

[@sebastiankamph]

Yep. Toughest job will be character consistency.

[@scottgust9709]

Why do you prefer Mikubill over Automatic??

[@grillodon]

But there is no longer a preview of the annotator? Especially for the canny?

[@ClayGoes]

How does the pose input do with multiple actors?

[@Steamrick]

The older controle net version already did fine. You'll have to use Latent Couple or a simiar extension if you want to give them very different descriptions, though.

[@Anonymouse007]

at 9:29, my webui don't have styles, how do I get it? edit, okay I found the styles and installed but the output not that good like yours.

[@SuperGari2]

cool, but how come I cannot see any list of Model after checking on Enable? any idea?

[@MADVILLAIN669]

hey I have the same problem, no list of models