ControlNet Union for SDXL - one Model for everything

[@andreabigiarini]

found the solution for "After loading the ControlNet Preprocessor node, in the preprocessor selection item there are only two things to choose from: "tile" and "tile"- You need to install from Manager " ComfyUI’s ControlNet Auxiliary Preprocessors." And now it list all the Controlnet preprocessors.

[@rhym8882]

That worked. Thanks!

[@OlivioSarikas]

thank you for adding that. I pinned your comment

[@andreabigiarini]

@@OlivioSarikas thanks to you for your hard work my friend!

[@electronicmusicartcollective]

Ive install the ComfyUI’s ControlNet Auxiliary Preprocessors but cant find the node "ControlNet Preprocessor"

[@Itazurana_Neko]

Thanks a lot!

[@FormatAndGarm]

Thanks for introducing this model! this saves me memory and work space! just some index error to solve in the "venture control net pre processor" but now it runs super smooth!

[@ZeroToNineTimes]

Short and sweet, thanks a lot. Looking forward to see more workflows with this ControlNet from you.

[@1lllllllll1]

@Oliver Sarikas, please do another follow-on, with that typical, useful, slow paced, detailed, informative, valuable explanation style of yours!

[@murphylanga]

Thank you for this information, it helps me to streamline and organize my workflows. In large, complex workflows, the "Normal map Controlnet" node of rgthree is interesting. For example, it can be used to remove the control net after the sampler to speed up the process when scaling up.

[@TransformXRED]

Have you tried the Depth Anything v2 preprocessor? If not, you should check it out. The depth map is way cleaner

[@TR-707]

nice ill use this one - i just checked with an existing Depth Anything node (not sure if its v2) and it was way cleaner than the one in the vid

[@Bicyclesidewalk]

It is great.

[@TR-707]

@@Bicyclesidewalk i actually ended up going down the tensorrt rabbit hole and just set it up with v2 . its fast but i think it also introduces weird noise at crazy settings like +2 or 3 controlnet strength while i didnt notice extreme corrupting with even v1

[@TR-707]

yeah just confirmed - v2 without tensort is really smooth and nice even with Base and fp16 - speed increase from tensort version is negligible - probably only useful for small res vids

[@ausbildungbew2189]

does it work with forge? if so how?

[@XastherReeD]

I'm most impressed by this controlnet model being 2.51GB for SDXL when the openpose SDLX model I downloaded two days ago was over 5GB by itself.

[@Teardropbrut]

But even while using the union model ComfyUI is still downloading the individual controlnet models, like Zoe Depth 1.4GB and saves them into \\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts\lllyasviel\Annotators\

[@DasHnezzEdits]

@@Teardropbrut isnt that just a preprocessor?

[@iddeegee]

It would be very helpful to have introduction explaining what's the point of using it and difference between old CNet models and this one? saving disk space, better quality?

[@Elwaves2925]

I don't see it being better quality from what I understand, it seems to be more about saving space and streamlining the process so you don't have to select the correct (and different) models each time. I might be wrong but that's how I understand it.

[@NevelWong]

@@Elwaves2925 It would be interesting to test if one could mix different controlnet types this way. Could you for example alpha over a pose on top of a depth map, and have the controlnet still work for the most part? This might work better than layering multiple controlnets sequentially. I wish I had the time to test it, but comfy ui has a habit of just.. disappearing the rest of your day once you start fiddling around, so I'll restrain for now :D

[@Elwaves2925]

@@NevelWong No idea and I barely use Comfy at all, except for a couple of things but it would be interesting to see if someone else tries it.

[@takiali8476]

i love your videos maan ♥♥

[@MaiWhisper]

For me this might be the best Comfy news I could ask for. Many many many many thanks!!!

[@mohamadelsawi]

i like all of your videos , I will be always your fan , thanks for everything

[@TransformXRED]

Does anyone have an idea why the "ControlNet Preprocessor", doesn't see any of the controlnet installed? I just have "tile" listed

[@PixelsVerwisselaar]

Finally a new video 😊

[@tripleheadedmonkey420]

Great video! Always great to save more hard drive space I say!

[@kaziahmed]

This is awesome! Thanks Olivio, you're the best 🙌🏽🙌🏽

[@AlanLeebr]

Hey Olivio! Did you see the new licence for SD3??

[@missty-Flux]

For the Promax version, this adds Inpaint and Tile to the extensive list of controls. I've tested it on Comfyui and got a few errors, so I'm going to use the Union controlnet while waiting to see if the Promax version evolves for Comfyui.

[@ristopaasivirta9770]

Awesome news/showcase video! Thanks for keeping it tight and informative.

[@andrhp87]

On the huggingface website there is now a promax model as well. I don't know what's the difference with that model, but can i just try it like the original model? Does it work the same way?

[@juanchogarzonmiranda]

Thanks!!!!

[@valerya91]

Amazing tutorial Olivio thank you!! Can I ask you why you are using the Jaggernaut model v9 and not another one? What is the difference there?

[@Ai-Art-42]

Very interresting, thanks

[@Steamrick]

What do you mean 'comfy only'? It works just fine with the preprocessors I tested in Forge Webui.

[@Elwaves2925]

He doesn't say "Comfy only" he says "as always it's working with ComfyUI." I like Olivio but he's too biased towards Comfy to even mention whether it works with other UI's. Good to know it works with Forge though as that's what I use, so cheers for that info.

[@twelvyvisionz7637]

I get colorful pics when I use it in Forge with Pony models. Any solution?

[@Elwaves2925]

@@twelvyvisionz7637 There's a few possibilities but without knowing your exact settings there's no way to be certain. 1) Check your VAE, is it the right one, do you even need one? 2) Try different samplers/schedulers. Some work, some don't. Euler A usually works and DPM++ 2M SDE Karras works for me. 3) Do you have the score numbers in place? If not, find a Pony generated image with a prompt and copy them across to your prompt. You can save that as a style fr next time. 4) Check your CFG scale and steps. Uses one's you find in pompts for Pony images as a guide. 5) If you use a refiner, try it without. Hope that helps. 🙂

[@Milkymalk]

@@twelvyvisionz7637 Apparently, Pony is evolutionary too far apart from SDXL that Controlnet has smaller or larger problems.

[@NOORIEAI]

Does it work for both sd1.5 and sdxl checkpoints

[@RuProject-i2j]

Thanks!

[@andreabigiarini]

Me too "After loading the ControlNet Preprocessor node, in the preprocessor selection item there are only two things to choose from: "tile" and "tile". do i've missed something' thanks Olivio!

[@AIKnowledge2Go]

Are the results of this, any good? The current controlNet models for SDXL are mediocre at best.

[@xteasy1045]

Why do I pop up such a prompt with the same connection method as you? Error occurred when executing KSampler: 'NoneType' object has no attribute 'shape'

[@sewrathfeatherstone5088]

Thank you, this is amazing! It certainly can simplifie a lot of workflows

[@alejandrogarcia9472]

Thanks for the video, I like how easy you made the videos, could you please share your comfyui workflow?

[@mohmmedalbihany5723]

I was gonna ask for the same can we get the workflow please ?

[@ryzhiialen]

If one day the rest of the interfaces stop supporting, I will cry. Comfy looks absolutely terrifying, like a giant equation full of logarithms. Sorry for my English

[@tripleheadedmonkey420]

It's not as complex as it looks nowadays. With the ComfyUI Manager, all you have to do is load an image into the browser window and it will automatically load the workflow and you can install the extensions automatically. Only needing to restart comfy and then you can hit the ground running just by inserting new prompts into their workflow.

[@Elwaves2925]

@@tripleheadedmonkey420 That's not exactly true though. Sometimes that works well but from my own experience (and why I don't like Comfy) the manager doesn't always install the correct (or up-to-date) nodes and even if it does, there can still be errors throughout the workflow. That may be down to the workflow being outdated or whatever but it gives you almost no information on how to fix it. I've spent hours scouring the internet trying to fix some errors/missing nodes.

[@tripleheadedmonkey420]

@@Elwaves2925 90% of the time the issue is that ComfyUI itself needs an update in order to work with the nodes, or the workflow is no longer supported due to updates. The amount of times that a node or extension actually requires follow up beyond that is negligible and for most users can be ignored fully.

[@Elwaves2925]

@@tripleheadedmonkey420 Not in my experience. It's been closer to 90% needing follow up, either due to nodes, errors or the workflow and that's after updating everything. 🙂

[@tripleheadedmonkey420]

@@Elwaves2925 That's really unlucky. I've been using it for a year and had barely any problems. Other people have had problems, but most of the time it's resolved in a few minutes by reinstalling. There are some exceptions, but usually this is because of manually installing things system wide which conflict with the python embed folder.

[@snarsule]

Is this working with openpose correctly for everyone else? I am getting the stick person with blurry background as the final generated image.

[@ProsonicStudiosLLC]

Exactly, I'm getting the same for openpose, it just doesn't work. I've spent hours on it... Tried it in all three UIs (comfy, forge and auto) and all kinds of models, settings. None of the controlnets in Union work for inpainting either. Either I'm doing something wrong, or there's still a lot of bugs to be ironed out.

[@vizsumit]

use 512x512, that works fine

[@rocren6246]

Controlnet preprocessor gives "list index out of range" for reason I don't see. Anyone have a solution?

[@ImHewg]

When I do OpenPose, I get this error with the preprocessor any idea why? AV_ControlNetPreprocessor list index out of range

[@Itazurana_Neko]

After loading the ControlNet Preprocessor node, in the preprocessor selection item there are only two things to choose from: "tile" and "tile"(

[@ebbownee]

same

[@Sebpv2006]

How is the animation in the end made ? mixamo character + motion ?

[@fritmore]

you do NOT need the PREPROCESSOR you can just plug ur source image directly to get you going fast. Of course for more control you may want to use the preprocessor node

[@jalpranjal]

Wow!

[@Spindonesia]

Sir why all like moving to comfy ui instead of webui

[@Elwaves2925]

Some people like it because it's customisation allows for a lot of power but despite what they say, it also brings complexity and a whole bunch of error solving with it. An imported workflow doesn't always work out of the box, even with the manager installed. If you don't need that power, the other UI's are still fantastic.

[@neitanrdandel2220]

Only for ComfyUi?

[@sirjcbcbsh]

any difference for the promax.safetensors?

[@derhofsklave3458]

Gibt es inzwischen eigentlich einen T2iAdapter_color für SDXL oder ein ähnliches Modell, um Farben aus einem Originalbild zu übernehmen??

[@ALFTHADRADDAD]

Awesome

[@richardtobing5012]

for those of you who have tried it, how does it compare to SD1.5's controlnet 1.1? particularly in lineart

[@TScottT]

Will it do the nifty QR-Code controlnet?

[@armin_koi55]

How to install this in webui forge

[@MilesBellas]

TensorRT support ?

[@magimyster]

Wow! 😮👍

[@olvaddeepfake]

iv'e tried comfyui but the quality of the images is pretty bad compared to automatic1111 or forge. it seems to be slightly blurrier and more pixelated and the colours are a bit more washed out. has anyone else noticed that?

[@GhostFS]

There is one also for 1.5?

[@РоманСырватка]

Amazing. I tried in a1111, when processing img2img, this universal database gave a better and more accurate picture than (lineart + canny )both! Not bad for a start. ps: However, the generation time has increased. Most likely due to a lack of 10GB of video memory (

[@electronicmusicartcollective]

Thanx 4 the good news and this makes the ControllNet handling very easy. TOP TOP TOP

[@marcdevinci893]

Would anyone know how to use multiple controlnet? Still new to Comfy and we could stack them in A1111.

[@TR-707]

is it me or is the Preprocessor node from this vid outputting 8bit depth maps in midas mode? I get crazy banding in the depth map augmented output

[@cubapawlac]

Does it work with A1111?

[@miguelarce6489]

X2

[@cubapawlac]

@@miguelarce6489 what X2?

[@makam2089]

@@cubapawlac X3

[@cubapawlac]

@@makam2089 ??? Answer the full sentence please. I don’t understand…

[@frostyden]

должно и может работать так как работает в Forge они сильно не отличаются

[@TheGiovany82]

Amazing

[@ShagBandit]

Dang, trying to get it to work. Everything goes fine until it gets to the KSampler. Hope some else has this issue and finds a way around it.

[@OlivioSarikas]

are you sure you use a Xl model?

[@ShagBandit]

@@OlivioSarikas Ah, that was it! I was being silly. Thank you!

[@studio2513]

Does it work nicely with SDXL 1.5?

[@EranMahalu]

I get this error - Error occurred when executing ControlNetLoader: Error(s) in loading state_dict for ControlNet: size mismatch for task_embedding: copying a param with shape torch.Size([8, 320]) from checkpoint, the shape in current model is torch.Size([6, 320]). size mismatch for control_add_embedding.linear_1.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is torch.Size([1280, 1536]).

[@Mohammedlll-jg1hw]

How do i save an image in JPG format in comfyUI? Please help me

[@SyamsQbattar]

how to change the cloth?

[@Vidanimatiopro]

Niche

[@PowersVideo3D]

Do you have a 101 video that walks through ComfyAI? Also what does a ControlNet do?

[@Elwaves2925]

He has some videos on both but if you can't find what you need there, there's plenty of others on YT.

[@edlondon3717]

Oli is AI mad

[@mcqx4]

Dear Olivio, how you made that decent animation at the end?

[@annyeong5810]

Am I crazy or its just downloading all the different controlnet checkpoints in a temp folder and still using one file each? this feels gimmicky also, Can someone explain me what is promax version and what is the difference?

[@buzukisbuzukis]

is there any controlnet for sd3?

[@eriuz]

works with rocm on linux ?

[@szzzak999]

Hello, works on InvokeAI too? Someone tested?

[@jonmichaelgalindo]

Does it work with normals??? Because we've never had an SDXL normal CN before.

[@michail_777]

Olivio, you mean this model fits all pre-processors? Oddly enough, I often use the depth map, normalbae or tile model with the openpose preprocessor. This yields some very interesting results. But it's for Animation Thanks Olivio, let's see what model it is.

[@gaming_one1846]

does this work for a1111 and tile for image enhancement like sd?

[@Cryunreal]

this is really cool

[@TheColonelJJ]

Picture me begging for an A1111 example! Please?

[@ericanderson4973]

Seconded

[@IceMetalPunk]

According to someone on Reddit, it just works directly with A1111 already. You just put the model in the ControlNet models folder, and then you can chose it as the model in the ControlNet extension in the UI and use any preprocessor with it. I haven't tried yet, but that's apparently the case based on a Reddit post 🤷‍♂

[@jungleelomdi]

Switch to ComfyUI already it's superior in everything

[@IceMetalPunk]

@@jungleelomdi Except in that it's node/graph based.

[@TheColonelJJ]

@@jungleelomdi Hey, at 71 I'm pretty happy that I'm able to run A1111, Forge, Fooocus, etc. ComfyUI is just hard for me to figure out.

[@11011Owl]

Hi I can't join disc, can you update the link?

[@MilesBellas]

Isn't Automatic 1111 deprecated?

[@TheSchwarzKater]

This for pony models would be great.

[@JimboSkinner]

How necessary are all the multi modal control images (depth, stick figure, etc). Do you need to provide all those images as the input to the new control net?

[@OlivioSarikas]

no, i just showed them as examples. you only need one (or as many controlnets as you want to use at one time)

[@hleet]

That's very clear, thank you

[@diegopons9808]

Hey! Theres a new file now, diffusion_pytorch_model_promax, is it any better/different than the diffusion_pytorch_model ?

[@foxmp1585]

You could check for the news in their original huggingface info page! The model with the _suffix "promax" in its name was designed for inpainting work.

[@You_are_wrong99]

so I guess we won't see anytime soon controlnet for pony.

[@squirrelhallowino29]

There's some that works though, although a bit janky. The problem is pony iirc, but the author is improving the dataset for the next version. But idk there's a lot of hate towards him on that side of the spectrum, we should really start supporting astralyte, otherwise corps might just take over and the last thing we'll have for anime is pony 6

[@AhKaj-jw8of]

Actually, we already do have. There's a model called 'anytest'. It can be used like various ControlNet models, just like this one, and you don't need to choose a preprocessor. It works with pony, animagine, and sdxl, and can reproduce even very complex poses without getting confused. the articles are only available in Japanese though

[@chadwick3593]

@@AhKaj-jw8of Do you mean AnyText? I don't see an AnyTest, and AnyText seems to only do text.

[@AhKaj-jw8of]

@@chadwick3593 No, the name 'anytest' is correct. The basic usage is extremely simple, works with 1111 and comfy. but if you want to know litte bit advanced, you'll need to translate the GitHub page EasySdxlWebUI and other articles into English. I can't paste the URL because KZbin deletes comments with link

[@ClumsyWoodsman]

Pony does things I can't replicate with any other checkpoint.

[@CharlesLijt]

China No.1 Author Xinsir yet another chinese gaint to bring light to the AI community

[@ProsonicStudiosLLC]

Looks promising, but most of it doesn't work. Still a lot of bugs/problems to work out.

[@150598leo]

Hi Olivio, on my ComfyUI as i included new nodes, i keep getting the "reconnecting" error that nevers recove, so i have to restart. Have you had this issue?

[@OlivioSarikas]

haven't seen that yet on my pc. best you ask in the comfyui discord. also check if you get any error messages

[@WiLDeveD]

Long time no SEE Olivio. missed you so much man... Are you get married ? 😉😊😄

[@OlivioSarikas]

not yet ;)

[@Martinsc7272]

please make a tutorial on how to upscale a video in Automation 1111

[@jcvijr]

Now the workflows will be simpler, it’s just one file to deal with

[@hintingimage]

Olivia, I haven't played with A1111 for a long while, because the interface is getting buggy and the temp/cache folder is getting very big and eat up all my drive space. Is it safe to delete that folder in My documents and download everything again?

[@OlivioSarikas]

yes, but move all the models to a new folder so you don't have to download them again

[@havemoney]

Why hasn't CivitAI lifted the ban on SD3 yet? 48 hours have passed, I’m sure someone made the lore and models past tense, where are they?

[@Elwaves2925]

They're likely running the newer license by lawyers to make sure of everything first. Give them time, 48 hours is not long enough especially as it's been over a weekend.

[@zZwowAi]

when a1111? 😢

[@Elwaves2925]

According to another comment it works right now.

[@moonduckmaximus]

FU AND COMFY UI

[@Elwaves2925]

According to other comments it works in A1111 and Forge, so possibly others too.

[@jevinlownardo8784]

i stop watching it when he said compfyui, thank you and no thanks

[@RhapsHayden]

Off topic but I figured you would know. For Project Odyssey, do they only accept animations or is themed lookbook acceptable?

[@cowlevelcrypto2346]

Thanks!

[@eveekiviblog7361]

Does it works for seg model from 1.5?

[@egyptoon]

is it working with Automatic1111?