Advanced AI Sketch to Image: Style transfer with comfyui and IPadapter

The sketch to render has been one of the most popular AI workflows using stable diffusion. The original method mainly relied on control nets however this new workflow I will show you has far more flexibility and control particularly when it comes to styling.
This is done through the use of ipadapters which are add-on for using images as prompts. You can use it to copy the style, composition, or a face in the reference image. Great to convert a sketch into a specific style render you have a reference of.
00:00 Intro
00:30 IP adapter install
02:37 Control net addition
04:37 Style transfer settings
06:00 Realistic style
06:21 Artistic style
06:50 Weather Style
07:16 Water colour style
07:44 Outro
Install: • One-click installs for...
Part I: • ComfyUI: Getting start...
Part II: • ComfyUI: Intro to Mask...
Part III: • ComfyUI: Intro to Cont...
Part IV: • Best Photorealistic Mo...
ip adapters: huggingface.co/collections/Te...
Checkpoint model: civitai.com/models/133005/jug...
Control nets: huggingface.co/TencentARC/t2i...
Instagram account: / urbandecoders
#ai #architecture #generativedesign #parametric #stablediffusion #upscaling #animation #comfui #image #design #generativedesign #painting #sketchtorender #styletransfer

[@centner86]

Can you place link to chekpoint model, please?

[@Urban_Decoders]

I have added all the links to the models, ipadapters and controlnets now in the description. For the Model checkpoint i used the juggernaut xl as it is one of my favourite but you can use any SDXL model: civitai.com/models/133005/juggernaut-xl

[@alejandrogalindocruz]

@@Urban_Decoders Thank you so much 🙂

[@edumaganto]

Thank you for the tutorial. I have installed the Ipadapter Plus node that you mention, but then I cannot find them within Comfyui, Ipadapter Advance, and Ipadapter Unified Thank you very much

[@Urban_Decoders]

Try updating the ipadapter plus node and also try installing the "comfyUI essentials" custom node too from the manager. In the manager you can also go to install models and install CLIPVIsion models for the ipadapters there.

[@edumaganto]

@@Urban_Decoders thx it works! ^^)

[@hamzehalthweib8931]

Thanks for the tutorial, i made same workflow but i got error by executing Ksampler ( Shaps cannot be multiplied) , have you any idea what it could be? thank you

[@Urban_Decoders]

It may be to do that you are mixing SD1.5 models, controlnets, ipa adapters and checkpoints with SDXL ones. Could try double checking that all the models loaded are for the SD 1.5.

[@hamzehalthweib8931]

@@Urban_Decoders thanks for the hint , i used a wrong Controlnet model. thanks for this tutorial keep it up

[@caramelizedtofu3079]

Tried it on juggernaut xl lightning but the results were not realistic, sort of like a clay. Can we have a chat for this? Thanks!

[@ilgaraziz]

Hi, I'm getting the error "Error occurred when executing IPAdapterUnifiedLoader." I've followed all the instructions and read through the comments, but I'm still unable to resolve the issue. I would appreciate your help. Thank you!

[@felizberto12]

I’m having trouble with the Line Art Preproccessor, an error message shows saying it cannot find such file or directory. I’ve already uninstalled and installed again. Do you have any idea on how to fix it?

[@Urban_Decoders]

You will need to download the sd1.5 line art preprocessor and paste in into comfyui/models/controlnet or you can go the the manager, custom nodes search control nets and select control net preprocessors. It will download it into your directory.

[@azadarhussain4808]

@@Urban_Decoders from i can download it link please

[@Valgrum]

@@Urban_Decoders Where can we download the sd1.5 line art preprocessor,? I am unable to find anything to download with that description.

[@midiablackbox6316]

Error occurred when executing IPAdapterUnifiedLoader: IPAdapter model not found everything is already updated. I checked the paths. it's nothing ;(

[@user-ig8yr7lp3p]

Is it suitable for clay render instead of sketch?

[@Urban_Decoders]

yes, same workflow just would recommend using a different control net (maybe depth) for voumes if you don't need details from your input image.