From real to anime (with IPAdapter and ComfyUI)

Easy and effective way to apply the IPAdater models to get a "realism" slider from cartoonish to ultra-realistic with everything in between. We talk about prompt engineering, upscaling and of course IPAdapter.
IPAdapter: github.com/cubiq/ComfyUI_IPAd...
CASharpening is part of: github.com/cubiq/ComfyUI_esse...
You can download the workflow from here: pastebin.com/tppqDVUp
00:00 Introduction
00:24 Prompt Engineering
03:34 IPAdapter
07:16 Upscaling
12:54 Sharpening
14:25 SDXL
17:24 AnimateDiff
🎵 Background Music
-- "Part A" by Alexander Nakarada (www.serpentsoundstudios.com)
Licensed under Creative Commons BY Attribution 4.0 License
-- "Nightmare City" Synthwave by Karl Casey @ White Bat Audio (whitebataudio.com/)
-- "Menace" Synthwave by Karl Casey @ White Bat Audio

[@Danbearpig]

These have been by far the most helpful and informative videos on Stable Diffusion I have found, please keep them coming! I appreciate that you don’t just show what to do, but explain why you are making specific choices and what happens if you choose differently. Thank you!

[@ericren5390]

Another great video, as always it gave me new know-how to give me a deeper understanding of the drawing process. This is definitely worth watching again. Thank you a lot.

[@rawkeh]

Your work with IPAdapter is astounding, man! Great job and keep it up!

[@pedxing]

the final animations are really quite charming. nice work!

[@enigmatic_e]

This is so awesome! Thank you for creating this!

[@mr2ti41]

Man keep up the great work! I switched over from A1111 and your videos are helping way more than what I learned from any A1111 videos!!

[@jepperisum]

Thanks for awesome tutorials! Like other people say, it is great to kind of watch over the shoulders of a real pro. So yes.. Please keep'em coming! :)

[@johnmcaleer6917]

Wonderfully clear and informative, please keep doing what you are doing...I love IPAdapter and Comfy and yours are the best vids on it I've found..👍🏻

[@ronnykhalil]

another gem! thank you. Looking forward to the next one :)

[@LucasNoce]

Thank you for your tutorials and animatediff nodes ! Cheers from south américa !

[@Inner-Reflections-AI]

Loving your stuff as always!

[@Gabriecielo]

Great video to explain deeper knowledge of IPAdaptor, you guys did a great job!!

[@bobchang325]

Your tutorial inspired me so much, Thank you for these awesome works!

[@vizsumit]

You are legend 💪 Thanks for quality tuts

[@maxehrlich]

Amazing! I can only hope to someday come close to this level of control and knowledge.

[@hakandurgut]

Great video, please keep posting videos like this one

[@comfyuiadrian]

Wonderful work again and I thank you!

[@PradeepKumar6]

Again Excellent Video. Thanks very much.

[@ALatentPlace]

Nice one. I really like the small details of how you work. Tweaking the CFG here, manipulating weights there, jumping between the schedulers and samplers... reminds me of trying not to stick to the kind of workflow you can get used to very fast because of your "daily work". If I wanna see more videos like this? Sorry, I don't get the question. 😉

[@lbf5984]

I listened to this the first time and thought he was saying muse-killer... i had really bought in to it by the end. Muss-queue-lar.

[@swannschilling474]

Thanks a lot for this one!!! Your channel is awesome!! 🤩

[@vinh-le]

Awesome tutorials!!!! Thank you very much.

[@elan4912]

Lots of things I have learned from your video. Thanks a lot.

[@angelotsk3173]

as always, there is something to learn from your videos, today it was Conditionning (concat) ..thanks

[@ysy69]

Thank you for a show and tell tutorial.

[@impactframes]

fantastic channel learning a lot here

[@hleet]

nice choice of music background

[@aleksanderk5695]

Great job, thank You!

[@aa-xn5hc]

Si', grazie, more videos like this, please!

[@Chad-xd3vr]

Yes please, more videos like this

[@reapicus557]

Your playlist of videos has helped me so much! I'm going to be coming back to these for reference for quite a while, and it'll be easy to recommend your videos to others. Getting into this SD and ComfyUI is so exciting. (>'.')👍

2 videos for the price of one!

[@alexgilseg]

thank you 🙏🙏

[@Iron-Painter]

Master

[@_VFX_Ninja]

Fantastici i tuoi video! Question, I do I install your ComfyUI_essentials?

[@alecubudulecu]

where did you get the image_encoder clip vision? mine doesn't automatically have it, and I couldn't find it on the git repo.....

[@FieldMajor76]

Is there a node for multiple lora's with weights? Right now I have my load checkpoint going into a load Lora, but this is just one at a time. Thank you

[@latentvision]

you can daisy chain loras

[@hindihits9260]

use efficiency nodes, it has a lora stacker node

[@FieldMajor76]

@@hindihits9260trying to install the Lora motion that will plug in to the Animatediff node today. For tilt, zoom and pan.

[@nocturne3455]

​​@@hindihits9260note that I noticed the efficient loader seems to start on the wrong clip skip setting. Default is -1 but -2 is the actual baseline. You can test yourself by eliminating the loader and generating a result with the same settings. Your image will differ until you run the loader at -2. But otherwise yes, Efficient Loader is awesome

[@drviolet396]

if you were to use KsamplerRave + animatediff would you connect the IPadapter to the RAVE sampler or still to the animatediff ?

[@kikoking5009]

Hi, can I ask for which use cases this workflow is exactly? U said it's to creat anime like realistic illustration, something in between. As far as I know I can use a Checkpoint to creat those. Iam not sure. Do you know any usecases that are only possible with this workflow?

[@sairampv1]

can you create landscapes and buildings as well with this method?

Hello, great work! And thanks for sharing. When UI tried to open the workflow, the node you use in the video named Apply IPAdapter seems to be missing from my interface. How can I solve this?

[@latentvision]

this is an old workflow, ipadapter has updated now, you need to use the apply ipadapter advanced node instead

[@kakochka1]

Great tutorial as always! Could you explain what's the difference between muting and bypassing nodes? And what's the difference between concat and combine prompt nodes?

[@latentvision]

Muting stops the execution and the following nodes are just ignored. Bypassing just ignores the bypassed nodes and keeps going to the next ones.

[@kakochka1]

Thanks for the answer!@@latentvision And what about ConditioningConcat vs ConditioningCombine? When do you use one vs another?

[@latentvision]

@@kakochka1 combine merges the prompts in one vector. concat keeps them separate. If you want less "bleeding" between the tokens use concat.

[@matyourin]

Can you provide download links to the files you need? I cannot find all via manager... the ipadapter page i found the models for "load ipadapter model"-box... and i also was able to find the deliberate_v3 checkpoint model. But what is that vae you are using to decode after the first ksampler? and in the "load clip vision box" what model is that? Where can i get it? On

[@bender203]

can you post the workflow as a json file? I don't know how to use the text download you linked to.

[@aa-xn5hc]

Amazing, grazie. What about a tutorial with only images as inputs, and very little weight for the text? an image for face, one for style, one for the body/content/background?

[@latentvision]

I'm prepping a video about that (not the next one, the one after)

[@nodewizard]

Matteo, can you do video tutorial using IPA with multiple LORAs? Thank you sir. Ciao

[@latentvision]

you can daisy chain loras. Checkpoint > Lora > Lora > IPAdapter > Ksampler. simple as that :smile:

[@samsiteone]

How do you use that workflow text file?

[@latentvision]

the workflows are text files in json format. Just download (or copy/paste) the text and name the file: whatever.json

[@sc0peAI]

Hi, I am getting this error "size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 1280])." pleae help

[@latentvision]

check the IPAdapter repository, there's a troubleshooting at the bottom of the main page

[@alpaykasal2902]

before you did a concat of the positive clips, you did not have a comma separating "perfect piercing blue eyes wasteland background highly detailed". i am pretty sure your carriage return does not separate those terms. Maybe I'm wrong. Might explain why your terms bled.

[@latentvision]

the concatenation happens at the tensor level. the two texts are concatenated after they are converted into a vector

[@luiswebdev8292]

what are your computer specs? It runs so much faster than mine... Also I don't understand why you use a 4x upscaler and then downscale my .5, why not simply use a 2x upscaler to begin with?

[@latentvision]

the video is sped-up sometimes even x10 otherwise it would become a 5 hours long video and would be terribly boring to watch :) Regarding the upscaler I use the 4x because that's the one that I like that is better than many 2x. But you are free to use whatever upscaler you want.

[@user-rk3wy7bz8h]

I need help i get a error when working with an SDXL checkpoint. RuntimeError 3 K Sampler. It shows: Expected query, key, and value to have the same dtype, but got query.dtype: struct c10::Half key.dtype: float and value.dtype: float instead. I have a gtx 1080 and use gpu. I want to continue using gpu without getting the error. In cpu it works without error but takes hours to generate a simple IP Adapter workflow. I really need help, please.

[@latentvision]

run comfy with --force-fp16 option

[@user-rk3wy7bz8h]

@@latentvision yess it works. Thank you very much. You are very helpful continue :)

[@user-hv3ey2zg4i]

What does it mean when an image is "burnt"?

[@latentvision]

the checkpoints are often "over trained", that leads the image to look over-saturated at medium/high CFG scale. When that happens we sometimes refer to that as "burnt image"

[@user-hv3ey2zg4i]

@@latentvision Thanks!

[@grahamastor4194]

Great video, thank you. Looking for a strong female actress... Sigourney Weaver?

[@luiswebdev8292]

suggestion: remove the music for the tutorials as I might want to have my own music in the background at low volume

[@latentvision]

that's something I didn't think about. I do a lot of cuts in the audio and the voice goes a bit up/down, the bgm helps merging all together and adds a little consistency.