Attention Masking with IPAdapter and ComfyUI

Exciting new feature for the IPAdapter extesion: it's now possible to mask part of the composition to affect only a certain area... And you can use multiple masks for a perfect result. This and much more directly from the developer of the ComfyUI Extension.
You can find a few workflows in the examples directory of the ComfyUI_IPAdapter_Plus extension: github.com/cubiq/ComfyUI_IPAd...
More complex workflows (just download the json format):
Masked conditioning: pastebin.com/DjYg6jMC
Complex Masked conditioning: pastebin.com/PsD13EVK
RGB Masking: pastebin.com/jBB9cAn2
00:00 Intro
00:21 Weight types
02:51 Attention Masking
05:17 Multiple Masks
08:57 Mask Conditioning
10:48 Conclusions
🎵 Background Music
-- "Part A" by Alexander Nakarada (www.serpentsoundstudios.com)
Licensed under Creative Commons BY Attribution 4.0 License
-- "White Gold" by Karl Casey @ White Bat Audio (whitebataudio.com/)
-- "CyberPunk City" by Peritune (peritune.com/)

[@latentvision]

I want to thank everybody for taking the time to write compliments and words of encouragement. I really do appreciate all of them and they honestly motivate me in releasing more videos. Let me know if there are topics you'd like to see covered and thank you all again!

[@EH21UTB]

These are fantastic videos. I am wondering if there is a way to use IPadatper to only transfer color pallet or other facets of an input image (but not the whole). I think for SD 1.5 there is a way to take only the color pallet with the t2iadapters but not with SDXL - maybe you know of a way?

[@latentvision]

I don't think there's a color controlnet yet for SDXL but you can use img2img with a certain grade of success. Set the colors how you want them laid out with a paint software then use that reference image as base for your img2img with a very high denoise. That should kinda work.@@EH21UTB

[@LouisGedo]

👋

[@ruizhu7140]

​ @LouisGedo Thank you so much for all you efford.Actually I am trying to make a flow in order to let a model wear some specific cloth provided as a picture. I tried lots of times, but the clothe weared by the model generated by the flow always different with the one I provided. I really want to understand how to do it . can you please explain about this .

[@ttul]

I’ve finally gotten into IP Adapter. Combining it with depth and pose control nets and ReActor face swap yields incredible results. For instance, I took photos of my parents and photos of vintage Christmas cards and produced a nice vintage style Christmas card that looks like them as a gift.

[@marshy..]

Could you please share your workflow? Must appreciated

[@checkmate559]

Bro I need to talk with u can u share ur insta id

[@jacobharvey1004]

You're such a great example of an open source dev. Always listening to feedback and working with the community.

[@heyitsjoshd]

The merge of styles is insane!

[@nocturne3455]

IPA Plus has been the most game changing thing I've found yet. It's capabilities are insane. Can't ask for much more.

[@andykoala3010]

Brilliant - thank you for these and for sharing the workflows

[@dasomen]

Amazing job as usual, what a great new feature update!

[@impactframes]

This is awesome Bravo. I tested the channel penalty this morning, it improved the result as it handles well weights more than one

[@CesarKonn]

that's beautiful, that's exciting. exquisite, incredible work! you are very honored. Thank you very much for everything, a great mind. you have no idea how many people you're helping, making our projects more fun, easier and better!

[@johnmcaleer6917]

You are a legend, loving the IpAdapter and your vids are wonderful, simple in style yet so informative ...Love em..You are appreciated..

[@simonogden1486]

Fantastic! A truly superb tool and fantastic explanation. Thank you!

[@latent-broadcasting]

I just tested it and it's amazing! The results are impossible to obtain with a normal prompt, this bring so many possibilities. Thanks a lot for your videos!

[@spider161]

Amazing! Keep up the incredible work.

[@TheShadowArtCom]

You, kind Sir, are a treasure. I absolutely love IPadapter. I am very very thankful.

[@julienaubert2422]

just to say thank you for you work and dedication !

[@dck7048]

Wonderful content and wonderful tech, thanks for your hard work!

[@user-fk4xr4ep3c]

Just started going through these playlist; This is truly a gem. Thanks much for sharing.

[@user-fk4xr4ep3c]

If you get a chance, do list the machine specs you use for controlNet + animateDiff +IPAdapter? It does take a long time even on a 16GB GPU. TIA

[@EH21UTB]

Fantastic! I will have to watch this several times!

[@imKaku]

Jesus christ, ive been in love with IPAdapter since its release and this just completely unlocks all the possibilities. This is an insane value tech, thanks a lot for the contribution.

[@ooiirraa]

There's not that many videos yet on your channel, but every video is a diamond. One of the most interesting and valuable content I've met on youtube so far. Thank you VERY much for what you are doing.

[@yotraxx]

This... IS... CRAZY !!! Thank you so much for sharing your fantastic work ! I'm using IP+ andd I'm impressed by the results, tanks to you !

[@BoolitMagnet]

This is just absolutely amazing!

[@testales]

A wizard in action! :) So this is how you are supposed to work with masks! I had to pause this video several times, so much information in such a short time, well done, very useful!

[@8959890]

This is just mind blowing at this point. I'm really really glad I came across you (and IPadapter) in my AI learning path!

[@Try4Ce]

This is pure (AI) magic and will help soooooo much in composition! Amazing results in your demonstration, and your tutorial is spot on, very clear to understand. Thanks so much for your hard work!

[@mr2ti41]

You’re now my #1 KZbinr for ComfyUI . Your channel deserves so much more subs - And it will!!!

[@aerofrost1]

Thank you for your work with IPAdapter. I've only been using it ever since I found your videos on it. It's so great, being able to have 1 image loras without the need to download others, and can just very quickly create them. The one Issue I've been having is it's been transferring the hair to every character design (I'm using a Harley Quinn reference I made for the adapter image and it's copying the pigtail hairstyle and face on pretty much every character lol)

[@WhySoBroke]

Matteo il MAESTRO!!! Many thanks for the great latent wisdom you give us!! ❤️🇲🇽❤️

[@ceegeevibes1335]

omg. amazing stuff... unbeatable feature.. you are brilliant

[@styrke9272]

That is pretty cool, thanks for the update and the tutorial

[@comfyuiadrian]

Fantastic work, thank you so much!

[@I_reject_your_reality]

I agree with others. Mind blowing feature. Thank you! 🤯👌🤘

[@LTE18]

Thank you for the work. Amazing tool

[@javierzzz4556]

Truly amazing. Many thanks

[@Elwandesign]

I used to love ipadapter now I adore it 😍😍😍

[@Inner-Reflections-AI]

Amazing Work as always!

[@Taimoorabdullah]

Holy! That was so impressive. Love your videos man! So much to learn and see it in action.

[@enriqueicm7341]

Thanks a lot! it was very very useful!

[@EdwardGu-ic2tn]

I want thank you for your great work, not only this video, but also other videos and the comfyui_adpater_plus, which help me learn great knowledge about how to control the AI model to draw images.

[@latentvision]

thanks! And have fun with AI! 😄

[@grebenshyo]

bravo , matteo! 10/10 questa roba! 🔥

[@BernardMaltais]

Really awesome stuff! Brilliant!

[@user-ox3xp9lq8j]

Mateo, you are a GOD! Thank you so much!

[@reekster30]

This is awesome - thank you 😎

[@alecubudulecu]

Amazing work as always!

[@petruschka222]

Great job as always!

[@YaBoyTwiz]

Hey there! I rarely comment on videos, but I feel compelled to express my appreciation for your unique approach. Unlike many creators who merely guide us to mimic their actions, you delve deeper, offering insights into the 'why' and 'how' behind each step. This approach not only makes your content easy to follow but also empowers us to adapt and understand the process, fostering genuine learning. I've eagerly watched all your videos, and the knowledge I've gained is invaluable. Your response to a comment in another video about the tedium of covering basics resonated with me. It's true that many overlook the importance of fundamentals. Your explanations are a rare gem in a sea of superficial tutorials; they enable us to think, not just copy. To alleviate the repetition for you, perhaps consider creating a series of detailed videos on these basic concepts. This could serve as a reference point for your viewers, and it's also a strategy that tends to be favored by the KZbin algorithm. What captivates me the most, however, is watching your workflow, especially in segments like the anime and pixel art transformations. Your real-time commentary adds immense value. Regardless of how many times a topic has been covered by others, your unique perspective and method of presentation make your videos a must-watch, even on well-trodden subjects."

[@latentvision]

A comment made by AI for a video about AI😅We live in weird times! I really enjoyed it thanks for the kind words

[@YaBoyTwiz]

I used A.I. to touch it up but its really how i felt

[@dflfd]

thank you, this is amazing!

[@hleet]

WOW ! you are the BEST ! It's just so hard just by looking at video but might be able to replicate by doing. Thanks for that man !

[@yvann.mp4]

Incredible, thanks a lot

[@nodewizard]

Gamechanger! There are many, many methods to achieve these end results; however, as we all know, they're tedious and extremely mind-numbingly time-consuming. Matteo's IPAdapter with noise method works the best. Work smarter, not harder!

[@user-ui2hw5of9l]

amazing~!! Thank you very much !!

[@stefanguiton]

Excellent work!

[@jacekfr3252]

wow, great job! Thank you!

[@Clupea101]

Superb , Awsome work

[@Ulayo]

Wow! This is a great update!

[@Andro-Meta]

Thank you so much for your insight :)

[@TheSleppy]

This is amazing! Thanks

[@brgtubedev001]

Your work is unbelievable. This is tremendous and I just can't believe we get to watch you develop in real time these tools that will revolutionize AI image generation.

[@enthuesd]

Overnight Latent Vision becomes top SD instructor

I am grateful...

[@pedrogorilla483]

Wow! You have unleashed one of the best things for Stable Diffusion since ControlNets! Thanks for developing these great tools and showing us how to use them!

[@arthitchyoutube]

This is amazing. I cannot express my feeling after seeing your video. This blows my mind. ❤😊

[@p_p]

Bella scoperta, un italiano! che bello. Iscritto subito

[@ronnykhalil]

Always a treat when Mateo puts out a new video 🙌

[@NHCH]

che lavoro della madonna però, grandissimo

[@kempict]

This tech is unreal! Your channel deserves to blow up. I'm loving your tutorial format.

[@florianbecke3076]

Completely blow me away! Thanks so much for creating the Adapter and these tutorials! ✌

[@MicheleBrugiolo]

Sei un grande!

[@RoguishlyHandsome]

Outstanding

[@kunalpuri9492]

Beautiful, simple, explosively relevant. You're inspiring.

[@rbscli]

amazing! Thank you!

[@bandinopla]

this is amazing!

[@ericren5390]

Mateo, this blows my mind, I'm so inspired by your great work, I can't express my gratitude enough. Your guidance hit the mark in a simple and easy to understand way which was definitely what I needed. Thanks again for the great work you do.

[@pk.9436]

Great Work again 💪

[@cwhiticar1]

amazing tools and workflow! Insanely impressive!

[@NG2C]

Thanks a lot for your work on this most excellent tool. This has given me so much possibilities regarding composition, and easily as well ! Less hassle and much better control/results. Impressive stuff.

[@AkedoNiko]

damn I had no reason to use comfyUI until NOW, this update is crazy

[@MikeLisii]

Wow, just wow. So detailed explanation and so easy to follow one of the best channels on Comfyui. I am always looking forward to your videos.

[@bobbyboe]

Very valid concentrated information... thx a lot! - I wished all these special infos would be provided as "note-nodes" explaining things right in the json-example. It is obvious to me that your knowledge about all these components and how they work together is superior to some other "random-masters" on YT. It is difficult though... to remember all the little facts, like which model needs what other model and so on...

[@latentvision]

I hear you, the problem with comfy (and all generative ai) is that things move so fast that any documentation is outdated in 3 months (sometimes less). That would mean reviewing all the workflows every now and then and it's a huge amount of work. I'm thinking of writing some articles especially for more complex topics... but do people still read? Everything seems to be video or burst

[@bobbyboe]

@@latentvision You are right Matteo, thank you for your answer. I also know that my "wish" included more work for you... I didnt mean it in a demanding way, I am thankful for any piece of information on a journey into comfy UI. The problem is, that I like to understand things in depth... while it seems to me, that some are ok if if only some nice picture result in the end... and in the beginning of research it is still difficult to seperate the good from useless sources of information. Not many developpers are as good in teaching as you are, or they do not teach at all. I think, much of the information, like - basic relations - will not be outdated so quickly... once the main principles are understood... understanding coming updates can also be challenged :-)

[@murphylanga]

wow, that is great🥰

[@andrewq7125]

Fantastic! .

[@ZeroIQ2]

Very cool!

[@CloakerJosh]

I've only recently discovered your content, but I just had to tell you how blown away I am at your knowledge and skill in this field. It's just breathtaking stuff you're getting up to. Kudos to you, and all the brains that make this technology possible. Could you imagine travelling back in time even just 20 years and showing this to people? It'd be science fiction.

[@isaactut2520]

Where do you come from you hero!! Amazing explanation once again! Thank you.

[@user-ly6cl5rm8u]

good job🥰

[@reapicus557]

You blew me away with compositing using multiple masks. That is so freaking cool!

[@paulofalca0]

I love this! :)

[@koningsbruggen]

super cool!

[@anderswesterlund4191]

thanks king

[@shenjason6619]

Beautiful, simple, explosively relevant. You're inspiring. Love you so much.

[@CuBixxx86]

superb

[@mhfx]

really incredible -- great demo, I can't wait to use this. I've been experimenting with some of your other tutorials to make animatediff clips. Excited to see the control I might be able to get from masking areas and being able to prompt each one.

[@tomschuelke7955]

Creasy stuff i have to say... I used A1111 and now have switched to Invoke AI, because as strong as Comfy ui is, for a company, using architecture ease of use is the most important feature, and here it gives you as well a linear standart interface (the best out there) and also nodes.. and usal users that have not only to use nodes but dozens of programes are better off with a simpler userinterface where the nodes and inputs dont change place each session.. But i have to say wow.. realy realy impressive

[@latentvision]

the biggest advantage of comfyui is how fast it is developed and how easy it is to develop an extension for it. This is of course not always a good thing because as you say a company needs stability and ease of use. but for bleeding edge comfyui is really amazing

[@BernardMaltais]

If invoke AI adopted ComfyUI as their node backend they would have access to all the amazing custom nodes rather than have to develop theirs from scratch... just saying... They are kind of re-inventing the wheel... they should adopt rather than compete.

[@latentvision]

@@BernardMaltais I believe invoke.ai uses diffusers which is an established standard but it doesn't move fast enough in this incredibly volatile technology. Building a tool like invoke over comfy would be no easy task as there are no update cycles in comfy and your code could break any day. I hope things will get easier in a couple of years when the technology settles a little

[@tomschuelke7955]

@@BernardMaltais agree as an enduser i would welcome this. also i havent yet looked into the nodes of invoke. but never the less it looks promissing. i am teaching invoke to a big architectural company that starts to integrate image ai into their workflows. Here in my opinion invoke excels because of its unmatches ease of use. this is key in more complex company workflows, because each teammember has to deal with a dozen of different and often even much much more complex CAD and BIm tools. So an interface like comfy ui is somewhat far overdone for them. never the less things will become easier and we even cant imagine which programs and workflows in 5 years will excel

[@RamonGuthrie]

Hey great work and equally great videos too, sorry if you have mentioned this before in a previous video, but what is node/model or setting/workflow to focus on the face?

[@latentvision]

I believe you refer to the IPAdapter face model. I talk about it in the advanced video. It's doesn't need a special node, you just have to download and select the face model

[@hentrykulakkattil]

🔥

[@seditioussov9622]

Bro is fuggin Yoda of ComfyUI.

[@latentvision]

thanks for the lol 😆

[@simonmcdonald446]

My Mask Conditioning was not really working. Would like to see more on that as I did not get anything similar to your results. But the rest was very useful!

[@latentvision]

you probably need to lower the IPA weight a little to get mask conditioning to work

[@laslo67]

wow. just that... wow.

[@Deadgray]

Great job, simply WOW. Little question, can you maybe say something about "full face" adapter?

[@latentvision]

it is called FULL because it takes the whole 256 vector space to describe the face. As a comparison the plus models use 16 tokens. The result should be a very detailed face.

[@Deadgray]

@@latentvision tyvm, so that is why it sometimes needs lower weight in apply. Ok, one more question. In Encode IPAdapter image, best practice is that sum of weights needs to be 1.0 or it's just relative and it can be something like 1.0, 0.8, 0.5? Oh, and 1.0, 0.0, 0.0 gives different result than disconnecting 2nd and 3rd image from input in Encode IPAdapter node, bug or I don't understand how it works?

[@latentvision]

yeah at the moment you can set any weight (it's relative to the image). a weight of 0.0 is actually 0.00001 because otherwise we would send a zeroed tensor which is not the same as sending no image at all@@Deadgray

[@jasemali1987]

This is yet another great video, thank you so much !!! I have one so many questions as I'm new commer from A1111 to ComfyUi and I just can't have enough of it, it made me feel I'm free and the limits are my imagination. I was wondering, what method is best for IP adaptor, if I want to use many photos for different stuff in my final image without using inpaint of masks, I'm trying to make a robust workflow that can take inputs and get an image that is as close as to the original prompt as possible, can IP adaptor help me with this or it is more like Loras where somthing has to be picked in start to control the outcome. One way I'm thinking to use it is to use it in a looping fashion, like generate an image with basic prompt, use that image in IP adaptor, use new more specific prompt and generate an image, use that image in an IP adaptor with new more specific prompt and so on.. Can this method yield a good results ? It is like using very large prompt with high CFG but without the problem of burning the images.

[@latentvision]

I would start with the main elements and attention masking, then upscale and inpainting