Cross Attention | Method Explanation | Math Explained

[@AICoffeeBreak]

Wow, what a great video this is, thanks! You do awesome explanations: you put things well into a nutshell but also spell them out later with awesome visuals. 👏

[@outliier]

Thank you so much! So nice to hear that!

[@ayushroy6208]

Hi ma'am , big fan .... A subscriber

[@AICoffeeBreak]

@@ayushroy6208 Thanks! Happy to see you here! 🙌

[@InturnetHaetMachine]

Another stellar explanation. Hope you continue making videos.

[@Mike..]

had to pause a rewind a lot and look stuff up, but now I feel like I have a much better understanding of how this stuff works! Thank you

[@ammarulhassan5851]

You are a gem. One of the finest teachers on KZbin. Thank you for recording these amazing videos.

[@SaidAzizov-c3l]

The best DL explanation content I've ever seen!!! Thank you so much, sir!

[@fritmore]

wow, excellent visualisation and explanation of the algebra ops inolved

[@deeps-n5y]

Story telling skills and animation are exceptionally impressive.

[@송지빈-r4h]

Thank you so much!!! I was finally able to understand Attention with your video!!!

[@ahmadhassan8560]

cleared my doubts for cross attention in a flash. Thanks a'lot fam

[@Melina36295]

The tone of your voice and your explanations are excellent.

[@kadises5032]

one of the best video ive ever seen

[@JidongLi-lb3zt]

thanks for your detailed introduction

[@Crouchasauris]

Great explanation!

[@TheSlepBoi]

Amazing explanation and thank you for taking the time to properly visualize everything

[@rajpulapakura001]

Thanks for the crystal clear explanation!

[@sureshmaliofficial]

This is really great. make it so easy to grasp such a complex stuff

[@RomainGratier]

Well done, just the right amount of details for anyone to understand the main concepts quickly!

[@cupatelj52]

Very well explained. Thank you sir.

[@420_gunna]

Incredible, please create more!

[@mr_tpk]

Thank you for this. Please continue making videos.

[@bigtimetimmyj]

I've struggling to understand image and textual cross attention. This video helped immensely, thanks a lot.

[@moajjem04]

Thank you very much. The video cleared the concept of cross attention a lot.

[@felixvgs.9840]

Another awesome watch, keep up the awesome work!!!

[@juandiegocastanogomez3854]

Great quality! How can you accomplish this visuals?

[@evilby]

Just the content I'm looking for! great video! Please keep up the good work!

[@孟星辰-c2e]

Thanks for explanation really helping!

[@kevon217]

Very intuitive explanation, thanks!

[@jingyang4949]

This is really clear!!

[@outliier]

Thanks!

[@RyanHelios]

really nice video, helps me understand a lot❗

[@matthewprestifilippo7673]

Great content

[@miquelespinosa8704]

This video is truly amazing!! Thanks!

[@masbulhaider4225]

Great explanation. But in case of image to image translation using U Net how i can use cross attention. Suppose for faded image to colorize image conversion how to prepare query for cross attention and should the K and V come from the colorized image as conditioning image? If yes should i use a transformer encoder to get K and V?

[@rikki146]

12:17 While I understand Q and K are just matrices to calculate similarities, why is V text instead of latent representation of picture? I thought Softmax(QK^T/sqrt(d))V is an input to a UNet that accepts latent representation of a picture? Or is the UNet trained to accept weighted word embeddings instead? From what I understand, the UNet accepts a latent representation of image as input and outputs predicted noise (in latent space). So why would weighted word conditions work? Let me know if I have any misunderstandings... thanks.

[@outliier]

Hey. The Cross attention is only used as blocks in the UNet. So its part of it. The input to the the UNet is the noise and then usually you have a stack of ResNet, Attention and Cross Attention blocks. And the Cross Attention blocks are used to inject the conditioning. And V is text because the latent representations of the image can attend to the word tokens this way and can choose where to draw information from. Does that help?

[@rikki146]

@@outliier To be honest, I am still puzzled lol. I will go back and study closer. I just thought in order to predict noise your input must be a noisy image (in latent representation), instead of weighted word embeddings. Thanks for the answer though.

[@outliier]

@@rikki146 and thats totally right. The input is a noised image. The text only comes in as a conditioning through the cross attention to give the model more information what the noised image could be. And the way this works is by letting the image pixels attend to the word embeddings. And then through the skip connection the pixels will be modified based on the word embeddings

[@alejandrorequenabermejo6608]

Amazing video!

[@任东皮]

Thanks for your video. It's so great!

[@aurkom]

Thank you for the video!

[@raphaelfeigl1209]

Amazing explanation thanks a lot! Minor improvement suggestion: add a pop-protection to your microphone :)

[@dan1ar]

Such a great video! Instant subscribe)

[@leeda36]

awesome, thanks!

[@r00t257]

well come back!

[@george6248]

Very helpful video👍

[@alioo4692]

thank you very very much

[@XuLei-ml2vd]

You explain it so clearly🤩.But I still have a question.Why is the text condition Convert into K,V, and the image Convert into Q？Why do they correspond to each other like this？Can image cobvert into K,V? look forward to your reply

[@outliier]

Hey there, thank you so much for the kind words! The reason why the text becomes k and v is because you first want to attend to the image. If you take the text to be q and v for example then the final shape of the output will not be correct. q determines the shape of the attention output. I had the same question once, but a friend explained this to me too! It might sound a bit confusing, but just do a calculation on paper and try to switch around the matrices for q k and v and you will see q has to come from the image and k, v from the extra conditioning. Let me know if that helps, otherwise I can give a concrete example :D

[@XuLei-ml2vd]

@@outliier Thank you very much for your reply. 😊I did it on paper. I found that the first dimension of the attention depends on Q, so in order to keep the output consistent with the dimensions of the original image, image must correspond to Q. But I have another question, can the matrix shapes of WCA^K and WCA^V be different?For example, set their shapes to (6,2), (6,3) respectively, and then set the subsequent Wout matrix shape to (3,3).Looking forward to your reply :）

[@outliier]

@@XuLei-ml2vd Hey! That's exactly what I was thinking about too. In theory yes, I don't know if people are actually doing it. Because in my mind, it could make more sense to use a smaller dimension for K since it "only" contributes to similarity lookup and a bigger dimension for V. But it is important to keep in mind that similarity lookups perform best in high dimensions, so a dimension of 10 probably is way to low to embed the inputs. But maybe there is a sweet spot that can be found which performs better than giving K and V the same dimensions. Would love if someone would make a comparison (or maybe there already are plenty?)

[@XuLei-ml2vd]

​@@outliier Thank you so much for your reply! I really appreciate it✨

[@akkokagari7255]

Wonderful explanation! Not sure if this is in the original papers, but I find it very odd that there is no nonlinear function after V and before W_out. It seems like a waste to me since Attention@V is itself a linear function, so w_out wont necessarily change content of the data beyond what Attention@V already would have done through training.

[@akkokagari7255]

Whoops I mean the similarity matrix not Attention

[@PeterJacobFloratos]

Amazing video and great visual! Nice work man. I have two questions for you: Why it's the case that with cross attention we can aquire higher evaluation to our model? In which case (regarding the nature of the conditional information) those models perform better that self attention ones? Thank you and keep the nice work

[@outliier]

Hey there. Thank you for the nice words. What do you mean with “higher evaluation”? Do you mean a better performance? For the last part, you can not really compare self attention and cross attention in that way. They both are attention, just between different things (self attention between the input itself and cross attention between input and conditioning). So you can not really say that cross attention performs better than self attention, because they do different things. Maybe I understood it wrong tho, so feel free to help me understand it

[@PeterJacobFloratos]

@@outliier no you get my point, thanks a lot 🙏

[@vaibhavsingh1049]

Thanks for the video. Can you explain how is an extra image (like in img2img) along with text input is used in cross-attention? In all there are 3 inputs, the noisy latent at x_t, the conditional input image and text. How are they mixed together?

[@outliier]

The normal image2image doesn’t use the image embedding in cross attention. You can just take your image, noise it (about 60-80%) and then just do the sampling from there. So the image is in no way used in the cross attention

[@youtubercocuq]

perfect video thanks

[@FrankJhang]

Thanks for the explanation, it's awesome! But I have a question. In cross attention, if we set the text as V, the final attention matrix could be viewed as a weighted sum of each word in V itself (the "weighted" part comes from the Q, K similarity). If I understand correctly, the final attention matrix should contain the values in the text domain, why can we multiply by a W_out projection and get the result in the image domain (add it to the original image)? Will it make more sense if we set the text condition as Q, and the image as K, V?

[@outliier]

If the text conditioning is q then it would not have the same shape as your image. So q needs to be the image

[@malavbateriwala9282]

Great explanation. Which topic are you planning to work on next?

[@outliier]

Paella 🥘

[@leerichard5542]

Long time no see!

[@harshaldharpure9921]

I have two feature x--> text feature , y--> image_feature and rag --> rag_feature (This is Extra feature) I want to apply the cross _attention between (rag and (x/y)) How should I apply

[@YASIZHANG-f4q]

Hi I really enjoy your video! Can you make a video on how you make high-quality videos like this?😃

[@kaanvural2920]

That's a great explanation and presentation, thank you. I have a question about "condition" part. Does cross attention use as image information instead of text also? If yes, what is the condition means for images then? Thanks in advance

[@outliier]

Do you mean if its possible to use cross attention just with images when not having any text?

[@kaanvural2920]

@@outliier Yes, I was tried to mean that 😅

[@outliier]

@@kaanvural2920 then it naturally becomes self-attention. What I explained in the first part of the video

[@kaanvural2920]

@@outliier I remembered that I came across cross attention when I was searching about detection or segmentation models. Maybe I am mistaken, I'll check. Thanks again 🙏

[@outliier]

@@kaanvural2920 ah I think I understand. As soon as you are talking about conditional information, in any form (either text information, image information (eg in the form of segmentation maps) or anything else) you can use cross attention

[@EngRiadAlmadani]

the best from now on i will not put the self attention in from of large feature map it will take long time😆

[@afasadaaffa3321]

Nice video! Why do we take Values from text if we want to produce image?

[@NoahElRhandour]

yo ist das manim oder wie hast du das alles (zahlen in den matrizen oder zb die pfeile) gemacht??? crazy gut!!!

[@outliier]

Ja mit manim

[@Tremor244]

The math makes sense, but how it results in a coherent image will forever be a mistery to me lol

[@lewingtonn]

this channel is certainly an outlier in terms of how good these fucking videos are

[@xiaolongye-y4g]

[@VerdonTrigance]

I'm lost on Self Attention. Man, you are too fast! Feels like u r telling it for someone who already knows it, but for others it takes time to chew on it. At the moment when I understand previous phrase you are much far away telling about something else. At the end I still don't get it, but thanks for you try. Especially for mathematic approach.

[@juansmith4217]

'Promo sm'

[@swaystar1235]

naruhodo (does not naruhodo)

[@saeed577]

Great presentation video, bad explanations.

[@outliier]

Thank you. Let me know what I could do different in your opinion for the explanation part

[@jayhu2296]

Thanks for the crystal clear explanation!