Yes, in a way, it’s analogous to cross-attention, taking dot-product between the features from the text encoder and image encoder. This dot-product similarity is used as the final output of the model to determine if an image and a text caption are related or not. Good question, thanks for the comment

Thank you for the suggestion! You're absolutely right. In this video, I focused on purely algorithmic approaches, not hardware-based solutions like FlashAttention. FlashAttention is an IO-aware exact attention algorithm that uses tiling to reduce memory reads/writes between GPU memory levels, which results in significant speedup without sacrificing model quality. I appreciate your input and will definitely consider making a video to explain FlashAttention!

Cool, glad that was helpful, thanks for the comment

That’s a good question! Making keys and queries different helps with the modeling power. It allows the model to adaptively learn how to match different aspects of the input data (represented by the queries) against all other aspects (represented by the keys) in a more effective manner. But note that there are some models that use the same weights for queries and keys too. But having different queries and keys results in more flexibility and a more powerful model.

Thanks for the great question! Each of these matrices play a different role that makes attention mechanism so powerful. We can think of the query as what the model is currently looking at, and the keys as all other aspects in the aspects. So the dot product q and k determines the relevance of what the model is looking at currently with everything else. Once the relevance of different parts of the input is established, the values are the actual content that we want to aggregate to form the output of the attention mechanism. The values hold the information that is being attended to.

Glad you liked it!

Thanks for your comment, I am glad you like the channel 👍🏻

Thanks for the suggestion, yes I agree. I have briefly described axial attention in the vision transformer series kzbin.info/www/bejne/mJLZl5SVh9dlnJYsi=0SB9Yc_0SasafhJN

@@PyMLstudio thats awesome, thanks you!

Thanks for the nice comment, glad you enjoyed it!

Thanks, this video and some of my earlier videos are made with Python ManimCE package. But it takes so much time to prepare them , so my recent videos are made with PowerPoint

Thanks for your comment, yes that’s a good point. I’ll make include results in future videos , but for these paper I’ll write an article and include the results there.

Thanks for watching. This video was not made with PowerPoint. All the animations were made using Python and Manim package

That’s a good question. Relative position bias is typically added to the result of the dot-product of queries Q and keys K in traditional attention mechanisms. For linear attention, which aims to reduce computational complexity through low-rank approximations, you can integrate the relative position bias in a similar way by adding it after these approximations are computed. I've made a video that covers relative self-attention, which could provide some additional insights on handling position biases. Please feel free to watch this video for more details: kzbin.info/www/bejne/jpXPnneclpebm9ksi=h9bywcuPAs7mqCSD

@@PyMLstudio thank you for your detailed response, i was working on 3D version of CoAtNet, i adopted the relative positon bias from video swim transformer, i kind of understand the relative positon bias for 2D image, but not understanding the neaty details for 3D image, and the position bias added to dot product of Q and K was one dimension vector, which is bit weird. but later i replaced relative positon bias embedding and vanilla attention with axial embedding and axial attention. to me, the axial positon emddeding was much easier to understand for 3D image. and 3D CoAtNet with axial attention outperformed 3D Efficientnet with CBAM and SE in my case.

So, let's assume query=y, key=x, and value=rag for our explanation, but remember, you can adjust this configuration depending on your specific needs. Given these tensors, our first step is to ensure that the dimensions of the query, key, and value match for the attention mechanism to work properly. Since y has a different dimension (512) compared to x and rag (768), we need to project y to match the 768-dimension space of x and rag: query_projector = Linear(512, 768) query_projected = query_projector(query) ## --> 8x768 With this projection, all three tensors (query_projected, key=x, value=rag) now share the same dimensionality (8x768), making them compatible for the multi-head attention, where each head involves dot-product between query_projected and keys, followed by Softmax and multiplying by the values. Remember that the assignment of x/y/rag to query/key/value can change depending on your use case and where these tensors come from. I hope this answers your question.

Thanks a lot sir @@PyMLstudio

Thank you for your question - it’s indeed a great question. So X represents the input to a given layer, much like inputs in traditional neural networks. Specifically, in the first layer of a transformer, X is derived by calculating both the token embedding and the position embedding. For subsequent layers within the transformer, X is simply the output of the preceding layer.

Thank you for the encouraging words! I’m glad you’re enjoying the series. Your suggestion is spot on - I plan to cover building a transformer from scratch and training it with a small dataset, as well as delve into fine-tuning LLMs using PEFT. These practical insights will indeed bring the theory into real-world practice. Stay tuned for more!

Thank you for engaging with the content! In practice, these linear methods have proven quite effective, especially for very long sequences where full attention is computationally prohibitive. While they often come close, linear attention methods may not always match the quality of full attention, especially on shorter sequences where the full attention’s quadratic complexity is manageable. However, in scenarios with long texts or resource constraints, their efficiency and performance make them very compelling alternatives.

@@PyMLstudio Thx for the answer and definitely makes sense. It reminds me of choosing sort algorithms based on the length of data sequence.

Thank you for the positive feedback on my Transformers series! I'm glad to hear that you're finding it useful. I am currently working on publishing supporting articles for these videos on my Substack page (pyml.substack.com/). There, you'll be able to download the images and view additional figures and plots that complement the videos. Stay tuned for updates!

Thank you so much for your kind words and for waiting! I'm thrilled to hear that you find the explanations helpful. Your support means a lot, and it motivates me to keep creating content that makes complex topics more accessible. Stay tuned for more!

In the scope of this series, I plan to begin with discussing the evolution of attention mechanisms in the image domain, followed by an exploration into vision transformers. It's important to note that the Non-Local Module (NLM) is distinct from vision transformers. NLMs, particularly when coupled with residual connections, can be integrated into any pre-trained model, such as ResNet, as demonstrated in the original paper. This integration is designed to enhance the model without altering its fundamental behavior. Stay tuned as we delve deeper into Vision Transformers later on, and will see how self-attention mechanism is utilized in ViTs.

Thanks, your comment made it clearer for me.

Glad you liked it!

I'm glad to hear you're enjoying the Transformer series on my channel! Thank you so much for your kind words and encouragement. It really means a lot to me. Absolutely, I'm already working on the next installment, and here's a little teaser for you: the upcoming video will dive into the fascinating concept of relative self-attention. Stay tuned for more advanced insights!

That’s a very good question. I think of Non-local Module as a generalization of scaled dot-product attention. The proposed non-local block can be added to existing architectures for capturing long-range dependencies. But Transformers are different. We will cover different vision transformer models in this series . The previous introductory video shows the topics that will be covered