Deep dive - Better Attention layers for Transformer models

The self-attention mechanism is at the core of transformer models. As amazing as it is, it requires a significant amount of computing and memory bandwidth, leading to scalability issues as models get more complex and context length increases.
In this video, we'll quickly review the computation involved in the self-attention mechanism and its multi-head variant. Then, we'll discuss newer attention implementations focused on compute and memory optimizations, namely Multi-Query Attention, Group-Query Attention, Sliding Window Attention, Flash Attention v1 and v2, and Paged Attention.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at / julsimon or Substack at julsimon.substack.com. ⭐️⭐️⭐️
03:00 Self-attention
07:20 Multi-Head Attention (MHA)
12:32 Multi-Query Attention (MQA)
18:45 Group-Query Attention (GQA)
22:47 Sliding Window Attention (SWA)
26:17 Flash Attention
31:28 Flash Attention v2
34:36 Paged Attention
39:00 The Hugging Face LLM performance leaderboard

[@HieuNguyen-qs6ig]

Thanks Julien, your videos are really helpful.

[@juliensimonfr]

Glad you like them!

[@jacehua7334]

your videos are always so good thank so much congrats on 10k!

[@juliensimonfr]

Thank you so much!!

[@cybermanaudiobooks3231]

Thanks Julien! I really liked this video. Many thanks for sharing your knowledge and experience.

[@juliensimonfr]

My pleasure!

[@teddybear7949]

Thanks Julien, it really kept my attention and made me understand a lot about attention

[@juliensimonfr]

That was my goal, thank you!

[@justwest]

nice overview, thx julien!

[@juliensimonfr]

Glad you liked it!

[@thepresistence5935]

Julien outstanding video, where can I get the decks?

[@rbrowne4255]

Thanks for the overview!! Excellent!!! In terms of sizing for inference, is there a way to calculate the KV Cache or maybe the overall HBM memory usage base on these optimizations.

[@juliensimonfr]

You're welcome. The model papers have some numbers, but I think the best resource is huggingface.co/spaces/optimum/llm-perf-leaderboard. You can see how much RAM is required to load a particular model, and of course compare same-size models based on different attention layers :)

[@xinzhou3744]

you video is great! Could you paste all the code links in the description of the video? Much appreciated

[@juliensimonfr]

Hi, not sure what you mean. If you're talking about the pseudo code boxes , they're available in the research papers mentioned on the slides.