Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer

[@ankush4617]

👏 Keep up the great job, Umar!

[@rahulsawant2093]

I haven't seen a channel with such informative videos on Data Science. Please continue doing this.... Great thanks to you and the team.

[@varunsaagars]

Requesting SSS4 and Mamba explanations. Great work😊

[@HimanshuSharma-eg5li]

What's SSS4?

[@unclecode]

Structured State-Space Sequence (S4) or Selective State Space Model, sort of linearity for attention mechanism.@@HimanshuSharma-eg5li

[@umarjamilai]

You're welcome: kzbin.info/www/bejne/boLCpaStpbmjjLc

[@pratyushrao7979]

@@umarjamilaiBruh is too OP

[@mamotivated]

Absolutely well written, clearly explained and very valuable content as always Umar. Keep perfecting your craft. 100k subs by Dec 2024 , you are opening lots of doors in AI education.

[@RayGuo-bo6nr]

Thanks! Great Job! 谢谢 !

[@umarjamilai]

谢谢你的支持

[@pauldevillers797]

Amazing explanations, best channel around to dive deep into LLM !!!! Only note is that Mixtral7x8B paper clearly states that they did not observe any pattern on topic selection for a given expert, but they did exhibit some patterns on syntactic.

[@RudraPratapDhara]

Thanks for listening for the request made last time for moe, thanks. You explain and elucidate the stuff in a very understandable way

[@snowflareai]

Thanks!

[@umarjamilai]

Thank you very much for your support! Let's connect on LinkedIn

[@andikunar7183]

Amazing content, you are a great explainer/teacher, thanks a lot!!!

[@manishsharma2211]

One heck of a video umar, thank you. PS : @ 16:44 the kernel will move in the next 3*3 grid only when stride is 1 [ just FYI who might have doubt in this ]

[@goelnikhils]

What a explanation of Sliding Window Attention, KV Cache , Rolling Buffer Cache , Mistral . Amazing Work. Amazing Content. I have been following Umar and whatever content he creates that is top notch.

[@jman5447]

Thank you! Your clear explaination really make my life easier!

[@rajgothi2633]

Really good explanation... Please keep uploading such content. It inspire many researcher.

[@karanjakhar]

Great content. Well explained. Loved it. Please keep up the great job. Thanks.

[@wilsvenleong96]

Your content is god-given! I live for your content! Thank you so very much!

[@kozer1986]

Amazing!!! Simply amazing! Haven't seen a channel with such explanation on those topics!!!

[@jasonma3449]

exceptionally clear illustration on the SWA concept！

[@andikunar7183]

Danke!

[@umarjamilai]

Thank you for your support!

[@saeidghafouri8501]

Thank you Umar! Please keep up the great work!

[@unclecode]

👏 I support and subscribe to anyone who demystifies AI and helps democratize it. Keep up the fantastic job, Umar! Thanks!

[@umarjamilai]

Thank you very much for your support! I wish you, your family and loved ones a happy new year!

[@unclecode]

@@umarjamilai your welcome, I wish the same for you and your loved ones. Would you please let me know do you have any content focus on the transformer last step, where a linear layer picks up the next token based in the output of decoder. Basically the head MLP. Thx again.

[@umarjamilai]

@@unclecode If you watch my video on how to code a transformer from scratch, you will learn all about the transformer, including the normalization and the last layer. I believe the best way to learn a model is to code is from scratch and see it in action.

[@unclecode]

@@umarjamilai Roget that

[@akashkumar-jg4oj]

This is literal gold!!!

[@rraviteja]

Super content & explanation thanks please upload videos regularly

[@raahuldutta]

Again another great video😊

[@AndreasAlexandrou-to5pw]

Excellent as always. Thank you!

[@aam1819]

Fantastic explanation! Thank you!

[@ryan-reynolds-q3u]

Thank you! I understood a lot from this.

[@angelinakoval8360]

Thank you for the video, a lot of new information for me!

[@islamtorky1762]

Great work! Can you do a video for flash attention? Thanks!

[@snehotoshbanerjee1938]

@umar, thank you for this video. As always, it is full of knowledge. I was wondering how much hard work went in creating this video. Referring to the code is not easy unless a person went though the code and extracted the relevant pieces. There is no doubt the content and concepts are very complex.

[@Paluth]

Thank you very much, your videos are excellent as always. Keep up the good work, if you have the time!

[@alessiocaffi5992]

watching your vids is worth the time even for ppl not too much into AI yet. got here from trying to understand Karpathy's vids, great Job. Would be nice if someone on yt would make a vid on how to create an attoGPT/ attoLM or call it bookGPT (bookLM) from any book, e.g DanteGPT🙂 , so to train, on consumer PC without advanced GPUs.

[@utkarshjain3814]

bro is doing god's work. Keep it up!

[@michellem6685]

amazing explanation

[@anshul.singhs]

Thanks! was waiting for it, can you do mamba and S4 next?

[@hichamelkaissi7786]

Quality content.. Thank you immensely ❤

[@yukewang3164]

great explaination, very helpful, thanks!

[@AndreasAlexandrou-to5pw]

A question on batching; As far as I understand, batching inputs together has minimal cost on inference. I.e. 100 forward passes through all the decoder layers take roughly the same amount of time irrespective of your batch size. The video mentions that compute is wasted whilst calculating attention for the padding tokens, and thus concludes that unrolling the batch is preferrable? I don't see how this makes sense from a performance standpoint. Compute is very underutilised during attention, so the "wasted attentions" do not really cost anything. On the other hand, unrolling the batch increases the number of forward passes by your batch size. For example; a batch of 5 inputs with a length of 100, takes 100 forward passes in the first case, but takes 500 passes after unrolling. Am I missing something here? Doesn't unrolling completely nullify the performance boost from the wasted attentions?? Edit: Tested this: - Sq length: 1024, batch size 1: takes ~ 38 seconds. - Sq length: 1024, batch size 4: takes ~ 39 seconds. - Sq length: 4096, batch size 1: takes ~ 155 seconds.

[@harshitkumar5147]

This is just awesome!

[@prasannaprabhakar1323]

Thank you!

[@Itay12353]

You Are King!

[@kenilshah-hb6fy]

I have one point! At 5:46, table is shown in which 2nd row 2nd column. You have written No. of Encoder Layers. My question: If the Mistral is Decoder layer, then why we are considering 32 as the No. of Encoder layer ?

[@aamir122a]

Open source Multi model modal models (MMLLM ) are also becoming main stream , please do an episode on them as well.

[@jatinarora6680]

Very detailed explanation! Thanks for the video. Could you also make a video on vision transformers like BEiT.

[@baothach9259]

This video is so good!!!!

[@GrifinsBrother]

Amazing job, keep going!

[@XartakoNP]

Around min 14 - you explain that the sliding window attention will result in fewer dot products. From your explanation I derive that the sliding window mask is applied after the Q@Kt operation, where we perform all the dot products within the Q and K tensors. Is that operation fused in some way or is there a trick to achieve it the reduction in the number of dot products?

[@waynelau3256]

Hey Umar, great video! I have some questions, how does SWA work at training? Because I am trying to wrap my head around how the previoius context is fed to the window. From my understanding in the mistral model, one of the tokens is catered to the previous attetntions. In this case, wouldn't this make it autoregressive and not parallelizable, because the previous attention needs to be computed?

[@cfalguiere]

Thanks for sharing

[@Yassjams]

Amazing video ! can you do Falcon architecture explanation 🙏🙌

[@trungquang1581]

great job, thanks a lot man

[@random-ds]

Thank you for this great video. I have a question though. When mistral released the intruct-v2, do they follow the exact architecture and change only the data and way of training, or, they can also twist a little bit the classic architecture of mistral? Thanks in advance!

[@gangs0846]

Helped alot thank you

[@lukeskywalker7029]

Another great one! Any chance you'll take on "The Era of 1-bit LLMs" paper next? ;)

[@subhamkundu5043]

Amazing content. Are you going to put some video on coding a MOE model from scratch?

[@snehotoshbanerjee1938]

@umar, one question... In the current setup, the current chunk has the attention scores w.r.t. previous chunk. But, is it not losing the attention score w.r.t. N - 2, N - 3 etc chunks? I mean is the attention score till the previous chunk is enough? Or are we saying that it has a rippling effect from left to right?

[@vinc6966]

Great video, but I have two questions about sliding window attention: 1. How applying mask to tokens outside of sliding window attention makes it more efficient? Since we still have to perform calculations on NxN matrix, but with some zeros. Are floating point operations on zeros faster? 2. Receptive field increases as depth increases. Consequently, in mistral only last layer can attend to all tokens, so tokens have less time to communicate. If we have a task that requires N steps to be solved and ALL OF information from the tokens, will the model be able to solve it? Thanks

[@umarjamilai]

Hi! 1. When you know that the two matrices you're multiplying will have many zeros in the output, you can use the "sparse attention", which basically represents matrices in a way very similar to Python dictionaries, so we only store the values of the non-zero indices. There are many deep learning frameworks that support sparse matrix multiplication, if I remember correctly DeepSpeed supports sparse attention calculation. 2. It is wrong to say that the last layer will attend to all tokens. One token only attends to W preceding tokens, where W is the size of the sliding window. But because of how the information gets "accumulated" in the embedding after each layer, we can claim that the information "flows" from one token to another even if they are outside the window. You're right in saying that the information that's carried this way is less "strong" (it's like you hear a news from a friend instead of reading it by yourself on the newspaper: every intermediate person will alter the real story). If a task requires the information of all the tokens, it MAY (we can't be sure) still able to perform it, but it all depends on how many layers you have and what's the size of the sliding window. Have a nice day!

[@vinc6966]

@@umarjamilai Okay, I think that answers my questions ;) Thanks a lot!

[@alexis91459]

Awesome! So is it true that KV cache length should be of the same size as the sliding window attention?

[@umarjamilai]

Yes, because anyway the model cannot "attend" to anything beyond the sliding window size, so the KV Cache size is limited to it.

[@mihirrege206]

Thanks!

[@pratyushrao7979]

I had a query regarding the rolling buffer cache. Why did they not use a Queue for storing the vectors instead of a rolling buffer cache? I know there's an issue with the implementation of a queue, but wouldn't that be time wise way less complex? Instead of O(n) you can roll back in O(1).

[@umarjamilai]

You can implement it however you like, but you should always avoid shrinking and growing tensors, because it may move data around the GPU memory, which is slow.

[@pratyushrao7979]

@@umarjamilaiOkay thank you. Your explanation was great!

[@amitshukla1495]

Wohooo 🥳

[@MrNathanShow]

Is the xformers part primarily used for training or more for just if we had a service and wanted to support the generation of the outputs. Also, for each expert are they trained independently? Or are they trained with the same dataset? From what I understand the MOE layer is just a feed forward lin layer that are weights. I think I might be wrong though... Thank you!

[@MrNathanShow]

Ok, so each "expert" is technically just a feedforward output that is gate controlled by a linear series of weights. The top two are selected to post process the token at the end.

[@MrNathanShow]

The whole data set is used to train each of these experts.

[@siqb]

When we are training or even inference and use as input "[SOS] Love that", do we use the embedding of 'that' for passing to the softmax to predict 'can'?

[@umarjamilai]

Only during inference. During training you just compare the entire output with the target to calculate the loss.

[@سودانتوك]

Great content as always. can you do a video about ControlNet?

[@elieelezra2734]

Hi Sir, great work as usual. However, I have a question regarding the gate in the 'Sparse Mixture of Experts' section. Is it a simple one layer network that produces 8 logits? Thanks! Keep up the good work !

[@umarjamilai]

Yes, for every token in the sequence it produces 8 numbers. The two highest numbers indicate which FFN the token should run through.

[@elieelezra2734]

Correct me if I'm wrong, it means that the behavior of this kind of block is not the same during training and during inference. During training token embedding goes through the 8 feed forward neural networks, then the output of the two best are selected according to the output of the gate, whereas during inference, the embedding token goes through the two best feed forward neural networks according to the gate. Again thanks a lot for your time and your explanation, I really appreciate it@@umarjamilai

[@tryit-wv8ui]

Yep the same question here@@elieelezra2734 @umarjamilai

[@tryit-wv8ui]

Is the next assertion from elie elezra below is correct@@umarjamilai ?

[@umarjamilai]

@@tryit-wv8ui hi! The behavior during training and inference IS EXACTLY THE SAME: what I have shown for inference is exactly what happens during training. Because that's how the gate function is trained in producing logits and selecting the best feed forward networks for each token and that's also the reason why some feed forward networks will "specialize" in particular subsets of the tokens (for example some may specialize on Japanese tokens, others on English tokens etc..)

[@Anson-rr6ej]

Great videos. Are the 8 experts and gating funtion in each layer are different ? So total there are 8 x 32 experts, is this correct?

[@umarjamilai]

Yes, each layer has different experts: 8 per layer, so in total 8x32.

[@Anson-rr6ej]

@@umarjamilai Thank you!

[@ihitsuperhuman3227]

thanks

[@sahilc7750]

is there a way to learn different boiler plate codes and how they operated provided by Xformers ? There github is not very intuitive.

[@madhusudhanreddy9157]

Please create a one vecotr database with LLM RAG Implementation video sir

[@zhenfutaofang2534]

Amazing Video !!! 加油

[@umarjamilai]

谢谢你！我在中国有个微信小组关于AI和深度学习，你想交流在领英给我发消息，我Invite你参加。

[@zhenfutaofang2534]

ok@@umarjamilai

[@ЖирайрАйрапетян-щ2у]

At 9:25, why are Q and K the same matrices in the case of self-attention? There are different linear layers for mapping the input sequence to queries and values, isn't there?

[@umarjamilai]

I recommend you watch my previous video on the Transformer, where I explain the origin of the Q, K and V matrices.

[@siqb]

Yup. Q, K, V are 3 different projections of the input. If they were literally the same, the QKt will be a symmetric matrix.

[@umarjamilai]

@@siqb you're right. I should have mentioned that. Because I was talking about the "tokens" they "refer to" and not to the single values they are made up of, it may have caused confusion. Thanks for pointing out

[@avogadroarts4366]

Thanks

[@GrifinsBrother]

But your explanation about specialising of experts is wrong. Because it is stated in the paper, that knowledge of each expert is distributed equally and there is no any specialisation. Check "Routing analysis" block of the paper.

[@umarjamilai]

The paper on the actual performance of the mixture of experts came AFTER I published my video. What I was talking about is not what happens actually (since I didn't have the data on the actual performance back then), but on what's the intuition behind creating a mixture of experts: the idea is that each model - hopefully - specializes in a subset of the data. It may also happen that each model does NOT specialize, like in the case of Mamba. I believe the authors of Mamba also hoped in some kind of specialization, but in reality it didn't happen.

[@邱雨-e1u]

amazing

[@reginoldlu]

Thanks!

[@reginoldlu]

谢谢！Request the flashattention and falshattention2! keep working!!😀

[@reginoldlu]

I just connected with you on linkedin

[@haralc]

Thanks

[@ml.9106]

Thanks!

[@farzinhaddadpour7192]

Thanks!

[@umarjamilai]

Thank you very very very much!