Mamba and S4 Explained: Architecture, Parallel Scan, Kernel Fusion, Recurrent, Convolution, Math

[@andrewhaslam8785]

Brilliant - you are easily one of the most lucid and accessible teachers of deep learning.

[@ItsRyanStudios]

this is absolutely FANTASTIC I watched Albert Gu's stanford lecture on state space models/ Mamba, and it was a great high level overview. But I really appreciate you taking it slower, and going farther into detail on the basic/ fundamental concepts. A lot of us aren't mathematicians or ML engineers, so it's much appreciated to be helped along with those concepts.

[@umarjamilai]

Thank you for your kind words. Please share the video in your network, it would help me a lot. Thanks!

[@danaosama4247]

I rarely comment on videos, but this one was worth it. Thank you so much for such a clear explanation. You explained all the nuances that I previously did not understand in a very clear way. God bless you.

[@remyshootingstars]

🙌 Still working through Transformers from scratch. Hopefully a Mamba from scratch is in the future!

[@anirudh514]

Your teaching approach is very good. You started from fundamental concepts and went deeper. This helped in gaining intuitions, understanding and avoid confusions in later part. Brilliant!

[@sid-prod]

I'm so glad i found this channel, you are a gold mine for such content, please keep them coming.

[@trungquang1581]

I just read about mamba and wanted to find a detailed explanation video. All you covered in this video is everything I need, thank you so much, keep on cooking

[@RabeehKarimiCH]

Best ever presentation I saw so far, thanks so much

[@aruns.v9248]

The whole lecture was very intuitive. Thanks for the efforts put into building this video!

[@Frederickawuahgyasi]

You're amazing. God Bless you. You made this the best hour i've spent on trying to understand MAMBA. Keep up the great work.

[@SatyanarayanSenapati-b1s]

Words will fall short to appreciate the work you put to create these videos. Simply BRILLIANT.

[@Andy-paw-Jessica]

You are just too amazing! You can understand these stuff in great detail. Then you take the time and explain to us in educative videos. A true gem channel!

[@trevorhobenshield]

Very high quality, this is great. Hard to find good content like this. Thanks Umar!

[@sari54754]

After I saw this lecture, I subscribed your channel. It is the most easy to understand Mamba lecture I've seen.

[@AUTO-g7s]

作为一个来自北京的大学生，谢谢你分享的这篇文章解析！best wishes！

[@arvyzukai]

This is gold! I really appreciate attention to the details. Thank you Umar!

[@ankush4617]

Thanks for the amazing work as usual! Keep it up - this is probably one of the highest quality content on LLMs on youtube.

[@mudassirkhan9054]

Thanks for explaining it in a way that anyone with some high school math background can understand, keep this up!

[@The_bioinformatician]

This is the best deep learning video I've ever seen. I will surely use some of your slides to teach my students

[@optomosprime]

Excited for the video. I was searching for a video on Mamba and today I saw this. Your Transformer video helped me alot previously. Keep it up!

[@周毅-b1h]

I'm very thankful for your explanation of this article, best wishes for you!

[@myfolder4561]

Thank you so much. Lots of useful details yet you curate through them at such a good tempo with easy to follow examples

[@celestchowdhury2605]

Thank you so much for your detailed video and thoughtful thinking of you that we will need help with the equations! You are a savior!

[@fabiogomez8250]

Best MAMBA video at the moment!

[@beincheekym8]

Brilliant video! Really clear and with just the right amount of details!

[@흰강아지-s4v]

this is just a pure art; thanks so much

[@majidemami577]

Excellent video! Thank you. I have watched a few videos about mamba and this one was by far the best.

[@purohitadey-bc9bg]

Understanding mamba couldn't be better than this !

[@raaminakbari]

Thank you for this great and smooth explanation. I think the model you are showing at 36:14 is valid if matrix A ( and B also to send each input directly to the corresponding ssm) is diagonal. Now in this way each hidden state at different canonical direction ( or different element of the vector) is independent of each other. So if A is not diagonal then assuming an eigen decomposition exist, then we may say there exist an equivalent ssm which can be represented independent ( if we change the basis to eigen basis) .

[@Mirai12377]

very good video!!! thanks a lot for your efforts!!!!

[@mcHsyu]

Great explanation!! This is the first video that mekes me comprenhad the whole mamba paper.

[@selayan4985]

Such a briliant work you have done. Really learned a lot, thanks!!!

[@ankush4617]

Thanks!

[@GenAiWarrior]

Thank you so much for your efforts to make such an amazing video on Mamba architecture !!

[@nishanthshetty435]

Thanks a ton! Excellent explanation and great analogies to introduce the more advanced material. This is an absolute masterclass on how to teach advanced material.

[@akshikaakalanka]

This is really helpful for another talk I am doing on Mamba. Thank you very much for putting this out.

[@ActualCode0]

This is one of the best ML explanations I've seen even though I didn't understand all of it but I definitely learnt something new.

[@wayneqwele8847]

Thank you. I appreciate the approach you took in explaining the major concepts.

[@SpandanMishra-z4r]

OMG ! this is such as amazing description , you made my day

[@Erosis]

As others have mentioned, you have a keen ability to explain difficult topics succinctly and completely. Keep up the awesome work! I could of used this when I took a class on time-series modeling! Hah!

[@TheRohit901]

Amazing explanation. I love this video because it covers sufficient depth and explains each concept with proper examples. I've subscribed instantly, and look forward to more such videos on recent papers.

[@belamipro7073]

Danke!

[@umarjamilai]

Thank you very very very much for your generous support! Let's connect on LinkedIn!

[@danamics]

Great job on this video! I learned a lot

[@soroushmehraban]

Love it! Keep up the amazing work.

[@prashlovessamosa]

Salute to consistency Thanks Umar sir.

[@tunatuncer5639]

wow that's a great explanation , thanks for the efforts!

[@Hello-tx7ug]

Thanks!

[@shoubhikdasguptadg9911]

Ohhh Man, why did I discover this gem so late :( This guy is a rockstar!

[@kwanhowong5065]

Really an amazing video! You save me a lot of time! Thank you!

[@BooleanDisorder]

Even I understood much of this. I have no education. Thank you! Mamba looks really cool. Especially like the long context and further refinement. It looks like a model that could be made to learn as it goes. Plasticity potential

[@lewylondon]

I did learn a lot! Many thanks for making this video.

[@我我-p3z]

最清晰的讲解！

[@EkShunya]

i always eagerly wait for your explainer. they are 🤯. thank you :)

[@umuthalil5001]

Hi, I was wondering if you could explain 36:40 a bit more where you talk about multi head attention. From what I understand each head in multi-head attention each head looks at the whole input vector. Our key value and query matrices are all of size Dx(head_size) where D being dimension of embedding, so when we find key say we do key = X @ key_matrix where X is an CxD dimensional matrix, C is context len. This means each head looks at the whole dimension of the embedding D and represents it a head_size vector meaning that arrows going into each head should point at every single input dim.

[@whisperlast6548]

This video is of great help!!Thank you very much.

[@bryanbocao4906]

Thanks for the video! Very informative! Just to check: At @1:03:42, 3. be "... save back the result to HBM."?

[@bulat_15]

Thanks man! This helped me a lot

[@divgill6062]

Amazing! So detailed. Well done sir

[@a123s1l]

Thanks for your clear explanation of MAMBA, coming from a control theory background, very much appreciate its usage in LLMs. One small error that I noted was that the A matrix must be N x N to translate the previous N-dimensional hidden states h(t-1) to h(t). I believe the A matrix is also time-varying to produce selective output tokens.

[@TheFitsome]

some people are just born to teach.

[@walidmaly3]

One of the best! I have one question if we apply conv in S4 on sequence of length L, what will be size of conv layer?

[@m1k3b7]

Brilliant explanations. Thanks.

[@jason988081]

Dear Umar, referring to 53:50, recurrent SSM is indeed similar as prefix-sum (i.e., y=x_0+x_1+....x_N), but I the difference is that h_t=Ah_{t-1}+Bx_t, where h_{t_1} depends on h_{t-2}. I know how Blelloch parallel prefix scan works for calculating the sum of constants, but I do not know how parallel scan works for h_t=Ah_{t-1}+Bx_t. Could you please elaborate on it ? Thank you. @Umar

[@deepikagurung9410]

are going to code it as well. I really liked the video it was easy and very comprehensive.

[@810602jay]

Thanks Umar! 🥰Very amazing learning material for Mamba!

[@samuelbeaussant3097]

Very good lecture ! Thank you very much for putting this for free on youtube :) I have question though, if my understanding of the HiPPO framework is correct, the A matrix is built to uniformly approximate the input signal (name HiPPO LegS in the paper). "Our novel scaled Legendre measure (LegS) assigns uniform weight to all history [0, t]". But however at 41:49 you explain that it is decaying exponentially similarly to HiPPO LagT. Do they opt for HiPPO LagT when moving to s4 and Mamba or am I missing something ?

[@mahmoudreda5054]

thank you for this video , really helped me

[@akashkumar-jg4oj]

Great explanation!

[@nguyenhuuuc2311]

Thanks for the awesome content! Hope the next one will be about DPO and coding it from scratch ❤

[@umarjamilai]

You're welcome: kzbin.info/www/bejne/nqeqkmiDl8ZnmZo

[@nguyenhuuuc2311]

@@umarjamilai Thank you!!! You're so talented at research and teaching!!!!

[@amitshukla1495]

Absolutely amazing 🎉

[@erfanasgari21]

In 57:00 isn't the time complexity reduced to 2*lg(n) in parallel scan? Thanks for the amazing explanation btw. 💚

[@immakiku]

Trying to follow the rationale/last-slides - one advantage of SSM/RNN was that they would scale to infinite context. But Mamba reintroduced L-lengthed parameters. Why is this not limiting to this architecture the same way it limits Transformers? Qualitatively, it seems the only remaining advantage over transformers is the inference is cheaper - could you help clarify? Thanks

[@alainrieger6905]

Awesome video as usual

[@pcwang7803]

Great lecture! It is easier for me to understand the work with your lecture. Can you give one for Reinforcement learning?

[@allengeng6660]

Very nice talk, thank you.

[@sayandas13]

Awesome explanation. Really appreciate such content. Can you please make a similar explanation video on the Mamba-2 paper?

[@eafadeev]

You're making very useful content, thank you!!! Maybe you could consider using larger text, so that one could read easily from a phone. Also a plus would be if the presentation were white on black (or bright color on black), it is less tiring to look at a dark screen for long periods of time.

[@팽도리-v6s]

Amazing video.

[@venkateshr6127]

Please can you make video on optimizers like adam,adagrad,...

[@Huawei_Jiang]

I have one question in terms of the example which you provided, 'the number of buddies'. I think the function should be like this : b(t)=5squ(3)^λt . please comment to me if I am wrong.

[@杨辉-l2g]

excellent work! Thank you

[@pawanpatil4715]

Hi Umar, amazing video. You are the best teacher. You are Karpathy 2.0. :) Please make a video on DPO :)

[@umarjamilai]

Done: kzbin.info/www/bejne/nqeqkmiDl8ZnmZo

[@pawanpatil4715]

@@umarjamilai thank you so much 😃

[@andreanegreanu8750]

Hi Professor! Very good explanation as always. However, I have huge difficulties to understand the dimensions of objects. Why the hell A matrix would be of (D,N) dimensions since it is used to project a vector h_t-1 of N dimensions into N dimensions? By the way, why is it written "Represents structured N x N matrix" ?????!!!!

[@abrahamsong6913]

this is so far the only video I found that described the math part in the mamba model. thanks a lot. One small issue. In 37:00, for the attention model, you mentioned each head takes only a portion of input dimensions, can you confirm this? I believe each head actually use all input dimensions.

[@abrahamsong6913]

It might be true for LLMs, but I believe this is not true for the original transformer model.

[@umarjamilai]

Hello! First of all thanks for the kind words. Yes, in multi-head attention, the idea is that each head sees the entire sequence, but a different portion of the embedding of each token. This is to make each head relate tokens in different ways. This mechanism is described in my previous video on the Transformer model.

[@rezagholipoor7900]

It was very informative

[@GoogleColab003]

absolutely fantastic

[@albertmashy8590]

Amazing video

[@artaasadi9497]

Thanks a lot that was very useful!

[@undefined-mj6oi]

Hey! Thanks for the details in this video. I'm confused about the HiPPO matrix, which seems to be fixed given N? However the paper stated that delta, A, B, C are all trainable. What did I miss?

[@undefined-mj6oi]

is HiPPO the initialization of A?

[@umarjamilai]

Yeah, just the initialization

[@undefined-mj6oi]

Thanks for clarification. Could you please further explain how the parameter of A is (D, N) in S4? If I have D*SSMs, one for each embedding dimension, shouldn't A have DN^2 parameters?

[@buh357]

you are the best.

[@ShubhamAshokGandhi]

Great explanation. Very through. Loved it. I struggled with understanding the SSM paper. You explained all the bits beautifully

[@andreanegreanu8750]

Anyone here to tell me what is the dimension of h_t? I thought it was a vector (D,1), but according to the slides, it seems it is a matrix of (D,N)!!?? Thanks in advance!

[@123456ewr]

Thanks, i hope you explain rwkv

[@passarodavide]

Bellissimo video, grazie!

[@umarjamilai]

Grazie a te!

[@toxicbisht4344]

amazing explanation waiting for new video please upload soon

[@НикитаБуров-ъ6р]

i've just started watching but guess this vid'll be much usefull

[@edsonjr6972]

Excellent video! I'm looking forward if you do a coding one. Thank you so much for your work to the AI community

[@umarjamilai]

Coding one is not very interesting, because the most interesting part is the selective scan algorithm, which is a CUDA Kernel. The architecture is not so different from any other language model. Of course it would be super cool to code the CUDA kernel from scratch ;-)

[@SandeepS-i4e]

Great❤

[@Huawei_Jiang]

Can we run Mamba via normal GPU?

[@RaghaVamsi]

You are amazing! How did you learn all this?