Mamba and S4 Explained: Architecture, Parallel Scan, Kernel Fusion, Recurrent, Convolution, Math

[@andrewhaslam8785]

Brilliant - you are easily one of the most lucid and accessible teachers of deep learning.

[@ItsRyanStudios]

this is absolutely FANTASTIC I watched Albert Gu's stanford lecture on state space models/ Mamba, and it was a great high level overview. But I really appreciate you taking it slower, and going farther into detail on the basic/ fundamental concepts. A lot of us aren't mathematicians or ML engineers, so it's much appreciated to be helped along with those concepts.

[@umarjamilai]

Thank you for your kind words. Please share the video in your network, it would help me a lot. Thanks!

[@danaosama4247]

I rarely comment on videos, but this one was worth it. Thank you so much for such a clear explanation. You explained all the nuances that I previously did not understand in a very clear way. God bless you.

[@anirudh514]

Your teaching approach is very good. You started from fundamental concepts and went deeper. This helped in gaining intuitions, understanding and avoid confusions in later part. Brilliant!

[@remyshootingstars]

🙌 Still working through Transformers from scratch. Hopefully a Mamba from scratch is in the future!

[@trungquang1581]

I just read about mamba and wanted to find a detailed explanation video. All you covered in this video is everything I need, thank you so much, keep on cooking

[@sid-prod]

I'm so glad i found this channel, you are a gold mine for such content, please keep them coming.

[@SatyanarayanSenapati-b1s]

Words will fall short to appreciate the work you put to create these videos. Simply BRILLIANT.

[@Frederickawuahgyasi]

You're amazing. God Bless you. You made this the best hour i've spent on trying to understand MAMBA. Keep up the great work.

[@jiegong529]

You are just too amazing! You can understand these stuff in great detail. Then you take the time and explain to us in educative videos. A true gem channel!

[@aruns.v9248]

The whole lecture was very intuitive. Thanks for the efforts put into building this video!

[@AUTO-g7s]

作为一个来自北京的大学生，谢谢你分享的这篇文章解析！best wishes！

[@sari54754]

After I saw this lecture, I subscribed your channel. It is the most easy to understand Mamba lecture I've seen.

[@trevorhobenshield]

Very high quality, this is great. Hard to find good content like this. Thanks Umar!

[@purohitadey-bc9bg]

Understanding mamba couldn't be better than this !

[@周毅-b1h]

I'm very thankful for your explanation of this article, best wishes for you!

[@raaminakbari]

Thank you for this great and smooth explanation. I think the model you are showing at 36:14 is valid if matrix A ( and B also to send each input directly to the corresponding ssm) is diagonal. Now in this way each hidden state at different canonical direction ( or different element of the vector) is independent of each other. So if A is not diagonal then assuming an eigen decomposition exist, then we may say there exist an equivalent ssm which can be represented independent ( if we change the basis to eigen basis) .

[@mudassirkhan9054]

Thanks for explaining it in a way that anyone with some high school math background can understand, keep this up!

[@The_bioinformatician]

This is the best deep learning video I've ever seen. I will surely use some of your slides to teach my students

[@arvyzukai]

This is gold! I really appreciate attention to the details. Thank you Umar!

[@ankush4617]

Thanks for the amazing work as usual! Keep it up - this is probably one of the highest quality content on LLMs on youtube.

[@흰강아지-s4v]

this is just a pure art; thanks so much

[@Mirai12377]

very good video!!! thanks a lot for your efforts!!!!

[@fabiogomez8250]

Best MAMBA video at the moment!

[@optomosprime]

Excited for the video. I was searching for a video on Mamba and today I saw this. Your Transformer video helped me alot previously. Keep it up!

[@a123s1l]

Thanks for your clear explanation of MAMBA, coming from a control theory background, very much appreciate its usage in LLMs. One small error that I noted was that the A matrix must be N x N to translate the previous N-dimensional hidden states h(t-1) to h(t). I believe the A matrix is also time-varying to produce selective output tokens.

[@myfolder4561]

Thank you so much. Lots of useful details yet you curate through them at such a good tempo with easy to follow examples

[@selayan4985]

Such a briliant work you have done. Really learned a lot, thanks!!!

[@majidemami577]

Excellent video! Thank you. I have watched a few videos about mamba and this one was by far the best.

[@wayneqwele8847]

Thank you. I appreciate the approach you took in explaining the major concepts.

[@celestchowdhury2605]

Thank you so much for your detailed video and thoughtful thinking of you that we will need help with the equations! You are a savior!

[@mcHsyu]

Great explanation!! This is the first video that mekes me comprenhad the whole mamba paper.

[@TheFitsome]

some people are just born to teach.

[@shoubhikdasguptadg9911]

Ohhh Man, why did I discover this gem so late :( This guy is a rockstar!

[@nishanthshetty435]

Thanks a ton! Excellent explanation and great analogies to introduce the more advanced material. This is an absolute masterclass on how to teach advanced material.

[@beincheekym8]

Brilliant video! Really clear and with just the right amount of details!

[@GenAiWarrior]

Thank you so much for your efforts to make such an amazing video on Mamba architecture !!

[@prashlovessamosa]

Salute to consistency Thanks Umar sir.

[@akshikaakalanka]

This is really helpful for another talk I am doing on Mamba. Thank you very much for putting this out.

[@ActualCode0]

This is one of the best ML explanations I've seen even though I didn't understand all of it but I definitely learnt something new.

[@BooleanDisorder]

Even I understood much of this. I have no education. Thank you! Mamba looks really cool. Especially like the long context and further refinement. It looks like a model that could be made to learn as it goes. Plasticity potential

[@SpandanMishra-z4r]

OMG ! this is such as amazing description , you made my day

[@Erosis]

As others have mentioned, you have a keen ability to explain difficult topics succinctly and completely. Keep up the awesome work! I could of used this when I took a class on time-series modeling! Hah!

[@ankush4617]

Thanks!

[@danamics]

Great job on this video! I learned a lot

[@tunatuncer5639]

wow that's a great explanation , thanks for the efforts!

[@我我-p3z]

最清晰的讲解！

[@umuthalil5001]

Hi, I was wondering if you could explain 36:40 a bit more where you talk about multi head attention. From what I understand each head in multi-head attention each head looks at the whole input vector. Our key value and query matrices are all of size Dx(head_size) where D being dimension of embedding, so when we find key say we do key = X @ key_matrix where X is an CxD dimensional matrix, C is context len. This means each head looks at the whole dimension of the embedding D and represents it a head_size vector meaning that arrows going into each head should point at every single input dim.

[@luisrperaza]

I did learn a lot! Many thanks for making this video.

[@TheRohit901]

Amazing explanation. I love this video because it covers sufficient depth and explains each concept with proper examples. I've subscribed instantly, and look forward to more such videos on recent papers.

[@Hello-tx7ug]

Thanks!

[@kwanhowong5065]

Really an amazing video! You save me a lot of time! Thank you!

[@deepikagurung9410]

are going to code it as well. I really liked the video it was easy and very comprehensive.

[@belamipro7073]

Danke!

[@umarjamilai]

Thank you very very very much for your generous support! Let's connect on LinkedIn!

[@soroushmehraban]

Love it! Keep up the amazing work.

[@divgill6062]

Amazing! So detailed. Well done sir

[@m1k3b7]

Brilliant explanations. Thanks.

[@EkShunya]

i always eagerly wait for your explainer. they are 🤯. thank you :)

[@whisperlast6548]

This video is of great help!!Thank you very much.

[@bulat_15]

Thanks man! This helped me a lot

[@bryanbocao4906]

Thanks for the video! Very informative! Just to check: At @1:03:42, 3. be "... save back the result to HBM."?

[@pcwang7803]

Great lecture! It is easier for me to understand the work with your lecture. Can you give one for Reinforcement learning?

[@jason988081]

Dear Umar, referring to 53:50, recurrent SSM is indeed similar as prefix-sum (i.e., y=x_0+x_1+....x_N), but I the difference is that h_t=Ah_{t-1}+Bx_t, where h_{t_1} depends on h_{t-2}. I know how Blelloch parallel prefix scan works for calculating the sum of constants, but I do not know how parallel scan works for h_t=Ah_{t-1}+Bx_t. Could you please elaborate on it ? Thank you. @Umar

[@mahmoudreda5054]

thank you for this video , really helped me

[@alainrieger6905]

Awesome video as usual

[@allengeng6660]

Very nice talk, thank you.

[@walidmaly3]

One of the best! I have one question if we apply conv in S4 on sequence of length L, what will be size of conv layer?

[@akashkumar-jg4oj]

Great explanation!

[@amitshukla1495]

Absolutely amazing 🎉

[@eafadeev]

You're making very useful content, thank you!!! Maybe you could consider using larger text, so that one could read easily from a phone. Also a plus would be if the presentation were white on black (or bright color on black), it is less tiring to look at a dark screen for long periods of time.

[@팽도리-v6s]

Amazing video.

[@nguyenhuuuc2311]

Thanks for the awesome content! Hope the next one will be about DPO and coding it from scratch ❤

[@umarjamilai]

You're welcome: kzbin.info/www/bejne/nqeqkmiDl8ZnmZo

[@nguyenhuuuc2311]

@@umarjamilai Thank you!!! You're so talented at research and teaching!!!!

[@pawanpatil4715]

Hi Umar, amazing video. You are the best teacher. You are Karpathy 2.0. :) Please make a video on DPO :)

[@umarjamilai]

Done: kzbin.info/www/bejne/nqeqkmiDl8ZnmZo

[@pawanpatil4715]

@@umarjamilai thank you so much 😃

[@810602jay]

Thanks Umar! 🥰Very amazing learning material for Mamba!

[@杨辉-l2g]

excellent work! Thank you

[@GoogleColab003]

absolutely fantastic

[@albertmashy8590]

Amazing video

[@toxicbisht4344]

amazing explanation waiting for new video please upload soon

[@sayandas13]

Awesome explanation. Really appreciate such content. Can you please make a similar explanation video on the Mamba-2 paper?

[@samuelbeaussant3097]

Very good lecture ! Thank you very much for putting this for free on youtube :) I have question though, if my understanding of the HiPPO framework is correct, the A matrix is built to uniformly approximate the input signal (name HiPPO LegS in the paper). "Our novel scaled Legendre measure (LegS) assigns uniform weight to all history [0, t]". But however at 41:49 you explain that it is decaying exponentially similarly to HiPPO LagT. Do they opt for HiPPO LagT when moving to s4 and Mamba or am I missing something ?

[@rezagholipoor7900]

It was very informative

[@ShubhamAshokGandhi]

Great explanation. Very through. Loved it. I struggled with understanding the SSM paper. You explained all the bits beautifully

[@immakiku]

Trying to follow the rationale/last-slides - one advantage of SSM/RNN was that they would scale to infinite context. But Mamba reintroduced L-lengthed parameters. Why is this not limiting to this architecture the same way it limits Transformers? Qualitatively, it seems the only remaining advantage over transformers is the inference is cheaper - could you help clarify? Thanks

[@artaasadi9497]

Thanks a lot that was very useful!

[@edsonjr6972]

Excellent video! I'm looking forward if you do a coding one. Thank you so much for your work to the AI community

[@umarjamilai]

Coding one is not very interesting, because the most interesting part is the selective scan algorithm, which is a CUDA Kernel. The architecture is not so different from any other language model. Of course it would be super cool to code the CUDA kernel from scratch ;-)

[@НикитаБуров-ъ6р]

i've just started watching but guess this vid'll be much usefull

[@123456ewr]

Thanks, i hope you explain rwkv

[@abrahamsong6913]

this is so far the only video I found that described the math part in the mamba model. thanks a lot. One small issue. In 37:00, for the attention model, you mentioned each head takes only a portion of input dimensions, can you confirm this? I believe each head actually use all input dimensions.

[@abrahamsong6913]

It might be true for LLMs, but I believe this is not true for the original transformer model.

[@umarjamilai]

Hello! First of all thanks for the kind words. Yes, in multi-head attention, the idea is that each head sees the entire sequence, but a different portion of the embedding of each token. This is to make each head relate tokens in different ways. This mechanism is described in my previous video on the Transformer model.

[@buh357]

you are the best.

[@passarodavide]

Bellissimo video, grazie!

[@umarjamilai]

Grazie a te!

[@venkateshr6127]

Please can you make video on optimizers like adam,adagrad,...

[@undefined-mj6oi]

Hey! Thanks for the details in this video. I'm confused about the HiPPO matrix, which seems to be fixed given N? However the paper stated that delta, A, B, C are all trainable. What did I miss?

[@undefined-mj6oi]

is HiPPO the initialization of A?

[@umarjamilai]

Yeah, just the initialization

[@undefined-mj6oi]

Thanks for clarification. Could you please further explain how the parameter of A is (D, N) in S4? If I have D*SSMs, one for each embedding dimension, shouldn't A have DN^2 parameters?

[@dotori-hj]

Fantastic

[@Huawei_Jiang]

I have one question in terms of the example which you provided, 'the number of buddies'. I think the function should be like this : b(t)=5squ(3)^λt . please comment to me if I am wrong.

[@andreanegreanu8750]

Hi Professor! Very good explanation as always. However, I have huge difficulties to understand the dimensions of objects. Why the hell A matrix would be of (D,N) dimensions since it is used to project a vector h_t-1 of N dimensions into N dimensions? By the way, why is it written "Represents structured N x N matrix" ?????!!!!

[@SandeepS-i4e]

Great❤

[@HosseinKhosravipour]

very great

[@LukasSmith827]

you're extremely underrated, I don't think I'll be able to use much valuable info tbh.

[@kunchangli9319]

Brilliant! 太棒了！

[@umarjamilai]

谢谢你！

[@Huawei_Jiang]

Can we run Mamba via normal GPU?