MAMBA and State Space Models explained | SSM explained

[@drummatick]

I've a question. Given that SSMs are entirely linear, how do they conform with universal approximation theorem? I mean a lack of non-linear activation should imply that they should be particularly bad at approximate functions, but they are not. Am i missing something? Also really loved the video!

[@AICoffeeBreak]

Exactly, SSMs are linear. 2:05 This is why they are part of larger neural nets, that bring in the nonlinearities needed to fulfil the universal approximation theorem. You can think of SSMs like of attention in Transformers: they are just a module (the module that aggregate information from the whole sequence, such that the current token is informed of its context). Great question, thanks!

[@drummatick]

Thank you! One last question. Since the output is realized rather through a convolution than a recurrent function, do we use Backprop through time for updating A B and C? Or just a normal backprop

[@AICoffeeBreak]

​@@drummatick You ask so many great questions! (I'll pin your original comment, as I think others might have the same question) As the state of token t depends on the state of token t-1, we still need to do backpropagation through time (otherwise we would not know what to backpropagate at t-1 if we did not resolve t first). But because of the linearity, backprop through time is reportedly more stable for Mamba than for e.g., LSTMs.

[@drummatick]

@@AICoffeeBreak thank you! Makes sense. That might also explain why mamba doesn’t suffer from Vanishing gradient problem as the gradients propagated vanish linearly? Correct me if I’m wrong

[@AICoffeeBreak]

@@drummatick Yes, no activation function to squash signals to zero.

[@partywen]

Thanks! Looking forward to a Hyena video :)

[@AICoffeeBreak]

Thank you so much! It seems like you paid for the coffee I'm drinking right now!

[@peabrane8067]

Thank you for the shoutout to me repo! I later realized it was an application of a known idea "heisen sequence", which is a pretty cool way to do certain associative scan operations via cumsum

[@theruisu21]

great video! thanks

[@AICoffeeBreak]

my pleasure

[@OlgaIvina]

Thank you very much for this thorough, well-curated, and comprehensive review of MAMBA.

[@AICoffeeBreak]

Thank you, for your appreciation! I just saw you on LinkedIn, let's stay connected!

[@ShadowHarborer]

I have to give a presentation on Mamba next week and I've been waiting for this video so I could finally learn what the hell I need to talk about

[@projectsspecial9224]

Don't feel dumb. The first time I heard about transformers used for LLMs, I thought it was a new movie prequel. Oh, well, I guess I'm dumber. 😂

[@sampadk04]

Us

[@alex-beamslightchanal8743]

Thanks!

[@AICoffeeBreak]

Omg, thanks a lot!

[@Thomas-gk42]

I understand 10% of that stuff, but the presentation is lovely.

[@AICoffeeBreak]

Thank you so much!

[@DerPylz]

I was waiting for exactly this topic! Thanks so much!

[@outliier]

Really cool video! Thank youuuu

[@AICoffeeBreak]

Thanks, nice to see you here, Dominik, aahm I mean Outlier!

[@cosmic_reef_17]

Hats off to you for this amazing video! Best explanation of Mamba I have seen.

[@AICoffeeBreak]

[@jamescunningham8092]

This is exactly the level of detail I needed right now. Thank you so much!

[@AICoffeeBreak]

Thanks! This was also the level I was looking for when trying to understand Mamba. So I decided to explain what I understood.

[@李洛克-m1u]

Thank you! This is by far the easiest-to-understand and most concise video that teaches the concepts of SSMs

[@AICoffeeBreak]

Thank You for the visit!

[@harumambaru]

Nice T-shirt! So excited to listen about new models!

[@MaJetiGizzle]

Thanks for the MAMBA video! I always appreciate your insight on these new, influential papers! Your thoughts always pair well with a good cup of coffee. 😁☕️

[@AICoffeeBreak]

Thank you so much!

[@ruchiradhar1589]

A big thanks for a comprehensive explanation of the Mamba Architecture & computations, @AICoffeeBreak!

[@AICoffeeBreak]

Thank you for your visit and for leaving this wonderful comment!

[@rodrigomeireles5966]

@AICoffeBreak, thank you for the awesome video. Very small pet peeve which had me re-check all the math. At 11:20 it would make the explanation much easier to understand if you kept x 0 indexed as that is the notation you were going for since the beginning. Also, maybe making it explicit that you're taking t = L, although this is kind of obvious. This was an awesome lecture, thank you again.

[@Emresessa]

Awesome video! I especially like the simple explanation and the visuals.

[@AICoffeeBreak]

That was our aim, thanks!

[@kumarivin3]

Thank you so much !! you really super simplified it for any beginner level deep learner to understand

[@hannes7218]

this explanation was excellent. Thank you very much :)

[@faysoufox]

Nice video, good overview, which is what I was searching for

[@AICoffeeBreak]

Glad you like it!

[@markr9640]

OK, Miss Coffee bean, you really made my head spin this time! Keep up the good work. We love you!

[@AICoffeeBreak]

[@고준성-m4g]

Great. There are a lot of failed explanation or completely wrong approach about SSM and Mamba on the internet, but finally I found the exact what I want. Thank you for the video.

[@AICoffeeBreak]

Thank you!

[@bareMinimumExample]

Does matrix A change during inference or only during training?

[@AICoffeeBreak]

Only during training. :)

[@doublesami]

Looking forward to see a video of vision mamba to know the points I missed while reviewing other content related to it

[@Ben_D.]

The ASMR at the end is nice. I always sit and wait for it. 🙂

[@AICoffeeBreak]

😂I should start a channel reading ML papers in ASMR style.

[@BooleanDisorder]

I can't wait for neural networks that are tokenless and takes bits directly. Where words are learned through images and sound rather than their own modality. And richer input with simultaneous multimodal vectors.

[@AICoffeeBreak]

Did you see MambaByte? 20:45 But exactly, I agree with you. This would be tremendous for both multilinguality and multimodality. Tokenizers are very tedious.

[@AM-yk5yd]

Great video. Speaking of VMamba. For some reasons it seems people in medical images field are more excited for mamba than in LLM. So many, they are competing between themselves. "Swin-UMamba demonstrates superior performance with a large margin compared to CNNs, ViTs, and latest Mamba-based models. Notably, on AbdomenMRI, Encoscopy, and Microscopy datasets, Swin-UMamba outperforms its closest counterpart U-Mamba by an average score of 3.58%". Like if you pick a random mamba paper from arxiv (there are dozens of them already), it probably will be related to medical image segmentation.

[@AICoffeeBreak]

Yes, this was exactly my impression when I talked to some friends working on medical imaging yesterday! 🤗 Unfortunately, I couldn't mention this insight in the video as it was already almost done with editing.

[@AaronNicholsonAI]

Thank you so much! Super helpful.

[@AICoffeeBreak]

[@jefferychen8330]

Thanks for the video! I’ve been confused between RNN and MAMBA for several days😂

[@AICoffeeBreak]

Happy to help!

[@serta5727]

Thank you for the great Mamba explanation

[@AICoffeeBreak]

Thank You for your visit and for leaving this comment!

[@darkswordsmith]

I'm not entirely sure on how SSM differ from RNNs, especially regarding how attention is being used. Theres still the bottleneck of h_t to h_{t+1} between time steps, which was one of the motivations for the attention layer -- so that information in one part of the sequence doesn't have to be squeezed before computation with information from another part of the sequence. Is the main innovation from RNN to SSM the fixed delta, A,B,C formulation such that the training can be done in parallel for all time steps?

[@AICoffeeBreak]

SSMs are linear RNNs. But linear is not powerful enough (universal approx. theorem). 2:05 This is why they are part of larger neural nets, that bring in the nonlinearities needed to fulfil the universal approximation theorem. You can think of SSMs like of attention in Transformers: they are just a module (the module that aggregate information from the whole sequence, such that the current token is informed of its context). But unlike attention, this information aggregation module works token after token, while attention has access to all tokens at once. So once a token is wrongly forgotten, it is gone. Therefore, the state of research for now cannot really leave attention alone, because it is so useful for retrieval and finding needles in a haystack: arxiv.org/abs/2402.04248 or arxiv.org/abs/2402.19427

[@AICoffeeBreak]

Thanks for the amazing question. And yes, coming to the last question: SSMs are linear RNNs which makes them computable in parallel at training time, exactly.

[@darkswordsmith]

Thanks for the great explanation! I suppose things will become more clear when I try to work through the math myself. When I first read about SSM, I saw a big connection between it and kalman filtering. But now I see the connection is probably just inspirational

[@davidgraey]

this was such a well summarized explanation, thank you! ❤

[@AICoffeeBreak]

Glad you like it!!

[@norabelrose198]

nitpick: with FlashAttention, transformers have linear memory requirements, but the compute complexity is still quadratic

[@JuliusSmith]

Excellent summary! Note BTW that the B discretization has been simplified to B * \Delta, because it made little difference (see mamba-minimal comments).

[@doctor_xin1452]

cool video!!!! Thanks a lot!!!

[@elinetshaaf75]

omg, omg, omg, this is such a great explanation of mamba!

[@HeliyaHasani-x7b]

Amazing explanation! Please create more content!

[@harumambaru]

My favourite kind of animation of Ms Coffee Bean is on 19:47

[@AICoffeeBreak]

🥂

[@DerPylz]

[@TimScarfe]

Awesome explanation Letitia!! ☕️

[@AICoffeeBreak]

Thank you, Tim!

[@dunamaiTheSheep]

Incredible work. Thank you !

[@AICoffeeBreak]

I hope I'll remember everything, so I can present this paper tomorrow in the reading group. 😅

[@Micetticat]

I knew you would have covered this hot topic! I bet now you are working on RWKV ! They are discovering so many alternative architectures these days!

[@AICoffeeBreak]

am I? 😆

[@juanmanuelcirotorres6155]

thanks a lot for this (love your sweater)

[@AICoffeeBreak]

Thank You for your kind words!

[@Theodorus5]

Yea that was good

[@AICoffeeBreak]

[@nerisozen9029]

thanks! also very cute merch!

[@AICoffeeBreak]

thanks!

[@LorenzoCassano-l2h]

I did not understand why do we need to perform a discretization step? Which are the advantages ? I understood what happens from math point of view, But why we need to change from Real numbers to discrete numbers ? Is it something related to Backpropagation ?

[@AICoffeeBreak]

Great question, and I do not know exactly the answer myself. I assume you could do without it and hope for SGD to find the right parameter settings. But I assume the theoretical soundness helps to stabilize training. I would be curious to find out more about this.

[@v-ba]

​@@AICoffeeBreakisn't it because if we have continuous function then we have to solve ODEs with proper solvers, but if we discretize it and apply the Euler method then we get recurrence that can be optimized with a backprop?

[@AICoffeeBreak]

Exactly. That's a great way to put it with the right background knowledge.

[@hannesstark5024]

Neat summary! Thank you ^^

[@AICoffeeBreak]

Thanks Hannes!

[@samanthaqiu3416]

only way I managed to understand up to 40% about how mamba is implemented was watching this video

[@AICoffeeBreak]

Give these things some time to crystallize. Your understanding will grow even bigger with time. Also, better explainer resources will keep coming.

[@kahoonalagona7123]

Amazing !!

[@AICoffeeBreak]

[@marzi869]

Wasn't your voice enough? now I have a crush on you!!!

[@jiahao2709]

Thank you a lot for your explain! may i know which software you using for making animation in your video? thanks!

[@AICoffeeBreak]

My editor uses Adobe Premiere Pro for video editing (this is also when Ms. Coffee Bean comes in). I use PowerPoint for all other visualisations.

[@jiahao2709]

@@AICoffeeBreak really beautiful!

[@commonwombat-h6r]

i barely grasped the transformers and now they have this new thing.....

[@tmjz7327]

well you had 6 years to look into transformers, so..

[@projectsspecial9224]

Don't feel bad. NLP and Linguistic Models had been around for decades. It really takes a long time because processing power was so expensive. I did NLP AI demos in year 2000 and they told me that the tech was too advanced for its time and they didn't know what to do with it! 😅

[@krttd]

I wish I could find a TensorFlow implementation with all the optimizations! I need to bite the bullet and switch to Torch soon.

[@AICoffeeBreak]

Good luck with switching your faith.

[@ultrasound1459]

Move asap bro. Pytorch > TensorFlow

[@TheRyulord]

The pytorch version has a bunch of specialized CUDA kernels so I hope you know CUDA too

[@arthurpenndragon6434]

I am in general happy that we are trying to leave transformers behind. RWKV and SSMs look promising. Transformers are inelegant, and by their very formulation a resignation from looking for better methods.

[@Summersault666]

Can tou share some info about the material you used to learn mamba?

[@AICoffeeBreak]

Sure! You can find most of these things in the video description. 1. Foremost, the paper and its references 2. 📕The Annotated S4: srush.github.io/annotated-s4/ I also found this after publishing the video: jameschen.io/jekyll/update/2024/02/12/mamba.html Yannic Kilcher has already done great video about Mamba and I wholeheartedly recommend it! My only problem with it, is that he sticks too much to the paper and the Mamba paper (while being well written), is scattering the architecture all over the paper. It also assumes prior knowledge about previous papers, such that I honestly did not understand how it all hangs together and what Mamba actually is and how tokens flow through it.

[@Summersault666]

@@AICoffeeBreak wow, those are great. Thanks!

[@AICoffeeBreak]

Celebrating our merch launch, here is a limited time offer! 👉 Get 25% discount on AI Coffee Break Merch with the code MAMBABEAN: 🛍 aicoffeebreak.creator-spring.com/

[@harumambaru]

Isn't it cannibalism when Coffee Bean drinks coffee?

[@AICoffeeBreak]

Have you checked the cup? 🤔 It's hot water. Beans drinking hot water is how coffee is made! ☕

[@kamiboy]

Do I understand correctly that in MAMBA to predict the current token the architecture only has access to an embedding influenced only by all previous down stream tokens, but with no influence from future upstream tokens? This is kind of a weakness, isn't it? I think the way LLMs are constructed their classification tasks can take into account all tokens in the sequence for predicting any token in a sequence.

[@AICoffeeBreak]

Yes, exactly. If the summary / history token misses something, it is gone forever. The hope is to learn to retain everything (important).

[@kamiboy]

@@AICoffeeBreak No, what I actually meant was something else. Let me explain it like this. Let us say the input sequence is "I LIKE TO [CLS] A BURGER", and I want to predict which token most likely fits into the [CLS] token. In an LLM the classification of [CLS] would be able to take into account all token previous to [CLS], "I LIKE TO", as well as all tokens after it "A BURGER". But I think from what I understand MAMA would only use "I LIKE TO" to predict, because the embeddings used to predict the current token only depends on what came before it in the sequence, am I understanding this right? This is certainly the case for the SSM's, but I am not sure with the selecting SSMs of MAMA.

[@AICoffeeBreak]

@kamiboy Ah, I understand, you mean bidirectional models like BERT that can classify things in the middle of the sequence. Yes, encoder models like BERT can look into the future. GPT models don't, because their attention is causally masked. There, the art ist to frame / prompt the problem such that the answer is at the end of the sequence. This is quite simple, and works for MAMBA too: "I like to eat a [CLS] burger. What word does [CLS] stand for? The answer is:" But there is no limitation onto MAMBA, as one can make it bidirectional as well, as Bidirectional LSTMs used to be: run the model once forwards to summarise the sequence to the left of CLS. Then run once again from the right until the CLS. Concatenate the forward and backwards summaries (token embeddings) and classify the CLS token based on that.

[@kamiboy]

@@AICoffeeBreak excellent, thanks for the clarification. I was already thinking that if I was correct then doing the bidirectional trick would be a solution. So that is how BERT works, neat to know.

[@edhofiko7624]

so whats next? kalman filter with learned dynamic?

[@MarcCasalsiSalvador]

min 11:30 x = (x_0,...x_{L-1}) Amazing video

[@RalphDratman]

Good job as far as possible, but I fear this topic may require more like a semester course than a single video. As a side note, I find it striking that mathematics is at the heart of the new (and miraculous) world of AI. If this rate of math penetration continues, kids will soon need to go directly from third grade to graduate school.

[@qichaoying4478]

how can i buy your tshirt?

[@AICoffeeBreak]

On the AICoffeeBreak merch store. 🤭 aicoffeebreak.creator-spring.com/

[@qichaoying4478]

@@AICoffeeBreak Oh I see. At the time I asked I hadn't watched to the end. 🤣

[@mkamp]

Awesome video. So for NLP we only have discrete inputs, hence we don’t need delta, right? And then we also cannot learn different deltas for different layers to be resolution independent as we could for continuous signals? No?

[@AICoffeeBreak]

Thank you! About your question: We always need the delta for discrete outputs. The delta-less formulation is for writing the differential equation which assume continuous transitions with infinitesimal steps. When we go to matrices and vectors and we are doing numerical approximations (all of deep learning), we need to discretize and choose delta steps.

[@mkamp]

@@AICoffeeBreak thanks for your answer and your patience Ms Coffee Bean. What I am stuck with is, that we use delta to discretize continuous inputs, but in the NLP applications of Mamba we already have discrete inputs, the tokens. What would we then apply delta too? Also, it would be conceivable that we could learn to fuse together for example every three consecutive tokens using delta, but then delta would be discrete (1, 2, 3, 4 tokens per window) and could not be differentiated/learned?

[@AICoffeeBreak]

@@mkamp Thanks for the follow-up, now I better understand the confusion. It all has to do with terminology and what word mean in different ML contexts. I think you talk about **the inputs**, whether they are discrete/continuous. Language is discrete and symbolic, but with our neural networks, we pretend like it would be continuous : we map words into tokens and tokens into vectors that can take continuous values. What I am talking about with SSMs is about the continuous / discrete nature of **the model**, in the way it updates the hidden state h. When we model sequences and want to compute hidden states for tokens in sequences, we are jumping in large (discrete) steps from token to token (word to word, as in "The big fox sat on the mat ..."). * In contrast, a differential equation is continuous: how h changes in time (h dot at 06:13) is an infinitesimal small step from the last state of h. So we are modelling the example sinusoid curve with the smallest step imaginable (the assumption of calculus that steps can be infinitely small - think of the integral vs. the discrete sum). I hope you can see that we cannot jump continuously from "The" to "big", as there is not much semantics in between those words (there are some semantic steps in between, but not infinitely many small steps. I mean, there is "cat", there is "woman", we have a "catwoman", a "woman cat" but not much else). So we cannot use differential equations to model jumps from tokens to tokens, so we use SSMs instead: 👇 * The SSM formulation is the discrete version of the continuous equation: we are modelling that sinus curve with larger steps (not infinitesimal steps), given by delta. Which means that we are jumping from token to token in our language / sequence applications. One h is "The", the next h has to model "big", the next "fox" ... So we are jumping in large, discrete steps from word to word (token to token). This is why we cannot use differential equations, but discretized versions of them: SSMs.

[@mkamp]

Thanks for the answer and for taking the time!@@AICoffeeBreak

[@AICoffeeBreak]

@@mkamp Videos are these static, uneditable things (in YT). I could have made this point much clearer in the video if I would have known about your question beforehand. And of course, it is all also limited by how much time I can invest into doing one video.

[@啊瑤]

解释得很好！爱来自中国💓

[@rock_sheep4241]

Looks more like a RNN model

[@AICoffeeBreak]

It is a linear RNN with a lot of Schnickschnack around it. 😅

[@pooroldnostradamus]

⁠@@AICoffeeBreakto use the scientific terminology

[@AICoffeeBreak]

@@pooroldnostradamus 🤣🤣🤣

[@andreicozma6026]

14:06 so they reinvented memoization and gave it a different fancy name?

[@science_electronique]

mamba for video

[@davidespinosa1910]

Transformer memory space is linear, not quadratic -- see the FlashAttention paper.

[@AICoffeeBreak]

True with Flashattention! Yes.

[@garyhuntress6871]

The explanation of the SSM sure sounds similar to a Kalman filter to me!

[@RFantiniCosta]

You are absolutely right, sir. That's because Kalman Filter is indeed a State Space Model

[@RFantiniCosta]

You are absolutely right, sir. That's because Kalman Filter is indeed a State Space Model

[@abdulhamidmerii5538]

Am I the only one that finds mamba much harder to grasp than transformers?

[@AICoffeeBreak]

Let's see: what was the first architecture you learned? Was it transformers or LSTMs?

[@MyrLin8]

merch lol

[@keeperofthelight9681]

Mamba jamba samma

[@AICoffeeBreak]

This spawns music in my head. 🎶😅

[@nobo6687]

Hey Ai Coffe Break create video on cited papers from Gemini 1.5 Pro paper 😮❤

[@csepartha]

Kindly make a tutorial to fine tune an open source LLM model on many pdfs data. The fine tuned LLM must be able to answer the questions from the pdfs accurately.

[@skullteria]

its kinda obv that you dont rlly understand this shit yet good enough to explain it better. Its like trying to reverse analyzing a circuit board instead of going from the intended use case, to the technical rules to the abstraction to the design to the analogies. You just explained me what a mosfet is for and how it works inside of a circuit board... yeah well nice. This isnt a test. Its a lesson. Try to explain it in human language, feel into it. THEN you can down to why all that stuff achieves that and which analogies could be used for it. This video right now just shows one thing... you are smart. But it doesnt show me that you understood teaching.

[@AICoffeeBreak]

Hi skullteria, with my videos I always have to make the difficult decision of what will be the target audience of the video. I have a general guideline that I set myself when I started this channel, which was to broadly aim the videos at computer science students somewhere in their bachelors. With a topic like mamba, I had to gloss over many complicated topics (for some of which I have already made some more in-depth explanations) to not make the video over an hour and still get into the level of detail that many of my regular viewers expect. You will find in the comments many people for whom this was exactly the right balance. But this of course means that there will also be people who will not find the video super useful, and I'm very sorry that you're one of them. If there's a specific topic for which you'd like a more high-level, more intuitive explanation, absolutely put it in the comments and I'll add it to my to-do list :)

[@গারমারাগেলো]

আমার কী পছোনদো তোকে বোলবো কেনোরে

[@paratracker]

I'd love to see your take on Hasani, Lechner, et al's Closed-form Continuous Time Neural Nets, Liquid Structural State-Space Models, Forward Invariance of Neural ODEs, ... and maybe compare with MAMBA SSSM