xLSTM: The Sequel To The Legendary LSTM

[@bycloudAI]

Join the waitlist now for exclusive access to the OnDemand platform: on-demand.io/contact let me know if you like these kinds of research breakdown too! my newsletter: mail.bycloud.ai/

[@khla.mp4]

I'm glad we have someone to translate research language into memes and references for us

[@animalnt]

Just need the soundboard and temple run in the corner and we skibidi

[@Wagner-uv6yp]

@@animalnt Yeah like what is this guy's background lol he sounds like a college kid not a PhD person.

[@animalnt]

@@Wagner-uv6yp "not a PhD person" 💀

[@thebrownfrog]

Good thing I'm 15. Perfect channel for me

[@revimfadli4666]

Other than bycloud and fireship (and maybe 2 minute papers and yannic kilcher), is there another?

[@SmallLanguageModel]

I heard people in the RWKV and EleutherAI discord complain that they used wrong hyperparameters for some other architectures, while they used the most optimal hyperparameters for xLSTM. So the results are not entirely honest and they try to hype up their own architecture, but what else is new...

[@bycloudAI]

I was a tiny bit suspicious when xLSTM didn't publish codes, but damn okay

[@w花b]

When money ruins research... But hey, it's been a thing since research exists.

[@adamrak7560]

The context extension comparison is practically wrong too, and highly misleading. If you train transformer for a small context length without any extra extension tricks it will perform really badly for large context, by design. Comparing to that is simply not useful. It is simply misleading for most readers/viewers. There are context extension tricks you can use which avoid perplexity explosion for the transformers, those would chance the graph very significantly.

[@nnnik3595]

Also when you actually look at the graph at 3:54 in detail you can see how the presented combination of both [1, 1] is worse than either of the previously existing methods in almost all metrics.

[@DrW1ne]

Every video of yours is like my own Christmas! Keep it up.

[@user-cg7gd5pw5b]

So many memes per second, my brain can't process everything.

[@benjamineidam]

(> complex topic = > memes / s) = AWESOME!

[@bobspianosbffl]

This video was so good what the hell Thank you for delivering me this complex info in an easily consumable way Subscribed

[@ij9375]

I am a simple man, I see King Baldwin, I read AI, I click thumbnail😂

[@sanderbos4243]

Kingdom of Heaven, my love

[@whoami6107]

Really like the way you make these videos, i couldn't have understood these things otherwise

[@Meleeman011]

i have first hand exp with lstms and they are amazing. i can wait to try or implement this architect a solution with it

[@literailly]

Editing is amazing lol. Nice job, bycloud

[@KW-jj9uy]

I wonder if this is good for real time robotics, where you get fast real time data in tiny chunks, and you need a fast model with memory

[@OnDemandAI]

This OnDemand looks like quite the hoot!

[@Lokmanne]

Impressive, xLSTM might actually be worth switching from Mamba to it. Can you please make a sequel video to this comparing xLSTM, Mamba, Transformers, Jamba and Mamba-2? I am so desperate to know.

[@initialsjd5867]

I find this all extremely interesting, but am having a hard time finding the right way into understanding these topics, does anyone have a suggestion on where to start?

[@Detril2000]

Start by a Neural Networks introduction course. Besides that, currently your only options would be to then study this on your own or enter a Master's program in Computer Science, as all "courses" on LLMs are currently extremely dumbed down and mostly just go over how to type on ChatGPT.

[@NicitoStaAna]

3b1b is a great start. You're going to need to understand matrix multiplication (okay, enough. It's just sumproduct in excel) Statquest for a great overview/deepdive for any false assumptions

[@aykutakguen3498]

Interesting, thanks for the le vid

[@alkeryn1700]

all these cool architecture never actually used on open models.

[@pallharaldsson9015]

3:33 the first line there is slightly wrong, former sLSTM should be mLSTM apparently (no harm done, since 0), but more importantly the ratio could I guess be e.g. [2:3] similar to hybrid Mamba/transformer models mixing different number of Mamba layers and transformer/attention layers. It should be interesting to know the optimal [x:y] ratio, and even if it makes sens to mix also more e.g. Mamba into this... As seen at 3;::28 none of the ratios seem optimal. mLSTM has d x d matrix and the d could also be tuned also I suppose.

[@dominiksvestka1587]

So could using only sLSTM blocks be superior to using LSTM blocks for predicting time series?

[@Ludifant]

We have seen time and again, that modeling stuff on what we see in humans helps. Now we are venturing in to macroscopic structures like memory, where we actually have working theories of how people do it, which can give clues to smart architecture. GPT is brute forcing, LSTM is social engineering. It's not like any is guaranteed to get you in, but it is good to have both tools in the toolbelt or both expertises and viewpoints in your team. As the landscape keeps changing, the winds of fashion favour one or the other.. But when motion pictures came out, they didn't stop painting or sculpting. But when talkies came out, silent movies were left behind. We need to get past this obsession with 'better' and see that as long as they differ enough, these are all valid strategies and the choice of tool depends on the nature of problem. It helps if you understand your tools and your problem.

[@ccash3290]

Put some clouds in the background or something so people aren't confused by the similar thumbnails to other creators

[@gualcasas528]

that is the goal, for people to get confused and click on the video

[@jonatan01i]

outperforms mamba AND TRANSFORMERS?!?!?!

[@monad_tcp]

nothing outperforms Decepticons thou

[@AMA14700]

Well at this point companies now should stop training their AIs for some time in order to wit for the best architecture 😂

[@w花b]

Reminds me of web development where there's always a new best framework and everything becomes "obsolete" every 2 months

[@mfpears]

I wish I had time to research this stuff myself. This is exactly what I would be researching. Transformers seem limited in some fundamental way to me. I want to see something more recursive and dynamic. But these are impressions only since I haven't had money/time to really dive in yet.

[@Alpha_GameDev-wq5cc]

LSTMs were all the hype before the Transformers dropped in 2017… Love to see the prodigal son return

[@AntoshaPushkin]

"xLSTM" sounds so lame. Should have been "2 LS 2 TM" or maybe "LSTM: Tokyo Drift"

[@nathanpotter1334]

Back from 2017

[@tom-et-jerry]

Very nice music !

[@beyse101]

I believe I just got Schmiedhubered

[@mawungeteye6609]

Up next xlstm-mamba hybrid

[@olegpetrov2624]

Superior meme taste. Thanks bycloud.

[@cdkw2]

With enough copium we sure can 🔥🔥🔥

[@catoleg]

This amount of memes per second remind me of the Bad Gear channel

[@SamArmstrong-DrSammyD]

Hell yeah, sigmoid FTW!

[@TheGiovany82]

My brain went out of Ram with this one 😂😂😂😂

[@ZenchantLive]

Love your B Rolls lol

[@anshulsingh8326]

Ok, so how to get started with AI development

[@BoHorror]

Try and predict the stock market

[@mrrespected5948]

Nice

[@Kenneth_James]

What happened to the videos that showed examples of where the best AI generated images and video and gave a dummy like me an idea of what could be done with the current 'best'

[@JayDee-b5u]

A person really can't 'shill' their own thing. That's not the right word.

[@andyizawsome]

i know some of these words

[@renanmonteirobarbosa8129]

Hopfield NN is the real GOAT

[@cdkw2]

The newsletter is fire though bro, thanks for that

[@fmmmtmm]

What's an "architecture"?

[@drdca8263]

Structure of the network

[@crackwitz]

Hoch-rei-ter Hoooch-reeeiii-teeer

[@agenticmark]

funny. subbed

[@butterbee2384]

02:40 "beated" lmao

[@JhonSabowsla]

bro othe memes got me crying 💀😆

[@MartinDxt]

to whoever thought or is thinking the the ai revolution is dead this is just the tip of the iceberg :D

[@prenomnom2686]

Pleeeeeease talk about YOCO 😢

[@Ludifant]

Hoch-reiter means High Rider... Sooo he might be high on copium..

[@newbie8051]

Fucking hate Language Models, they have single handedly shitted on the entire community making everyone focus on chatbots. Gone are the days when ppl used to showcase their work on some image-data. Meh, ig I just don't like Language Models that much. transformers as an idea is really amazing, images is make intuitive for me, cannot grasp much stuff from "embeddings for words". I'll try xLSTMs as regressors, will definitely make a good projects, thanks for the video buddy 💖💖

[@BrianGarcia-t9n]

Rogahn Branch

[@8jhj345gg]

Fireship clone?

[@TwiceVisible]

Everyone copying the Fireship thumbnail style these days

[@Adventure1844]

The entire social media world is based solely on copy & paste style.

[@Amasglobulaires]

good video but the music is badly mixed with the music

[@Pepcen]

Hmmm I sure wonder where he got the idea for this thumbnail format

[@ucngominh3354]

hi

[@newyorthtimes4496]

the way I filter papers nowadays is through code. If they come with a github repo, I check. If not, just toss to the bin

[@jasonhemphill8525]

2nd

[@lulboiking5806]

Noice!👍

[@JosephJair97]

xd

[@JAD3N]

This guy, copying Fireship thumbnails 😂

[@ThomasButlergT6]

Johnson Susan White Amy Taylor Sharon

[@andiiacob1627]

That off key minecraft sounding music in the background is really distracting

[@ScottSummerill]

Not watchable with all the ridiculous movie clips. Leaving.

[@FaultyTwo]

Ah yes. LSTM: EXTREME EDITION! 🔥🔥🔥 HIDE YO KIDS, HIDE YO DATA. THIS RNN IS OUT THERE FOR BLOOD 🩸🩸

[@monstercameron]

xLSTM BYTE when?

[@xv0047]

too many gifs that make the video too distracting.

[@jonnylukejs]

wtf thats mine i made the block matrix lstm thats lame they made a paper and just claim it