LLM Attention That Expands At Inference? Test Time Training Explained

Take your personal data back with Incogni! Use code bycloud at the link below and get 60% off an annual plan: incogni.com/by...
RNN's hidden states be like: "You know, I am something of an ML model myself"
check out my newsletter:
mail.bycloud.ai/
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
[Paper] arxiv.org/abs/...
[Code PyTorch] github.com/tes...
[Code JAX] github.com/tes...
This video is supported by the kind Patrons & KZbin Members:
🙏Andrew Lescelius, alex j, Chris LeDoux, Alex Maurice, Miguilim, Deagan, FiFaŁ, Robert Zawiasa, Daddy Wen, Tony Jimenez, Panther Modern, Jake Disco, Demilson Quintao, Shuhong Chen, Hongbo Men, happi nyuu nyaa, Carol Lo, Mose Sakashita, Miguel, Bandera, Gennaro Schiano, gunwoo, Ravid Freedman, Mert Seftali, Mrityunjay, Richárd Nagyfi, Timo Steiner, Henrik G Sundt, projectAnthony, Brigham Hall, Kyle Hudson, Kalila, Jef Come, Jvari Williams, Tien Tien, BIll Mangrum, owned, Janne Kytölä, SO, Richárd Nagyfi, Hector, Drexon, Claxvii 177th, Inferencer, Michael Brenner, Akkusativ, Oleg Wock, FantomBloth
[Discord] / discord
[Twitter] / bycloudai
[Patreon] / bycloud
[Music] Massobeats - Noon
[Profile & Banner Art] / pygm7
[Video Editor] Silas

[@bycloudAI]

Take your personal data back with Incogni! Use code bycloud at the link below and get 60% off an annual plan: incogni.com/bycloud maybe we are all bots and the dead internet theory is true

[@mine.moment]

Please create your own style of thumbnails and stop trying to mimick Fireship lol... I'm being honest rn but the contents in your videos don't feel as interesting/ funny/ easy-to-understand as his. Hope you take that as constructive criticism because you do cover lots of cool topics that Fireship doesn't.

[@Miaumiau3333]

I disagree with the comments complaining that the video is too technical. I really like that you provide enough detail to roughly understand the technique, awesome video!

[@FireOfGott]

Agreed, this is very approachable to someone who knows some architecture fundamentals!

[@seriousbusiness2293]

I found the pacing a bit off. In general very well eddited and summarized information. But its hard to keep track with all the vocabulary, personally id ether need to linger longer on these details or get an even shorter overview on those aspects. I really like the style of Yannic Kilcher Paper reviews but his videos are also 3 times as long, so in any case its a tradeoff what one prefers.

[@xClairy]

​​​​@@seriousbusiness2293Honestly, I feel like it's because his target audience was different, and now it's more technical, so he'd need more time to explain those concepts instead of expecting a baseline understanding. But going more in detail would scale logarithmically with video length, which would also hurt his YT channel, considering we all expect at best 5~15 minute videos from this channel. So, yea, it's a trade-off.

[@w花b]

Yeah they might as well just watch Fireship because that's what they're asking.

[@mine.moment]

@@w花b But the problem is that bycloud tries to mimick Fireship's thumbnail style to lure in Fireship viewers then throw them off with 10+ minutes videos of overly technical stuffs, who prefer ~5 mins of mixed interesting, meme-y, simplified contents instead.

[@MrJaggy123]

Turn all the hidden states into ML models? That scream of pain you all just heard was from the interpretability researchers ;)

[@QuantumConundrum]

OK, but then their employment is secured forever LOL

[@CurioCity-Curiosity]

We need black boxes for the black boxes!

[@koktszfung]

But imagine if those ML models are CNNs and you can see how the kernel adapt to the context of the input in real time, wouldn't that be actually easier to interpret?

[@naumbtothepaine0]

@@koktszfung CNNs are more like DL, ML models are simpler

[@revimfadli4666]

​@@naumbtothepaine0which simpler ML models? XGBoost? SVM? Because CNNs are ML models too

[@Eianex]

In conclusion, Trouble in Terrorist Town is cooler than some transformers and some snakes.

[@manuelburghartz5263]

This channel explaining AI and using anime references in the visuals is exactly what I needed. Great video!

[@heavenrvne888]

holy shit this method is so interesting. and the way they encapsulated the entire idea into the title LOL!

[@karlkastor]

I love that you tell us how the method in the paper roughly works. A lot of KZbin channels just say this new technique is better without any explanation and just show results, so I have to skim the paper to get the gist of it.

[@krollo8953]

Yup makes you feel like you're learning something rather than information without enough context

[@FunBotan]

I would never suspect that this video would help me write my PhD, but the "compression heuristic" is exactly the term I needed but didn't know to express my idea. Thank you!

[@papakamirneron2514]

Please make a video explaining all of these terms, apart from that, keep the technical videos coming!

[@flamakespark]

Another day, another attempt to re-invent LSTMs

[@babyjvadakkan5300]

Whats that now?

[@zyansheep]

​@@babyjvadakkan5300type of rnn that google used to use (or still does?) for language translation before we got transformers

[@Bencurlis]

It is more of a generalization of both LSTMs and Attention, it is theoretically much more powerful IMO

[@keypey8256]

It's definitely an interesting idea

[@mikairu2944]

"too technical for this video" man you lost me at the thumbnail

[@cdkw2]

me to bro yet I still watch he entire video 💀

[@Dannydrinkbottom]

My brother speaking greek

[@OumarDicko-c5i]

As an IA, thanks you for teaching me this, i will use it to train myself

[@ginqus]

intelligently artificial

[@IN-pr3lw]

​@@ginqusinteligencia artificial

[@truongao5425]

intelligent anti-africa

[@TheRealUsername]

​@@truongao5425😂 troll

[@divandrey-u3q]

As always, thank you for the video! I really appreciate the amount of technical details here. Don't know why other people complain but I love it!

[@FaultyTwo]

"Mom! They are adding more weights to the models again!"

[@OperationDarkside]

Let's put transformers into transformers. Maybe we end up with baby transformers.

[@revimfadli4666]

Ah yes, hot transformers in transformers action

[@TheNewton]

Good short dense overview of an even super denser subject matter. Still waiting for the paper that modularizes all these component processes and flows then runs training against all the permutations to bootstrap itself.

[@DarrenReidAu]

It’s trainable models all the way down! Great video, thanks!

[@cdkw2]

2:32 Waiting for bycloud to be on that page like others!

[@athul_c1375]

It's some mamba jamba

[@marshallodom1388]

I got up to 6 minutes and loved the ride! Gonna have a snack and p and dive right back in!

[@QuantumConundrum]

More videos like this, please.

[@heavenrvne888]

that intro was amazing

[@ismailnejjar]

Love the video!!

[@XenoCrimson-uv8uz]

How do we know the ones complaining about the bots in youtube chat aren't bots themselves?

[@Alice_Fumo]

I have definitely seen bots complain about bots before. In fact, you could also be a bot. Who knows at this point?

[@picmotion442]

I might be a bot

[@leftybot7846]

I'm definitley not a bot, what a stupid idea.

[@turgor127]

Ban both then. Spamming is annoying ether way.

[@Cloudruler_]

The interesting thing is it's probably cheaper for a bot to spam "bot" than create LLM comments.

[@HarperChisari]

TTT is literally short term memory. Wild.

[@SimGunther]

Audience: Less reading, more technical content! Also audience: AAAAAAHH, MY EYES! TOO TECHNICAL FOR MY EYES AND EARS! 😢

[@sh4ny1]

4:11 Why not use wavelet transform for this? I think it would be useful here since

[@fnytnqsladcgqlefzcqxlzlcgj9220]

Perfect amount of complexity, please do not make your longer videos like this more simple, im not involved in any form of computer science but ive kept up with ai since tensor flow was brand new and i understood almost everything first try

[@JorgetePanete]

6:08 it resembles Trouble in Terrorist Town

[@spoonikle]

Earth shattering.

[@guilhermecastro3671]

Cool video, for a beginner all these terms together seem very technical, can someone suggest a playlist to learn more in depth about these topics ?

[@jasonhemphill8525]

What part are you struggling with?

[@StefanReich]

Super well explained. And full of memes

[@Ryu-ix8qs]

Good video, thanks

[@registered_dodo]

I love words.

[@fra4897]

great video but transformers in practices do not have quadratic complexity, only if u implement it in the vanilla way

[@jondo7680]

I want a TTT-Linear (T) with TTT-MLP (M) as it's inner loop.

[@quocanhnguyen7275]

I tried to read this when u wrote about in your newsletter. But it was not an easy paper

[@samarthpatel8377]

Sooooo many bot comments!

[@bolon667]

Putting innocent comments to change it into ads later

[@samarthpatel8377]

@@bolon667 I think you are right. The comments which I noticed earlier have gone?

[@dhillaz]

I know some of these words!

[@flinkstiff]

Bumblebee is my favorite

[@koktszfung]

Wouldn't this model be slow in operation if it has to train on the context?

[@Vagabundo96]

This is crazy

[@David-lp3qy]

MAMBA IF YOU CAN HEAR ME PLEASE SAVE US

[@bloomp7999]

did i understand this

[@CraftMine1000]

Training on test data,,, unless I severely miss-understand this I'm just going say; "jikes, nope, get out, and don't come back"

[@noctarin1516]

Nahh they actually cooking with this architecture though

[@narpwa]

my brain is exploding send help

[@BooleanDisorder]

Next up is cisformers

[@amafuji]

detransformers

[@ginqus]

biformers

[@anywallsocket]

formers

[@BooleanDisorder]

@@anywallsocket forms

[@sarveshpadav2881]

performers?

[@krollo8953]

Lol thats an intense amount of memeage

[@LukasNitzsche]

Does this relate in anyway to liquid time constant neural networks?

[@simonesborrinpz]

good videos👍

[@scientificaly_restful_one]

Well, some year ago or so I had thoughts about going into ML, but you have lost me on this one. 👍 I guess it's only gonna get more complicated from now on.

[@kamilbxl6]

Nowadays its easier to learn ML than ever. You should start with something simple enough that you understand around 80% and only actually doing 20% as smth new. There are lots of free shared classes like MIT, Stanford etc.. lots of tutorials, examples, code documentation. First get a general yet simple view bout NN, then chose what you'd like to specialize: image recognition, text or smth else

[@ricardocosta9336]

Dude no kidding, I came up with something similar a month ago. In concept. I'm afraid I have a limited num of insights in my life time. And without timento persue them I will never make any diference in the world. 😢. But hey that also proves, to me at least, that my math intuition is on point. 😅

[@ccash3290]

A lot of people have zero insights. Its important to work on your ideas to test them in reality

[@anywallsocket]

If you thought of it other people thought of it or will, so don’t worry about not being the one who gets credit, what matters is that the idea is in the memosphere

[@someonetrustme161]

so nobody gonna talk about how we just got rickrolled? at 3:43

[@Acceleratedpayloads]

This looks block recurrent transformers by DL Hutchens

[@Dom-zy1qy]

Whenever a new architecture takes over, the tech companies heavily invested into developing hardware specifically optimized for the transformer architecture are gonna be sad.

[@DanielJoyce]

A single brain neuron needs something like 5 layers or so to encode its behavior. So this kinda maps each node now to somethibg like a neuron. I know biological features map poorly to neural nets but neurons in the brain change how and when they fire as the brain learns.

[@bobsoup2319]

Bro this model is too complicated to be simplified more. Keep up the complexity it’s what makes it interestijg

[@anywallsocket]

Wouldn’t that take forever to train??

[@Wobbothe3rd]

The human brain is a recurrent neural network, not a transformer. Eventually, recurrent will win.

[@athul_c1375]

But who said the human brain is better than the transformer

[@-mwolf]

tell me the current paradigm is hitting a dead end without telling me

[@ONDANOTA]

why is every llm's OUTPUT context window fixed to 4096?

[@geli95us]

AFAIK, output context windows are not a thing for the models themselves, the model is just called once for every token it has to generate, you can perform that process a million times if you want, however, it's not useful if the LLM outputs text up to a point where its prompt gets out of its context window, so in the early days the "output window" was just set to whatever the model's context window was, nowadays, it's probably capped for economic reasons, LLMs get more expensive the longer the input is, so by limiting the output window, they force you to pay for tokens several times, once as the model's output, and subsequent times as input to the next outputs

[@spoonikle]

To stop it. While still giving enough space to make “satisfying” answers.

[@4.0.4]

It seems very convoluted, but I guess it should learn with less data? That could be good for startups that don't have big datasets.

[@pladselsker8340]

Imagine giving money to a service for a sense of security because it is now the status quo to let every substential company out there infringe on your privacy rights. Just a thought. What parallel universe is this?

[@FenrirRobu]

Tho didn't they warn us against meta-optimizers due to the alignment becoming impossible?

[@jymcaballero5748]

just give them more memory!

[@PhilsArtVibes]

No, no, no, I do not want to add neural networks to recursion, I JUST BEGAN TO UNDERSTAND RECURSION DON'T DO THIS TO ME!!!

[@Originalimoc]

That's unexpected

[@algorithmblessedboy4831]

guys I'm in high school and I'm trying to choose a career path. my no.1 choice considering the things I like and that I'm good at is becoming an AI reaearcher, can anyone in the academic world tell me if it would be a fun job or not?

[@user-vg2ui3wg8n]

It definitely is. But the field is getting increasingly complex, fast-paced, and hyper-competitive. I'd recommend studying computer science and mathematics, since you will not be able to compete in this field without a very strong mathematical background. Except for that, go for it. I'm a researcher in parallel processing and numerical high-performance computing. It is definitely fun and rewarding, but be prepared for a painful journey.

[@pmosg9649]

很棒😀

[@notnotandrew]

Yo dawg, I heard you like ML models...

[@donson3326]

Short answer: no

[@pedrogorilla483]

I watched half of the video and this is too technical for me. I’m skipping this one. Congrats to everyone who understands this video!

[@bycloudAI]

it's like RNN's hidden states are just ML models thanks for watching till half way tho

[@MuhammadakbarAK47]

Just watch it 3 times

[@sashank224]

​@bycloudAI ili bro I explain hold up, I'm getting what hes saying. You need break it down in simple terms that relate to real world apps. Visualize.

[@homeyworkey]

@@bycloudAI btw this was posted on r/singularity where there are more normies - obv u need normies if you want growth though, but any technical video is automatically going to have a very niche audience understandably so, so you probably dont mind that aswell. i mean i watch ur stuff and most of it goes over my head but interesting regardless, but just letting u know the feedback here is kind of skewed.

[@Guedez1]

Yeah, if you made up everything you said in the video I wouldn't be able to tell at all. Stuff is getting harder and harder to understand.

[@falsechord]

fractal ai models

[@GodbornNoven]

Nice explanations but go easy on the vocabulary. I don't reckon every joe out there knows will understand all the terms. The pacing is too quick too.

[@pauljones9150]

I'm here for the waifu memes Good video tho

[@multipurposepaperbox]

damn yeah that's AI stuff right hahaaa? tbh I understand a quarter of this, but I really enjoy a lot of your videos

[@themultiverse5447]

what?

[@boricuaxflow9669]

Are we all botted comments?

[@kingki1953]

You should consider to ban bot in your channel.

[@kingki1953]

You just upload and 3 bots already comment, dark internet is scary 😢

[@StefanReich]

@@kingki1953 Actually dark internet is really lame right now. You can spot these comments from a mile away Your videos are always so informative and interesting! Thank you for that! Thank you for your work! Your videos are always top notch! Always a pleasure to watch your videos! I will be looking forward to new episodes!

[@punk3900]

I hate such advertisment shockers that are not separated adwautely form the main material. Not gonna subscribe to a channel that does that.😢

[@muscifede]

look at the amount of bots lol

[@StefanReich]

This is nothing. Check out any popular video about trading

[@09jake12]

leenear

[@AlphaProto]

This video was too much for me.

[@mariusj.2192]

The quadratic complexity is not the main problem of current LLMs. It's that they are dog sh*t at reasoning (and tasks that depend on it) and a better scaling with context length won't solve that.

[@mitulsolanki6066]

would love to collaborate and learn with you

[@mautkajuari]

First!

[@Ps5GamerUk]

Nah, the bot beat you to first bro

[@mikemaldanado6015]

dude are all your videos infomercials for half the video????????

[@lynx_pinata]

Bro your thumbnail and fireship's thumbnail are looking similar. Someone has to change/alter their thumbnail

[@aspenlog7484]

Yeah i did not understand shit. Basically better archetecture

[@alexijohansen]

“unlock linear complexity having expressive memory bla bla bla bla bla bla” was this written by chatGPT?

[@Miaumiau3333]

It sounds human to me, even if it contains some technical jargon. ChatGPT writes differently

[@Sculptoroid]

what a load of bollocks