Linear Transformers Are Secretly Fast Weight Memory Systems (Machine Learning Paper Explained)

[@st33lbird]

"because love is....I'm becoming like Lex Fridman" -- genius!

[@marouanemaachou7875]

😂😂😂

[@nju415]

Hahahaha I was about to comment on the same thing

[@kimchi_taco]

What is meaning of life?

[@G12GilbertProduction]

Friedman is really subjectionised reductionalist guy.

[@konghong3885]

Schmidhuber: I invented transformer

[@mgostIH]

Yannic Kilcher Is (not so) Secretely A Fast Paper Reviews System

[@yimingqu2403]

Me a few month ago: I'm not interested in this work so I won't watch it. Me now: Yannic made this so I'll watch it.

[@morkovija]

Oh my, first time i recognized the author in the paper =0 Yes, its that Jurgen

[@scottmiller2591]

The key, value, and query terminology always felt like someone who knew databases trying to explain something that is not a database. It's like trying to explain welding in terms of nails - it's really more confusing than explanatory.

[@АлексейТучак-м4ч]

the principle of storing a single piece of data in several entries of a matrix resembles holography holograms store information about light sources in a distributed fashion

[@TechyBen]

I love when mathematicians, logisticians, programmers/computer scientists and physicians find out one thing is the same as another. :)

[@saurabhkulshreshtha8953]

I know discovering symmetries! :)

[@whosthisguythinkheis]

Love your videos! On the pixelation, you might have some luck turning the PDF into high res images, importing the pages into something like Krita or Gimp which is free and do your writing on top there. (krita has better pen support though I think)

[@morkovija]

16:00 Interesting, I was not expecting the concept of arrow of time to appear in ML..

[@wqchen3535]

Thanks for your tutorial! It's very good! But I have a different opinion about what you said in 44:00-46:00 about the "d_dot=2d_key*niu". I think the paper is right. You can simply compare fai_i1(k) and fai_i3(k) when d_key=2. Although fai_11=fai_23, fai_21=fai33, fai_31=fai13, fai_41=fai_43, which is your main concern, they are different among the order of i in [1,4]. Different i corresponds to different dimension of fai(k). So, niu can be chosen from {1, 2, ……, 2d_key-1}, niu_j and niu_2d_key-j is not the same because the different order of i.

[@mdmishfaqahmed5523]

paper headings are amazing these days

[@LouisChiaki]

The keys sound very similar to what physicists call orthonormal basis. Or, very similar idea used in quantum mechanics.

[@etiennetiennetienne]

cool video and cool paper! could you provide some intuition (or some link to previous video) why softmax attention can hold exponentially many keys with respect to number of dimensions ? (i think you mentioned it for hopefield networks ?)

[@sharannagarajan4089]

This is amazing! I think underrated

[@matterhart]

I think you can load the pdf into python/acrobat and 2x/3x each page size and then save as pdf or image before dropping into one note.

[@siyn007]

I would recommend putting the logo and youtube link at the top left or top right since we never try to observe text there

[@piratepartyftw]

cool paper, clear explanation

[@andres_pq]

How long back in history does the application of vector products to route information goes? It seems plausible that the key to AGI may already be solved by an obscure paper from the 80-90s.

[@Uni1Lab]

Just for activate the algorithmes.

[@andres_pq]

To be honest, I did not quite understood the ML street talk on kernels as well as the videos you make.

[@MachineLearningStreetTalk]

We will do better 👌🙌

[@andres_pq]

@@MachineLearningStreetTalk You are great guys!

[@siddharthbhargava4857]

Thank you for the explanation.

[@mathematicalninja2756]

I sometimes feel great that I decided to do masters in maths because I was able to understand this

[@bhargav7476]

I recently picked up 'Mathematics for Machine Learning' by A. Aldo Faisal, I just know calculus at this point. Would I be able to understand maths in this paper after reading that book?

[@mathematicalninja2756]

@@bhargav7476 I don’t know as I haven’t read that book yet

[@bublylybub8743]

this does not require "master" level math to be understood. just saying.

[@444haluk]

Yannic be like: I am gonna drop transpose sign to mess up with my audience and butcher the math.

[@tho207]

awesome explanations as always. why don't you just change apps? try pdf expert

[@Erotemic]

Yannic: It's magic we can just add these outer products to get a database. Me: Wow amazing, how was I never aware of this?! Yannic: oh btw the keys need to be orthogonal. Me: ... so you can only store as many elements as the dimensionality of D... ok... cool, but I'm not nearly as impressed anymore.

[@freemind.d2714]

I actually think the autoregressive self-attention mechanism V(softmax(KT)q) is intuitively easy to understand than Fast Weight Memory Systems (V(KT))q, even though they are basically same concept with different formula

[@dr.mikeybee]

LOL! Hi, Lex.

[@corgirun7892]

oh my god

[@dr.mikeybee]

I just watched this vlog on writing out the fast network as long-term memory: kzbin.info/www/bejne/jnPKqKRqfsqDgtk

[@integralogic]

Dense Title.

[@dimonenka]

these fast weight memory systems look like glorified hypernetworks

[@benibachmann9274]

Yannic „lächerliche Geschwindigkeit“ Kilcher in Hochform!