Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention (AI Paper Explained)

[@andreassyren329]

If nothing else, the contribution to model naming is a clear increment to SOTA.

[@jonatan01i]

Nyströmer clearly is.

[@andreassyren329]

@@jonatan01i I will agree with that.

[@VikasSingh-jv7fn]

Hello Yannic, Your comment about the order of operations is correct. It is one of those things where you set out to check how poorly it performs and find out that it could work empirically (at least in limited settings). The lemma is not practically useful but merely evaluates the setting that if/when everything is idealized, the results/procedure does not lead to nonsensical conclusions. The choice of F early on in the paper was to avoid a conflict with D (D and d were both used) and E (ones matrix).

[@tchlux]

What about the A^+ comment, was that actually a typo in the paper? kzbin.info/www/bejne/o17do5ajh8lqe5Y

[@zhanpengzeng8592]

@@tchlux Yes, that is a typo. We somehow left out the pseudo inverse sign.

[@xiongyunyang9643]

@@tchlux This is a typo. We will update it. Thanks for the catch.

[@JoelHough]

I have seen this exact type of lemma in many discussions about approximations. It did not seem out of place to me. It's nice to know that in the limit your approximation will agree with the ground truth which is certainly not the case in all approximation methods.

[@herp_derpingson]

0:42 Nyan-storm-former! . 3:30 Time for my weekly Transformer explanation :) . 27:00 That was a really sweet and easy to understand explanation. . 35:00 I wonder if we can have a DNN just predict a landmark tensor..

[@xiongyunyang9643]

Thanks for making this great video. Nice catch for the typo. We will update the draft soon.

[@mdmishfaqahmed8356]

the pronounciation of those author names were clutch :D

[@RobEnglebright]

top work pronouncing the authors

[@jamiekawabata7101]

If I lift box 1 onto shelf 1 and box 1 onto shelf 2 and box 2 onto shelf 1, then I can predict the effort in lifting box 2 onto shelf 2. Great analogy, thank you.

[@lucidraisin]

Lol, unexpectedly mentioned 😅 thanks for the video!

[@mizupof]

By the power of Yannic, I rename you!

[@poesaste]

Incredible breakdown, subbed!

[@benjaminho351]

Nobody: Yannic: uöu 1:07

[@Anirudh-cf3oc]

Very nice explaination sir, Thank you!!

[@mathematicalninja2756]

What I hear: nice transformer

[@АлексейТучак-м4ч]

that's a nice trömformer

[@MausamJain]

How did you import pdf to onenote with such a good quality? The printout option generally inserts a very poor quality images of the pages.

[@YannicKilcher]

It's definitely poor for me, too, it's right on the edge of being useful.

[@G12GilbertProduction]

Sweet Thursday with more sweety kind of paper. Buon appetite, Yannic! :)

[@KennedDansker]

It is F because it is forward attention right? (Then it would fit with B being backward). It is not entirely right (A contain part of the forward attention) but I think that is the intention

[@osiris42]

Does it even matter that the softmax doesn't commute, if the softmax is just a heuristic / hack in the first place? Or is there something inherintly special about softmax in the transformer architecture?

[@tchlux]

I don't know if I'd call it "special", but I like to think of it geometrically. When you use a softmax, you make it so that the layer immediately after the softmax only has to model a "surface" that lives on the inner wedge of the unit cube (points with 1-norm

[@mathematicalninja2756]

@@tchlux that is a good perspective

[@JamesAwokeKnowing]

So is that like a softmax over time, where it's valid kindof because over many iterations it's pulling random samples? Well hope a better way is found.

[@NilabhraRoyChowdhury]

I bet you wish you could: from previous_videos import SelfAttention every time you make a video related to transformers

[@otaviodzb1]

One thing that I still couldn't understand is how backprop works in a transformer. Does someone have a good reference or video that explains it?

[@pg1337ful]

seems like you have fundamental gaps in ML.

[@YannicKilcher]

It works like in any other neural network, by applying the chain rule to all involved operations

[@jonatan01i]

I struggle to believe that it actually is named as Nyströmformer. I'll call it Nyströmer, as suggested and should be.

[@scarcommander5517]

We like this transformer!

[@chaitanyaparmar888]

Love this video!

[@BorrWick]

Didn't this came out like yesterday??

[@ingusmant]

And?

[@BorrWick]

@@ingusmant Just amazed by the speed Yannic can read, understand and produce these videos :o

[@YannicKilcher]

You're right, it's already old now... ;)

[@muhammadsaadmansoor7777]

I was not expecting this until a month later. But where do keys queries and value come from

[@IRWBRW964]

They are learned.

[@kicckicc]

just fyi, I tried to implement this the day before yesterday, but got NAN. I checked the code and realized that the formula (14) isn't accurate and also Z0 = AS/(||AS ||1||AS ||∞) should be Z0 = transpose(AS)/(||AS ||1||AS ||∞).

[@xiongyunyang9643]

You mean the NAN is from your own implementation or our implementation? The accuracy to approximate pseudoinverse using formula (14) depends on how many iterations. Z0 is AS^T/(||AS ||1||AS ||∞). We will fix the typo in our update.

[@kicckicc]

@@xiongyunyang9643 thanks for the reply. After I used correct (14) and correct z0, the nan is gone. Just fyi, formula (16) is also inaccurate, but it is easy to be noticed.

[@xiongyunyang9643]

@@kicckicc Cool. Formula (16), similar to average local pooling, is to compute landmarks efficiently.

[@JamesAwokeKnowing]

The name was designed to sound like 'the nice transformer'. So leave the name as is.

[@pratik245]

Have you heard Michelle Srivastav but more probably you would hear Peter Chakraborty. If you can tell me the reason, you would know a lot about caste and region based targeting in India.

[@pratik245]

So, no body hates India when they are born, but, as you keep growing you see these divisions between people, majoritarianism, govt repression, targetting of intellectual class, poverty, corruption and then you start seeing trends in these concepts all in the name of highly preached American democracy and capitalism... But, Surely everything is a joke even misery.. Right Guys?

[@ZedaZ80]

I have no idea what most of this means, but the lemma was funny

[@Xrey56Cheyz]

To be honest, I expected the Performer to be the ImageNet moment for transformers, but it seems there is still a long way to go and random Fourier features are not the best way to do the thing. Somewhat sad cause Performer's idea looked so cool and well grounded :(

[@redjammie8342]

Big leaps come through simple ideas like ReLU, convolution, drop-out, residual connections, self-attention... The moment an idea becomes too convoluted, it is less likely to be game changing.

[@charlesfoster6326]

What are you waiting for? If anything, the transformer revolution seems like it's come with even more force and speed than ImageNet.

[@ahmadmoussa3771]

*The NICEtrömer*

[@NextFuckingLevel]

Indeed

[@visionscaper]

Hi there!

[@YannicKilcher]

hi!

[@weizhu2230]

OK i vote down for this work and i think the "asymmetric non local neural network for semantic segmentation" should be a better one.

[@yaaank6725]

In the last twitter chart, it's quite surprising that Performer has the worst performance across the other efficient transformers. Is this also verified by other tasks?

[@yaaank6725]

Or other people maybe..

[@xiongyunyang9643]

We have released the scores on individual LRA tasks. It will be interesting to see how Performer works for other tasks beyond LRA tasks.

[@lennartvandergoten6592]

Grüße an meinen alten ETH Kumpanen Yannic, richte Jonas schöne Grüße von mir aus :-)

[@Ronnypetson]

Noice

[@CandidDate]

I'd bet a million dollars that AGI, when discovered, uses frequencies of waves rather than any matrices.

[@kimchi_taco]

mathematically ugly but somehow works well. I don't feel good in that both Nyströmformer and Performer rely on random sampling.

[@xiongyunyang9643]

No, Nyströmformer does not rely on random sampling.