Longformer: The Long-Document Transformer

[@ArnavArora]

Great video! Very intuitive. Slight correction, when explaining the sliding window attention using the graphs in the paper, either the range should be [i-w/2, i+w/2] or the window size is 2w.

[@YannicKilcher]

Absolutely true, thank you!

[@gary9630]

I just finished reading the paper and then watched the illustration you give in this video. TBH, you give such an informative and intuitive explanation! Nice job! I really love it.

[@jeonghwankim8973]

Great explanation. You literally tap into the main part of the paper and explain it in the most intuitive way possible. Thank you.

[@shairuno]

I love how you are always skeptical and try to justify the claims from the paper. Great vid!

[@DistortedV12]

Wow that was fast! this paper just came out. Will check vid tomorrow.

[@mnk6436]

Awesome video and great explanation! Thank you!

[@sagumekishin5748]

Speaking of convolution and attention, there are papers suggesting self attention can replace convolution completely in vision tasks. I think those papers are worth covering. "Stand-alone self-attention in vision models" "On the Relationship between Self-Attention and Convolutional Layers"

[@priyamdey3298]

Very interesting line of work. Thanks for the info!

[@laveenabachani]

Thank you for making this. This was very helpful. Loved it!

[@b.jardim2079]

Great video, you made it very clear! Thanks.

[@Arwin_Unbeatable]

Great video. A big thumbs up

[@chrisber]

This was excellent, thank you so much for your channel!

[@kikimajo6850]

Love this series！

[@OccultDemonCassette]

Hmm, I wonder why "special tokens" are turned off on a lot of the Collab tests I've seen on longformers. Seems like they would be beneficial?

[@dummyrezajzadeh]

beautifully explained thank you

[@GeekProdigyGuy]

If there are 2 layers with kernel size 3, and only the second layer is dilated skipping every other unit, the second layer will not "miss" any local information simply by the adjacent windows overlapping. So I don't think using dilation only at higher layers necessarily goes against the importance of locality.

[@YannicKilcher]

Correct. Information can still aggregate with depth. My point was that their argument for the sliding window was the importance of locality and the dilation is directly counter to that. But yes, they solve that by only dilating the higher layers where they argue that locality does not matter as much anymore.

[@qian2718]

Didn't realize the memory consuming would be same until I watched the video😭

[@iliemihai949]

This video is so awesome, and well explained. You are a great teacher ! Also was wondering if you know if the document classification datasets that they use for testing are available ? Thank you and keep it up.

[@АлексейТучак-м4ч]

Idea inspired by global+sliding window: n1 nodes would connect to c1 previous nodes, n2 to c2, n3 to c3 etc. and they all are randomly shuffled

[@YannicKilcher]

Nice idea, but you'd lose the inductive prior that neighbors are important

[@شهریارصادقی-ت8ز]

great video

[@mehdimashayekhi1675]

great job! keep it up

[@drdca8263]

Question: could it work to have the model select on-the-fly some subset (with a small maximum size, like, at most 10 or so, idk) of nodes to treat as special, and as being able to connect to everything? (And, which nodes are counted as special would differ between layers) Like, have something estimate for each node how useful it would be to have that node be checked against all of the nodes, not just the nearby ones, and then when computing the dot products, and associated matrix, compute the dot products for the nearby pairs and the pairs where at least one of the two was selected as important? I imagine maybe that would be hard to train because not differentiable? And also might be slow to compute? Disclaimer: I don’t know what I’m talking about.

[@YannicKilcher]

You're absolutely right. First, yes that could be a very valid idea and second, yes it would be very hard to train, because as soon as you introduce "hard" attention like this, you have no learning signal flowing back.

[@drdca8263]

Yannic Kilcher Thanks!

[@sauravmukherjeecom]

Weirdly, your idea is very similar to bigbird. Your idea was definitely very valid. For Yanic's skepticism of information flowing back, they kept the subset same across layers and changed the subsets only across different sequences.

[@riasingh2558]

Intuitively, how is Longformer different from Transformer-XL? Finally, how do Transformer-XL, Longformer, and Linformer compare with each other if Long and Lin -former bot have linear complexity/ Thanks for great content!

[@YannicKilcher]

In transformer-xl you don't explicitly train the carry-over mechanism, as I understand it. The linformer projects the sequence length down, while the longformer only attends to a sub-part of the sequence.

[@freemind.d2714]

Great video as always!! Did you think those sliding window and dilated sliding windows is very similar to idea of WaveNet architecture ? for make the deeper layer gain the all input informations more efficient

[@YannicKilcher]

It's certainly related, but the dilated convolutions will then pull the dilations together in the next layer, not sure if that's happening here.

[@herp_derpingson]

It would be interesting to see what this model would look like under OpenAI microscope. If that is even possible.

[@YannicKilcher]

Not directly. The microscope optimizes inputs in continuous space. Here you'd have to optimize the discrete text input. Not entirely clear how that would work.

[@majorminus66]

If local attention is really what matters, why bother building a model that can attend over entire massive documents? Why do we need to overcome the 512 seq limit if local attention is really most important?

[@LegoGunshipper]

It achieves better results. There is no 'why'

[@Timtom0707]

If every way of coloring the matrix is a valid way of cutting down on the attention calculations, maybe it would be interesting to do some kind of architecture search over possible colorings? It seems unlikely that the assumptions they've made are the optimum - maybe there's room some kind of hierarchical structure? hmmm

[@paramsraman3948]

Great review of the paper! Very clear and helpful.. Mind sharing what tools you use for the presentation? (to zoom in, annotate with your markers, get to whiteboard on the side of pdf etc).. It is pretty cool

[@YannicKilcher]

It's just OneNote

[@Chr0nalis]

Thx for the paper. Sound level is on the low side, perhaps you could look into normalizing the level before uploading.

[@YannicKilcher]

Noted, thanks for the feedback.

[@pratik6447]

In the long former model the config file has > and >. What does these 2 parameter means? Which one is token size?

[@ty7521]

4098 is the token size

[@priyamdey3298]

A thought crossed my mind: Do you think special tokens act more like the cell states of LSTM?

[@YannicKilcher]

That's very possible

[@taku8751]

It just aggregate all seperated parts processed by formal transformer.

[@adamtran5747]

i love it

[@zingg7203]

The volume is really low. Have to max to hear your talk

[@johngrabner]

A more symmetrical approach: split seq into n. 2 layers of attention, first across n, second first element of n fer every n. Am I missing something?