Swin Transformer paper animated and explained

[@SomexGupta]

Awesome Video, explained concept in very easy to understand, Small query at time 2:21 when we divided 256*256 pixels in 16*16 pixels then total number of token should be 256 according to me, as (256*256)/(16*16) = 256 tokens but in explanation it's mentioned 16 tokens can you guide on this.

[@AICoffeeBreak]

Hi, you are right, my mistake. Pinned your comment, thanks!

[@CristianGarcia]

Alternative title for the paper: Convolutional Transformer.

[@astroferreira]

Great video! I think the passage in the abstract is related to the fact that text has a fixed scale compared to images. The smallest piece of text you can have is a single character while for images, a single pixel can represent wildly different scales and can't really be considered the 'smallest scale possible'. In microscopy a single pixel can have scales of 1e-4 m while for astronomy a single pixel can represent kiloparsecs or ~1e19 m.

[@minhquanao7492]

I think the idea of applying Transformer over a small window also appears in "Deformable DETR: Deformable transformers for end-to-end object detection". However, like Deformable Convolution, this paper lets the model learn the location where each patch pays attention to rather than fix the attention window (e.g. the immediate 3x3 neighborhood).

[@SuperShadowmasterZ]

I saw a similar tranformer useage in Fastformer: Additive Attention Can Be All You Need

[@visintel]

I love the low-key comparison to simple convolution. Looks like we made a full circle lol.

[@AnilKeshwani]

My gosh these video explainers are good. Fantastically clear and intuitively presented

[@AICoffeeBreak]

Glad you think so! :)

[@tane_ma]

I am a new fan of the channel. Always good and quick explanations and logic/storytelling, animations, segmented sections, length of videos, and link for the paper and repo in the description ❤

[@AICoffeeBreak]

Hey, thanks for the kind words! Happy to have you here.

[@veggeata1201]

I'm not sure of the origin of windowed attention, but it is used in big bird along with other sparse attention methods.

[@soumyasarkar4100]

isn't shifted window based self attention similar to local attention in longformer ?

[@AICoffeeBreak]

🤫 you're diminishing the novelty.

[@LKRaider]

@@AICoffeeBreak LOL

[@ThamizhanDaa1]

I think SWIN Transformer perforcmance should be compared with other convNets for semantic segmentation, including DeiT regular size.. you're right its pretty deceptive to ignore those results haha. But then again, this is a good idea for self-attention, regardless of this

[@lucasbeyer2985]

That paper where you've seen this before is either HaloNet or SaSaNet (standalone self-attention)

[@erdemakagunduz2078]

great video. But if we must compare a Fyodor Dostoevsky novel to something in vision, it is not an entire single image, it is a Andrei Tarkovsky movie. So moral of the story, vision still rocks! :)

[@sachinlodhi8542]

at 2:23 how 16x16 pixel patches generated from 256x256 image would sum up to 16? Would there not be total of 256 patches of 16x16 ?

[@AICoffeeBreak]

Hi, you are right, my mistake. I've pinned a comment explaining this, thanks!

[@debanjanchakraborty9946]

really love your content and i actually shifted algoritms coz they dont run on my system and i wanted more accurate results

[@paoloceric6464]

Nice video, but I think you made a mistake when calculating the number of patches (both in the 256x256 and 1920x1920 example). A 16x16 patch would produce 256 patches in the first image, and 14400 in the second, not 16 and 120.

[@AICoffeeBreak]

It's totally possible I made a mistake, but for the moment, I do not get it. We said that a 256^2 pixel image would need 16 of those 16^2 patches. A 1920^2 pixel image would need 14400 of those 16^2 patches. How do you calculate this?

[@spongemeryl]

Same comment/doubt here, maybe I didn't quite get it right, but isn't 256^2/16^2 = 256, and 1920^2/16^2 = 14400?

[@paoloceric6464]

@@AICoffeeBreak Okay, then it seems I don't get what a patch/image vector actually is. You said "if the image is 256x256 pixels then extracting 16x16 patches would lead to 16 patches", but why only 16? If we divide a 256x256 image into 16x16 squares, we get 256 squares, that's my only point. If we indeed only use 16 of those 256 squares then my question is - why?

[@AICoffeeBreak]

@@paoloceric6464 An image of 256x256 pixels has 256 pixels in width, 256 pixels in height. A 16x16 patch, is a pixel tile of 16 pixels in width and 16 pixels in height. How many of these patches do you need to achieve a complete tiling of the image? 16. Because 256/16=16. So We need 16 patches to tile the image with them.

[@patakk8145]

@@AICoffeeBreak are you sure? You can’t just divide 256 by 16, that only gives you the amount of patches in one dimensions (e.g. width). In order to fill the whole area you need 256 patches. Or you can think of it as 256x256=65536 total pixels that you’re filling with 16x16=256 pixel patches. There’s obviously 256 of them in the whole image.

[@reasoning9273]

I think you forgot to square 120. 1920x1920 resolution will generate 14.4k image tokens of size 16x16, which is 3164 times more computation compared to 256x256 case when calculating dot product attention. I don't think any single GPU can manage this calcuation.

[@undefined-mj6oi]

2:28 Could you please explain why 16*16 patches lead to 16 image tokens here?

[@AICoffeeBreak]

😅 No, I can't because it leads to 256 image tokens. See the whole comment and thread by @Paolo Čerić in here where he was the first to make me realize this mistake.

[@undefined-mj6oi]

@@AICoffeeBreak Got it! Thanks!

[@asn9329]

can this transformer be used for super-resolution task, for unpaired data.

[@anirudhthatipelli8765]

Thanks a lot! This was very detailed!

[@keroldjoumessi]

Very nice video. I have really enjoyed it as it was quite easy to follow with no prior knowledge. Therefore I don't quite understand why we still need to transform the patch vectors (features dimensionality) into another dimensionality C. In other words, what is the idea behind this transformation (from the initial features dimensionality to another C-feature dimensionality)?

[@philip2.042]

Because we re merging multiple vectors from self attention layer into one, we enlarge our representarion vector (C) under a hypothesis that it will better capture more info coming from larger patches

[@anomanees]

At 3:45, 256x256=65536, so wouldn't we have that many tokens, instead of 63504? Very nice video, thank you!

[@AICoffeeBreak]

Yes, indeed. Thanks for noticing!

[@littlevu735]

Great channel, keep going!

[@Jack-gb1nw]

was it potentially the Longformer or the Reformer NLP papers that reminded you of localised attention?

[@Jose-pq4ow]

The tricks needed to efficiently run these models on computer vision tasks seem to be too "complex" in comparison to standard CNNs....

[@AICoffeeBreak]

Yeah, it looks quite messy at the moment. On the other side, tricks to get CNNs to work were complex at their time too (pooling, dropout, fully convolutional architectures, batch norm, etc.). It's just that we got used to it (and educated about it). After the proliferation of tricks to make the transformer more data-efficient and get it to work on long sequences, there will be half a dozen of tricks that will stick with them and will be taught to posterity as actually quite simple tricks. It looks like quite a mess because we are not there yet.

[@kristoferkrus]

Great video! And I know you published it close to two years ago, but about the window-limited self attention, I guess that's pretty standard in generative LLMs nowadays, such as Llama or the GPT family by OpenAI?

[@kristoferkrus]

But maybe I'm diminishing the novelty now 😁

[@AICoffeeBreak]

Yes, it is the case for Long Context Transformers. But the problem there is the network forgets at the end what was said in the beginning. So the paper on attention sinks is a simple hacky solution to that.

[@kristoferkrus]

@@AICoffeeBreak Thanks; I will check it out!

[@ishaqkhan5418]

Its really great video, but maybe you had to explain the architectures in a little more details like 3 4 minutes more would have made it the best. Anyways thank you for the great content!

[@AICoffeeBreak]

Thanks for your feedback! :) Appreciate it!

[@madhavjariwala4548]

Thank you for this video. You're the best!

[@AICoffeeBreak]

Glad it was helpful!

[@nilsmuller9286]

Great content as always. :)

[@AICoffeeBreak]

Glad you think so! Thanks for watching.

[@chez8990]

Longforner restricts attention window to expand token limiit

[@AICoffeeBreak]

Thanks for this, Longformer is a great reference. Even before Swinformer, there were papers restricting the attention window. This idea has now become even more represented.

[@alouped]

Nice videos, thanks for putting in the work.

[@DerPylz]

Shifted WINDOWS transformer by Microsoft research 🤔🤔🤔🤔

[@AICoffeeBreak]

It's because they could not shift the Linux, lol.

[@DerPylz]

@@AICoffeeBreak I prefer shifted Apple transformers. Even though they are often confused with pizza...

[@kaustavdas6550]

Casa? Cascading Self attention seems similar?

[@toyuyn]

Isn't that just local attention? "Yeah but you can achieve global attention at later layers because of the receptive fields" Isn't that what CNN's do? Then why bother with transformers? "..." Something something attention, something something dynamic convolutions.

[@elinetshaaf75]

Whatever buzzword makes a publication these days.

[@AICoffeeBreak]

It seems that a lot of research nowadays is to introduce some of the inductive biases of the CNN into the transformer. What is better than a complete related work? An incomplete one and a paper that claims to be the first to have invented the wheel. :)

[@VVi11]

pretty much

[@cipritom]

My thoughts exactly. So the gains must come from somewhere else (over the ConvNets). And indeed, a few months later, we have ConvNeXt showing the gains do indeed come from other parts, not from attention

[@RAZZKIRAN]

Great channel , thank u

[@mrigankanath7337]

if image size is 256 x 256 and patch size is 16 x 16 shouldnt there be 256 tokens? ((256 x 256)/ (16 x16)) = 256

[@AICoffeeBreak]

Hi, you are right, my mistake. I've pinned a comment explaining this, thanks!

[@Peebuttnutter]

thanks!!

[@giantbee9763]

Very nice video :) !

[@AICoffeeBreak]

Thanks for the visit! I saw you commented something on the "Generalization - Interpolation - Extrapolation video" but the comment is no longer available. Either: 1. you removed it 2. YT removed it (did you have a link in there?) But I did not remove it. I am actually quite curious to know what you had to say there. :) I am mentioning this because I had previous experience of good comments being removed by YT without any of my doing and people were a little perplexed and confused why I am censoring them. 🤐

[@giantbee9763]

@@AICoffeeBreak Hi Letita, Yup I did comment on the video but I ended up removing it, so it wasn't the KZbin algorithm this time. :D That's all :D I had been living under the rock of "not using twitter", so I'm probably quite late to the party anyway.

[@AICoffeeBreak]

Haha, great to hear then that YT is not messing with comments this time. :) Still curious what you had to say. I guess it will stay forever a mystery. 🤫

[@sachinlodhi8542]

at 3:30 how 256x256 pixels result in 63504 ?

[@AICoffeeBreak]

Hi, you are right, my mistake. I've pinned a comment explaining this, thanks!

[@Harry-jx2di]

Thanks!

[@yusufani8]

I am putting here a counter for how many times I forget what does Swin Transformer. Counter = 1

[@AICoffeeBreak]

😅

[@shinkai791]

A little bit like "local attention Transformer"?

[@AICoffeeBreak]

Bingo.

[@lucasbeyer2985]

Haha no need to be triggered. By "scale of visual entities" they mean "size of things in the picture", so that sometimes an orange covers just 10 pixels and sometimes it covers 1000 pixels. This effect indeed does not really exist in language.

[@shubhamsuryavanshi1461]

Great work 😃, could you please make a video on deformable transformers for end to end object detection? ☺️

[@syedadzha362]

Amazing video

[@AICoffeeBreak]

Thanks!

[@amreamer362]

Very awesome

[@subhanshubansal4704]

Local Attention ? (Shifted Windows)

[@fast_harmonic_psychedelic]

is it able to encode text or is the image projection able to be compared via cosine similarity like clip? can this replace clip? Let me know in the comments below

[@AICoffeeBreak]

It's s transformer, so sure you can have the two branches in CLIP to be replaced by two Swim Transformers.

[@fast_harmonic_psychedelic]

@@AICoffeeBreak i tried it but i cant figure it out. so many outputs to swin that are different shapes that are incompatible. I tried to have it encode separately side by side with clip and then maybe get a mean of both encodings but theres just too many errors and parameters to change i ended up giving up. what i dont understand is -- whats the point of this without some sort of text module? Like.. what does it do.. lol .. it just takes the image and outputs the same image?

[@fast_harmonic_psychedelic]

Like - i can understand if this was replacing clips VIT it would be magical to get attention on all these different scales. But alone, with no understanding of token embedding similarity to image patches -- is it just good for benchmarking or what lol

[@AICoffeeBreak]

@@fast_harmonic_psychedelic I get your problem. So, no. This image-only transformer (in its current form) basically autoencodes the image, yes. But there are special [CLS] tokens to solve tasks like image recognition.

[@fast_harmonic_psychedelic]

@@AICoffeeBreak is there some sort of map of CLS tokens that someone could refer to in order to activate certain features?

[@yimingqu2403]

ICCV 2021 best paper

[@AICoffeeBreak]

Really? You're attending?

[@yimingqu2403]

@@AICoffeeBreak not me, but my colleagues at MSR

[@AICoffeeBreak]

@曲一鸣 Well then, congrats to the authors! 👏

[@erengurses123]

1920x1920 image have 120 image tokens where patch size is 16x16 ???? At least 120 should be the square of something.

[@gauravlochab9614]

using detr for face recognition

[@CyrusVatankhah]

Can you get rid of the coffee bean? Or if it is your "brand", at least don't change/move it throughout the video. It is super distracting!

[@AICoffeeBreak]

Thanks for sharing your feedback. We had this discussion in a video before, so I did a poll on this: kzbin.infoUgkxU0F0Y69SrC6HhZ6uD97gVxrANlH1CElk I personally am quite attached to her.