ConvNeXt: A ConvNet for the 2020s - Paper Explained (with animations)

[@Chris9-z7v]

Hello Letitia, thank you for this great video! Correct me if I am wrong, but I am pretty sure that I have seen the 1/4 size ratio you talk about in 12:38 in both the original ViT paper and the "Training data-efficient image transformers & distillation through attention" paper that I have read. In the original ViT paper they use this MLP block ratio in all almost all of their experiments, without mentioning it implicitly whilst in the second one, they mention the 1/4 ratio of the MLP block in page 5 of the paper. I am a newbie in Deep learning and transformers though so take everything I say with a grain of salt 😅

[@AICoffeeBreak]

Thanks! Yes, it's Table 1 in the ViT paper. Then we totally misunderstood what that factor 4 was referring to while making the video. 🙈

[@charlesfoster6326]

Tacking on, an expansion ratio of 3 or 4 in the MLP is also pretty standard in transformers for natural language tasks.

[@CristianGarcia]

They went all in with the storytelling on this paper, they even extracted the core design choices as "wisdom bits". I really don't believe they achieved the final architecture this way but reading the "linear improvement story" was very entertaining.

[@DerPylz]

First coffee bean of the year!! 🎉 congrats on the 11k subs!

[@Kartik_C]

Thank you Miss Coffee Bean! The 60 sec explanation of translational equivariance was amazing!

[@MrMadmaggot]

3:32 I love how SKEWED that fookin graph is maam is just fkn nuts.

[@ElQaheryProductions]

This is a really nice way of reviewing papers! Keep it up!

[@hannesstark5024]

10K subscriber congrats! ^^

[@AICoffeeBreak]

Yes! Thank you! 🤝 Means a lot from an early subscriber like yourself.

[@marverickbin]

Cant wait to try a unet with convnext backbone

[@pourmohammaddeveloper2034]

many many thanks from Iran

[@RodrigoCh]

Fastest 20 min ever! Thank you for the clear explanation. I especially like how you animate the explanation! May I ask what do you use to do the animations? Maybe you could add some FAQ section; I can imagine you get this question a lot.

[@AICoffeeBreak]

Thanks, this comments makes us very happy! I do not want to make a FAQ section: comments and questions are good for making the Algorithm believe it should push us further up into your recommendations: I animate everything but Ms. Coffee Bean in good old PowerPoint (yeah, tools are what you can make of them 🙈 ). Ms. Coffee Bean is animated in the video editing software: kdenlive (available for all operating systems and open source).

[@giantbee9763]

Great point on how we jumped right into transformers and forgotten to exactly pin down the effect of small tweaks. Great video again! :D

[@edwardbrown2873]

Love this. Superb. Keep it up!

[@AICoffeeBreak]

Thank you! Will do! 😀

[@giantbee9763]

I think what they might have meant by inverted bottle neck : Key, value, query and the residual connections :D Though would you call that an inverted bottle neck? What do you think @letitia?

[@AICoffeeBreak]

No, it is a tiny detail that concerns how the MLP layer is built. d -> 4d -> d. Here is Alexa explaining this (link with the right time stamp: kzbin.info/www/bejne/n5XMep-ifraCnLc )

[@AICoffeeBreak]

I missed the point there in the video when talking about inverted bottlenecks. I thought about the Swin Transformer 🙈

[@giantbee9763]

@@AICoffeeBreak That's right! I forgot about how the positional feedforward layer is constructed.Which indeed is an inverted bottleneck.

[@butterkaffee910]

LeCun must be so happy right now

[@AICoffeeBreak]

Absolutely. 😆

[@hyunkim2172]

Many thanks!

[@hararani]

Hello Letitia, thank you so much for you video it's great inspiration for my thesis. If you don't mind can I ask you question? In your opinion Is it possible if I do research paper that compare between ViT, DEiT and ConvNext for image classification in 10.000 images as newbie? because the model is considered new and not so many paper already implement those models. Thank you.

[@JapiSandhu]

Can convnext be used for video classification with time series data? Can there be a 3D-Convnext ? Like how there would be a 3DCNN?

[@AICoffeeBreak]

I do not see why this wouldn't be extendable to video. :)

[@TimScarfe]

Awesome 🔥🔥😎😎

[@gabrieldealca4829]

What is the best state-of-the-art architecture for regression tasks involving images?

[@stewartjohnston7213]

Needed to hear this 🙌!! Get the stats you deserve = P r o m o s m!