EfficientNet from scratch in Pytorch

[@AladdinPersson]

These from scratch videos & paper implementations take a lot of time for me to do, if you want to see me make more of these types of videos: please crush that like button and subscribe and I'll do it :) Support the channel ❤️: kzbin.info/door/kzW5JSFwvKRjXABI-UTAkQjoin Original paper: arxiv.org/abs/1905.11946 Paper review: kzbin.info/www/bejne/lYC9pHqHfadnZ9U Code: bit.ly/2ORFxR9 Timestamps: 0:00 - Introduction 0:45 - Imports 1:00 - Architecture config 6:10 - Implementation Structure 7:10 - CNNBlock 10:10 - SqueezeExcitation 13:05 - InvertedResidualBlock (w. Stocasthic depth) 23:44 - EfficientNet 36:22 - Running a small test case 37:55 - Ending

[@nikaize]

Masterpiece

[@jeroenritmeester73]

Good video! I guess this video didn't include dataloading, but it would be great to see how that's done, as well as saving the model after training.

[@connor-shorten]

Awesome channel, wish I found this sooner!

[@AladdinPersson]

I've watched your videos for a while! I think they're super chill & a really nice way to get an overview of a paper

[@AladdinPersson]

The kind words means a lot coming from you :)

[@Carbon-XII]

That was hard, but I've understood most of it. Thank you!

[@tonystark9497]

Great implementation, please try student-teacher architecture (training method ) basically it's knowledge distillation

[@linglongxu4196]

This is the best Pytorch Tutorial I've ever learned

[@mrigankanath7337]

I was waiting for this implementation!! thanks, I would really like a video where you try to make a very good model for a computer vision Kaggle competitions.

[@AladdinPersson]

Any competitions in particular?

[@mrigankanath7337]

@@AladdinPersson to be honest there are object detection github models and we can just clone it and apply to the data; but as in kaggle we have to improve it. So i was thinking how would you approach an object detection competition and use different methods to get a higher score.

[@SK-zs6en]

@@AladdinPerssondon't have any names in particular but the ones which allows you to showcase good amount of feature extraction and tricks in training. Also, a monthly one competition video or whatever that suits your routine on a long run would be a really informative playlist resource. Cheers.

[@seanbenhur]

@@AladdinPersson how about hubmap!.

[@edelweiss7680]

Thanks for your amazing work really ! It helps me a lot !

[@pallavidubey5482]

Hi Aladdin, Thank you! It is very helpful. I wanted to request if you could please do a similar video on Retina Net Algorithm? It does perform pretty well when compared to YOLO for vision systems, but there is not much out there on it. For a beginner, it will be very helpful! I appreciate the time and effort you put into these videos.

[@anoubhav]

At 16:43, shouldn't the kernel size of the expansion layer by 1x1 and not 3x3? Thanks for the video!

[@AladdinPersson]

I think you're right on this part, could you refer where you found this info?

[@willy8898]

@@AladdinPersson Hi, I think it is mentioned in the Table.1 of the original MobileNetv2 paper.

[@lazythor1716]

@@AladdinPersson ​at line 62, the kernel size of the expand_conv should be 1 instead of 3 because we're trying to increase the #channels using 1x1 Conv github.com/lukemelas/EfficientNet-PyTorch/blob/7e8b0d312162f335785fb5dcfa1df29a75a1783a/efficientnet_pytorch/model.py

[@SK-zs6en]

Can you make a walk through+ model building on styleGan family as well. Good work again!

[@UnpluggedPerformance]

Bro when I was watching your awesome videos about Machine Learning i thought you were a invisible machine learning God, but now I finally see you have a face and are human :-D

[@baoanhphamngoc635]

alright alright alright

[@AladdinPersson]

alright

[@baoanhphamngoc635]

@@AladdinPersson your channel and Yannic Kilcher are da best bro

[@tranquangkhai8329]

@Bảo Anh Phạm Ngọc: Same favourite channel as me

[@skylarwhite5044]

thanks for the good video. can i ask something, in list of base_model, how you know the stride, for it was confusing since never explained in paper?. or this has calculation to determine the stride or other. thanks

[@JorGe-eu3wi]

awesome! can you do efficientdet from scratch next?

[@AladdinPersson]

It's a good suggestion, I added it to my list. For now I want to do some Kaggles but I will revisit this idea

[@maciejbalawejder]

Good video! But why do you use conv in Squeeze and Excitation? As far as I'm concerned they only use convolutions instead of fully-connected as a NoSqueeze block, which performs worse than the SeBlock.

[@hamzaarslan6490]

Hey Aladdin great video, I just want to ask that I wrote your code and check the summary of total param size and I found that (around)14 million for b0. The paper is showing 5 million, do I miss something? Thank you great work 👍

[@dickymr1878]

i also noticed it, it's a problem or what?

[@moarshy]

Thank you so much for all ur videos!

[@moarshy]

how about EfficientDet next?

[@AladdinPersson]

I'll look into it ^^

[@anaydongre1226]

Can you please mention the github link for this code.

[@radhikapatil8003]

can you please make a video on custom image classification using FixEfficientNet ,it would be a great help thanks in advance

[@jitmanewtyagi565]

Awesome 💫

[@jijie133]

Great video!

[@shalini286r9]

Great video.Can you implement code for efficient as a encoder and UNet as a Decode for the following paper "A robust framework for glaucoma detection using CLAHE and EfcientNet

[@bennisyoutube]

Love the Video! Thank you so much for your content

[@박민영-k6r]

It is assigned in its parent class , nn.Module.

[@mariomedhat792]

I have a question Why are you implementing papers using pytorch not tensorflow ? Can you tell us the differences between both of them ? Another thing, there is a lite version of pytorch like the tensorflow one ? Thank you

[@AladdinPersson]

I am more productive in Pytorch and things seem to make more sense. Personally I don't see a benefit to me using Tensorflow for now. I feel that we're seeing a trend of people moving away from Tensorflow, internally Google has started the JAX project which seems to diverge from Tensorflow. OpenAI recently said that they are going to only use Pytorch moving forward. There are things lacking with regards to deployment with Pytorch but I think they are working on it with torchserve etc, with regards to TensorFlow lite I don't think there's an equivalent in Pytorch

[@yimingqu2403]

Awesome!!!

[@goksuceylan8844]

27:04 i thought i got rickrolled for a second 🤣

[@kamikon755]

In other implementation, squeeze excitation is implemented with fully connected layers. Why are you using conv layers in your implementation?

[@marcinelantkowski662]

I'm super impressed with the pace at which you produce these tutorials... How much time goes into creating a video like this? Do you get it right with the first try? Amazing content!

[@AladdinPersson]

It's quite varied, this one in particular was quite smooth and took me less than a day to go through the paper and implement it. Other implementations I've done, like the Yolo from scratch tutorial took me over a week

[@datnguyentan4093]

16:20 I wonder whether you made a mistake when using reduced_dim = int(in_channels/reduction) instead of int(hidden_dim/reduction).

[@lazythor1716]

I think it's a mistake as well

[@willy8898]

It is a really really great video for the deep learning beginner like me! Thank's for the detailed implementation in this video!

[@laimeilin6708]

Been watching videos from your channel for a while now, the concepts and implementations are very well explained!! Thanks for the amazing content! Is it possible to go through efficientdet as well (the efficientnet variant for object detection task) or DETR?

[@AladdinPersson]

I think I want to do some more Kaggle a bit but will get back to those, which of those two do you think would be most interesting, EfficientDet or DETR? I think the idea of DETR is cool but the performance is not great

[@laimeilin6708]

@@AladdinPersson yeah I agree maybe DETR would be more interesting I think integrating transformers into computer vision has become quite popular recently

[@prod.kashkari3075]

I wonder if I’ll ever get to this level of understanding in deep learning.

[@AladdinPersson]

You just see the refined version when you're watching the video. There's a lot of dumb mistakes, thing not working, me feeling like I'm never going to understand it behind every video. Stick with it whatever it is you're working on and you will improve 👊

[@buh357]

thank you for creating such a great content, I am working on a 3D medical image classification, how do i change 2D efficientnet to 3D efficientnet? which parameters should i change? thank you.

[@muhammedjaabir2609]

instead of using conv2d change it to conv3d , from my knowledge this is the easiest way to implement one

[@williambittner8828]

Amazing video! THanks :) I would actually love to see an implementation video of the recent Nerf Paper: HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling Cheers and keep the good videos coming! :)👏

[@asiskumarroy4470]

Hello,,your videos have been really helpful for me. As per my request,you made the object detection playlist and bangggg!!!!!,,,it was of great use to me. Now here i am stuck in another problem.I am learning tensorflow and ofcourse following your tutorials,but I am stuck in something.Can you please put up a video tutorial on how to do data augmentation on a tf.data dataset consisting of images and its respective masks?

[@abdelrahmankhaled3595]

hello, cool video. I have a question, how do I implement a model in EfficientNET for depth estimation. I already have my dataset. I just need to train the model. How do I do it using this? Thank You

[@omarabubakr6524]

I hope you will see my comment. can you please implement EfficientDet model? I know maybe it's long but please try to implement it in a video because I want so much. And thanks for that great content.

[@kirillkharitonov8506]

Great! Thanks a lot! I was trying to implemet EfficientNet myself, but three days in a row stacked with different problems. Then had a look at your video and took some tips and trick. Now it's working! Awesome!

[@xiangsong8942]

great work! like crushed and suscribed! can you also make a video for Bottleneck Transformers for Visual Recognition from google?

[@dimuthuanuraj4507]

@Aladdin Persson Could you please do a video about implementing EfficientNetV2 from scratch in pytorch?

[@kamikon755]

when I print out the parameters count it says 40 million parameters instead of 12 million like in the paper(b3 , 20 mill vs 7.8 mil for b0)

[@DanielGomez-ws7jj]

Aladdin!! Awesome video! Very well explained, clear. Helped me understand network details that could not get from the paper video. BTW the paper video is also very helpful. :) Thank you! It would be amazing to have similar content on DenseNet

[@adityakane5669]

@Aladdin Persson Great video! I just have a small question. Can you please demonstrate how to calculate the actual model architecture for bigger models (B1,B2,etc.)? This bit is not clear (atleast not to me) and the newer paper also uses similar techniques, so it would be great to know. Other than that, awesome video! I always learn so much watching these videos.

[@LAKXx]

Thank you good sir, this was immensely helpful

[@neilsbohr1s1]

Please make a video on how to train images from this scratch code

[@neilsbohr1s1]

NotImplementedError: what should i do

[@Martin-lv1xw]

Great content br...so brief and clear explanations

[@notimportant9018]

The absolute best channel. Thanks so much!

[@amineleking9898]

Man your videos are amazing

[@igoragoli]

like button smashed

[@НиколайНовичков-е1э]

Thank you :)

[@saurrav3801]

Nice video

[@harinkumar1073]

Hey man great work but i found one minor bug torch.randn() in stochastic_depth() sometimes returns numbers less than 0 or more than 1, this might be a problem so try using torch.FloatTensor(x.shape[0], 1, 1, 1).uniform_(0, 1) instead, as it stays in [0, 1] range hope this helps

[@AladdinPersson]

Are you sure I wrote torch.randn and not torch.rand? Could you give time stamp?

[@harinkumar1073]

@@AladdinPersson sorry my bad😅😅, you wrote torch.rand, my dumbass thought it was randn

[@maximiliandippel3414]

Great video! One question though: you said that a default 3x3 convolution is in fact a cube and that it takes 3 channels into account. But isn't it actually 3x3x(number of channels)? That's why the default value for groups is 1, which means that all channels are in that one group and are being taken into account.

[@AladdinPersson]

You're correct, yeah my bad. Only the first convolution will be a cube (assuming RGB input)

[@maximiliandippel3414]

@@AladdinPersson thanks for clarifying :)

[@siddhantshete1744]

module 'torch.nn' has no attribute 'SiLU' I dont understand why I'm getting this error. can you please help me out

[@AladdinPersson]

Update your torch version

[@siddhantshete1744]

@@AladdinPersson I updated to latest version but still it gives the same error but it worked for SELU. Can you suggest the version i should be using?

[@AladdinPersson]

@@siddhantshete1744 I don't know what could be causing the error then.. The docs have it: pytorch.org/docs/stable/generated/torch.nn.SiLU.html

[@seanbenhur]

Awesome!

[@AladdinPersson]

Thank you 👊