Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

[@kyuhyoungchoi]

I really love the way you explain. You are using very standard language which the old pre-deeplearning guys are familiar with.

[@Vroomerify]

This was great! Your explanation of batch normalization is by far the most intuitive one I've found.

[@lakerstv3021]

Love your content! Some of the best explanations on the internet. Would be amazing if you could go through the Neural ODE or Taskonomy paper next.

[@dvirginz4001]

That's the best video on youtube about batchnorm, thanks for going over the paper.

[@deepakarora987]

This is really cool explanation , would love to hear more from you.

[@KulkarniPrashant]

Absolutely amazing explanation!

[@黃一-h6b]

Thank you so much for your intro. I had a hard time grasping the concepts, this helped a lot. Thank you :)

[@madhavimehta6010]

thanks for putting effort in explaining in simpler manner

[@ross825]

Why don’t you have more subscribers, so helpful!!!

[@manasagarwal9485]

Hi, thanks a lot for these amazing paper reviews. Can you make a video about layer normalization as well, and why is it more suited for recurrent nets than batch normalization?

[@payalbhatia5244]

@ Yannic Kilcher. Thanks , it has been best explaination. You simplified maths as well. Would request you to explain all recent papers in same way. Thansk

[@tudor6210]

Thanks for going over the paper!

[@Konstantin-qk6hv]

Thank you for the review. I like to watch your videos instead of reading paper

[@Laszer271]

Batch Normalization doesn't reduce internal covariate shift, see: How Does Batch Normalization Help Optimization? arXiv:1805.11604

[@Engidea]

What is the app are you using to edit and write on the the paper.

[@MLDawn]

Absolutely beautiful man. I love how you mentioned the model.train and mdoel.eval coding implication as well. OPut of curiousity: 1) What software are you using to show the paper (not adobe right?) 2) What kind of drawing pad are you using? I have a Wacom but since I cannot see what I'm doing on it, it is annoying to teach using it really.

[@abdulhkaimal0352]

why we normalize the data and then we multiply by gamma and add it with beta i understand it produce the best distribution of data but rather cant we just multiply by gamma and add the beta without even do the normalization part ?

[@shinbi880928]

I really like it, thank you! :)

[@wolftribe66]

Could you make a video about group normalization from FAIR?

[@BlakeEdwards333]

Awesome thanks!

[@kynguyencao1630]

thank you so much

[@matthewtang1489]

Wouldn't it be cool for some professors to make the students derive the derivatives in the test =).

[@hoangnhatpham8076]

I had to do that my DL exams. Just feedforward though, nothing this involved =)

[@adhiyamaanpon4168]

someone clear the following doubt: does gamma and beta value will be different for each input feature in a particular layer?

[@YannicKilcher]

yes

[@adhiyamaanpon4168]

@@YannicKilcher Thanks a lot!!

[@AvielLivay]

But why do you need the lambda/beta? what's wrong with just shifting to average 0, variance 1? And also - how do you train them, you mean they are part of the network and so they are trained but I thought we want things not to be shaky but you are actually adding these that are adding to the 'shakiness'... what's the point?

[@rbhambriiit]

Idea is to learn the better repression. Identity or normalised or something in between. Think of it as data preprocessing

[@rbhambriiit]

Agreed it does question the original hypothesis/definition of normalisation at input layer as well

[@rbhambriiit]

It's not that shaky. It's another layer trying to learn the better data dimensions. With images identity layers work well. So the batchnorm learning should effectively reverse the mean/variance shift.

[@yamiyagami7141]

Nice video! You might also want to check out "How Does Batch Normalization Help Optimization?" (arxiv.org/abs/1805.11604), presented at NeurIPS18, which casts doubt on the idea that batchnorm improves performance through reduction in internal covariate shift.

[@dlisetteb]

i really cant understand it

[@yasseraziz1287]

YOU DA MAN LONG LIVE YANNIC KILCHER

[@michaelcarlon1831]

All of the cool kids use SELU

[@ssshukla26]

If not more then almost as complicated explanation as in the paper.

[@garrettosborne4364]

You got lost in the weeds on this one.

[@rbhambriiit]

The backprop was bit unclear n perhaps the hardest bit

[@stefanogrillo6040]

lol