Batch Normalization (“batch norm”) explained

[@deeplizard]

Machine Learning / Deep Learning Tutorials for Programmers playlist: kzbin.info/aero/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU Keras Machine Learning / Deep Learning Tutorial playlist: kzbin.info/aero/PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL

[@rey1242]

I already asked it on another video, but just to cover the most area as possible Could I possibly normalize the weights to have mean 0 and variance 1 on weights initialization?

[@coolbits2235]

I am in debt to you for teaching me so much in one day. I would have kissed your hand in gratitude if you were in front of me. NN are such a convoluted mess but you have made things easier.

[@Itsme-wt2gu]

Can we make a game where ai have their own life and we live as their family and social system with our friends

[@kareemjeiroudi1964]

I'm deeply impressed by the quality of your videos. Allow me to say that these, by far, are the most helpful video tutorials on Neural Networks. I seriously appreciate the time you spend researching such information and then putting it in such a concise pleasant way, that's also easy to comprehend. Trust me without you, I wouldn't have been able to understand what changes these parameters make in the network. That's why, thank you very very much for both the time and the effort you put into this! And please, please, keep making more tutorials. Also, I'd like to remark that the topics of these videos are so sequential, so if you're following the playlist from the very beginning you'd absolutely be able to make sense of everything noted in the videos, regardless what your prior knowledge of Neural Networks is. Besides, the Keras playlist is also complementary and adds up a lot to the learning experience. All in all, this is - in one word - "professional work".

[@deeplizard]

Wow kareem, thank you so much for leaving such a thoughtful comment! I'm very happy to hear the value you're getting from this series, and we're really glad to have you here!

[@vdev6797]

i don't allow you to say..!!

[@WahranRai]

It was the purpose of this *deep learning* videos : to be *deeply* impressed by the *learning* you get

[@dr.hafizurrahman9374]

God Bless you, my dear Teacher. I saw in every lesson that you put the whole ocean in a small jar. This is the unique qualities and very few teachers have such good quality.

[@deeplizard]

Thank you, Hafiz!

[@PatriceFERLET]

Several days that I read several articles to understand what really does Batch Norm, and I found your video. Perferctly explained, thanks a lot !

[@tamoorkhan3262]

One of the few youtube series I have completed in my life. Instead of beating around the bushes, you kept it to the point with the tons of info just in few minutes. Hope to see more such series.

[@woudjee2]

Literally watched all 38 videos in one go. Thank you so much!

[@deepcodes]

Finally completed the series of deep learning, Thank You for such amazing videos and blogs for giving free on KZbin. It's great quality!!!

[@tanfortyfive]

Top-notch, I finished it all, kudos to the Deeplizard team, love you all, love you Mandy, your sweet voice keep up us.

[@linknero1]

Thanks, I'm writing my thesis thanks to your explanations!

[@FernandoWittmann]

Great video! But from my understanding, only g and b are trainable. In 4:23, it is mentioned that the mean and std are parameters as well ("these four parameters ... are all trainable")

[@deeplizard]

Thanks Fernando, you’re right! The blog for this video has the correction :) deeplizard.com/learn/video/dXB-KQYkzNU

[@davidireland724]

came looking for this comment! thanks for stopping me losing my mind trying to reconcile this explanation to the paper

[@nasiksami2351]

THANK YOU SO MUCH FOR THIS AMAZING PLAYLIST! One of the best channels for learning deep learning. Absolutely loved your content. It was explained in the easiest possible way and awesome graphical illustrations. You really worked hard on the editing! Thanks again!

[@smithflores6968]

I found, pure gold ... ! Great video ! I understood perfectly !

[@robinkerstens516]

just like all other comments: i have just finished your video series and I am impressed by the quality of explanation. Many videos go into tiny details way to fast, before making sure that everyone at least understands the terms. Kudos! I hope you make many more.

[@deeplizard]

Thank you Robin! Much more content available on deeplizard.com :)

[@JoeSmith-kn5wo]

Great playlist!! I went through the entire Deep Learning playlist, and have to say it's probably one of the best at explaining deep learning in a simplistic way. Thanks for sharing your knowledge!! 👍

[@PritishMishra]

Hurray, Completed the series (The only series on KZbin which I have seen from the first video to last without skipping a second). Amazing job Deep Lizard Team. Highly Appreciated! Now I am going to see the Keras Playlist and den I will see the Pytorch series and den Reinforcement learning

[@deeplizard]

Congratulations! 🎉 Keep up the great work as you progress to the next courses!

[@parthbhardwaj2262]

I am really fascinated by your hard work that bring such quality to your videos ! I would be really happy if you could make as much more stuff as possible. Channels like yours only keep up the spirits of students like us really high! Just one word to sum it up....... OUTSTANDING !!

[@stwk8]

Thank you Deeplizard!. The playlist of Machine Learning & Deep Learning Fundamentals made me understanding the concepts of ML super easily. Thank you so much :D

[@yelchinyang148]

The online tutorial is very useful and helps me understand in detail batch normalization concept, which has confused me for a long time. Thanks very much for your sharing.

[@deeplizard]

You are welcome!

[@OKJazzBro]

Batch norm according the paper is actually applied before the activation function, not after. For this reason, they even recommend dropping the bias parameter of the layer itself because batch norm comes with a learnable bias term. The output of batch norm then goes to the activation function.

[@aniketbhatia1163]

These tutorial videos are one of the best ones I could find. The explanations are extremely lucid and so easy to understand. I really hope you expand your pool of videos to include other topics such as RNNs. You could also dedicate some videos to hyper-plane classifiers, SVMs, RL, even some optimization methods. All in all the set of videos is just amazing!

[@ranitbarman6471]

Cleared the concept. Thnx

[@lingjiefeng3196]

I love your tutorial. The illustration is just so concise and easy to understand. Thank you for all your effort in making these videos!

[@baqirhusain5652]

Beautiful !! super clear !

[@hamzawi2752]

Very Excellent, I hope you continue this series. Your explanation is so clear.

[@al-farabinagashbayev5403]

I think every machine learning specialist even specialized one will find in you course something new for himself.:) Great course, Thanks a lot!

[@senduranravikumar3554]

Thank you so much mandy... i have gone through all the videos... 😍😍😍 .

[@silentai3826]

Wow, this is awesome. Kudos to you! Perfect explanation. Was trying to understand batchnorm from some websites and articles, this was much better than any of them. Thanks!

[@qusayhamad7243]

thank you really you are the best teacher in the world. I appreciate your efforts

[@deeplizard]

Happy to hear the value you're getting from the content, qusay!

[@qusayhamad7243]

@@deeplizard I am so happy for your reply to my comment ^_^

[@HasanKarakus]

The best explonation I ever watch

[@sciences_rainbow6292]

i completed thes series of this videos, can't wait to watch more on your playlist!

[@deeplizard]

Awesome job! See all of our deep learning content on deeplizard.com :)

[@karelhof319]

finding this channel has been a great help for my studies!

[@khalilturki8187]

nice short video and great way of explaining! I will follow this channel and watch more videos! Keep up the great work

[@rowidaalharbi6861]

Thank you so much for your explanations!. I'm writing my phd thesis and your tutorial helped me a lot :)

[@punitdesai4779]

Very well explained!

[@ahmadnurokhim4168]

Great quality content, subscribed ️‍🔥

[@pallavbakshi612]

Wow, thanks for putting this up. You deserve every like and every subscribe. Great job.

[@jonathanmeyer4842]

Nice tutorial, clear, professional voice and animations ! Looking forward more deep learning videos :) (I'm aware of your Keras tutorial series and I'm going to watch it right now !)

[@deeplizard]

Thank you, Jonathan! I'm glad you're liking the videos so far!

[@yashgupta417]

Very well explained

[@gurpriyakaur2109]

Amazing explanation!

[@ericdu6576]

AMAZING SERIES

[@marioandresheviacavieres1923]

I'm deeply thankful 🤓

[@jerseymec]

Thanks for the amazing series! I really enjoyed your videos! Keep up the good work! Hope to see more complex networks made simple by you!

[@ejkitchen]

Great content. Like many others have said, one of the best series on ML out there.

[@robertc6343]

Ohhh what a wonderful narrative. I really like the way you explained it. Thank you and I’ve just Subscribed to your channel👍🏻

[@alphadiallo9324]

That was very helpful, thanks

[@julianarotsen6521]

Thanks for the amazing explanation!! By far the best tutorial video I've seen!

[@aravindvenkateswaran5294]

I have successfully binged (across 2 weeks) this playlist and found them really helpful! Thank you for all you do and keep up the good work. Hope to watch more vids getting added here or elsewhere on the channel. Lots of love:)

[@deeplizard]

Thank you, and great work! Check out the homepage of deeplizard.com to see all other DL courses and the order in which to take them after this one!

[@richarda1630]

Just wanted to say kudos and thanks so much for your awesome series :D I have learned so much! Now I'm off to your Keras w/TF series :)

[@deeplizard]

Great job getting through this course!

[@richarda1630]

@@deeplizard Thanks! moving to your Deep Learning and Keras series next :)

[@shaelanderchauhan1963]

Just WoW! Amazing content. Please make series on Explainig research papers

[@simonbernard4216]

just woaaa ..! Please keep making these videos, it's by far the best explanation I got here

[@Yadunandankini]

great video. precise and concise. Thanks!

[@arohawrami8132]

Thanks a lot.

[@jacky2476]

your video is very clear! Thx

[@from-chimp-to-champ]

As always, very well done and clear, thank you!!

[@gaurav_gandhi]

Clearly explained, good animation, covered most areas. Thanks

[@orcuncetintas2258]

Great video, very clear and understandable. However, I want to point out some mistakes. In the batch norm, only b and g are trainable; not the m and s. Moreover, batch norm is applied after fully connected/convolutional layers but before activation functions. Therefore, it doesn't normalize the output from activation function; it normalizes the input to the activation function.

[@anshumaandash137]

Nice Explanation. However, there is a small mistake you can correct. We batch normalize the outputs of a layer (Conv or Linear) before squashing it through the activation function. That way, the activations never overshoot or understood, leading to a stable output and easier convergence. This also allows us to use bigger learning rates. I hope that helps..

[@karatugba]

I am sorry that this is the last video in the playlist I want more😢

[@fanusgebrehiwet6286]

gentle and to the point. Thank you.

[@travel_with_rahullanannd]

I really enjoyed learning with your videos. Can you please create videos on RNN.!!

[@entertainment8067]

I watched this complete playlist of deep learning. It was totally very amazing. but my suggestion is that please add some video about RNN and also make a separate video playlist about Supervised learning, Unsupervised Learning, Imitation Learning and Deep Reinforcement Learning. Thanks you mam.

[@deeplizard]

You're welcome, I'm glad you enjoyed it! We have some of the topics you've suggested already available in other courses. Check them out here: deeplizard.com/courses

[@amirraad4437]

Thank you so much for your great work ❤

[@UtaShirokage]

Amazing and concise video, thank you!

[@roxanamoghbel9147]

so helpful!

[@prasaddalvi3017]

These are really good set of videos for neural network. I really liked it a lot and enjoyed watching it. Great work. But there is just one thing which I would like to suggest, you guys have explained Back propagation really well, better than most that I have seen, but it would be really helpful in understanding back propagation better if you could add a small numerical problem for back propagation calculation.

[@pranav-24-f9t]

awesome...I am going to watch the playlist....

[@nikhillahoti7628]

This is a gem! Thank you very much!!!

[@diogo9610]

Wonderful work. Thank you for setting up all this content.

[@mukulverma8404]

Very good Explanation..watched this whole playlist.Thanks for making understanding DL so easy and fun.Moreover your funny stuff made me laugh.

[@yuriihalychanskyi8764]

Thanks for the video. So do we have to normalize data before putting to the model or batch normalization does it itself in the model?

[@ogsconnect1312]

Thanks

[@mkulkhanna]

Very nice tutorial, thank you

[@akhtarzaman7864]

thankyou for amazing explanation

[@mustafacannacak9279]

Love your channel

[@FuryOnStage]

this was an amazing explanation. thank you.

[@deeplizard]

Thanks, Nika!

[@rapunziao2929]

i started to fall in love with the voice

[@srighakollapuajith4015]

very nice video

[@lucaslucassino]

Hey I have a question! It is sometimes preferred to have a batchnorm layer after a convolutional layer and after the activation layer. Does anyone know why?

[@bl7395]

@deeplizard please do a series on transfer learning, or more in-depth teaching on NLP/CV :)

[@Hi-zlv]

I finished all 38 videos. Great great great explanation! Can you also do some sample projects?

[@deeplizard]

Great job finishing the course, Zehra! Many projects are included in our other various deep learning courses. Check out all the courses listed on the home page of deeplizard.com. We give the recommended order for which to take the courses there as well.

[@Hi-zlv]

@@deeplizard Sure! I will check the website. I also recommend it to my friends. Thank you, Mandy!

[@shamikbanerjee4713]

Thank you Ma'am. :)

[@oriabnu1]

I have seen your all videos,i am Ph.D. student truly learn many things from you ,if you have time please teach how can variational autoencoder use in CNN

[@peteabc1]

Ahh, explained in human language..thank you :). What I don't understand is where to insert those layers? Intuition tells me just everywhere, right? Btw the scale problem and such equations are called stiff equations (the NN is an equation solved by using numerical methods). But another problem is denormalization with numbers close to 0, that can cause 200-300 times slowdowns even with modern CPUs.

[@deeplizard]

Thanks, peteabc1! Yeah, you would want to insert batch norm after your "typical" layers, like dense, conv, etc. that are followed by an activation. From my experience, determining when/where to add batch norm involves testing and analyzing my training results after adding or removing more batch norm layers. But yes, you certainly can add a batch norm layer after _all_ of these typical layers and observe how your model performs. Also, thanks for the stiff equations info!

[@anirudhgangadhar6158]

Could you intuitively explain how having scale differences between features could lead to the “exploding gradient” problem ? That wasn’t clear to me ..

[@sanaullahaq2422]

{ "question": "What kind of parameters are g and b?", "choices": [ "Learnable Parameters", "Hyperparameters", "g learnable and b hyperparameter", "g hyperparameter and b learnable " ], "answer": "Learnable Parameters", "creator": "Sanaulla Haq", "creationDate": "2021-07-19T09:27:20.333Z" }

[@florianhofstetter6859]

Why do we standardize the data via a batch and not for every single record?

[@ArgumentumAdHominem]

Could you please clarify why would one normalize a RELU layer, as shown in the example? In the discussion you suggest that it is to prevent cascading effects due to too large weights. However, later, you state that BN normalizes the output of the unit. How are the two related? Further, why normalize a RELU unit that is already bound to [0, 1]?

[@Iamine1981]

Parameters m and s are not learned in the classical sense via SGD, but rather are estimated from the mini-batch sample population. The other two parameters scale and offset are learnable.

[@deeplizard]

You're right, this has been corrected in the corresponding blog for this episode: deeplizard.com/learn/video/dXB-KQYkzNU

[@RandomShowerThoughts]

This video was amazing

[@ismailelabbassi7150]

Great course, but i did not understand when u said that axis = 1 means that features should be normalized. Because Batch norm is to normalize the output of the activation function in a layer !!!!??? Help please

[@donfeto7636]

thanks for your video I was wondering We normalize the Z which is Z=W*b, not the A which is The output of the activation function and the output of the perceptron also

[@kartikpodugu]

In the example you mentioned, about miles driven in 5 years. Why did you mention that the data isn't necessarily on the same scale? I didn't get that. Can you elaborate? 1:48

[@saluk7419]

I did not understand what the axis argument is doing at all. How is it related to the features? What axis is it referring to? Can someone explain in a little more detail?

[@user-qt3jo9tw6m]

Good stuff, thank you

[@ismailelabbassi7150]

Another question ma'am u said that this calculation with the two arbitary parameters g and b sets a new std and mean for the data. can u provide me with a little explaination for those two parameters and how they set a new mean and std for our data??? thank u in advance

[@736939]

4:26 I think only "g" and "b" are trainable, but not the standart deviation or the mean that are calculated.

[@deeplizard]

Thanks, you're right. It's corrected in the corresponding blog: deeplizard.com/learn/video/dXB-KQYkzNU

[@mhdalkadri9228]

Brilliant !!

[@RH-mk3rp]

The batch norm paper has the normalization step right before nonlinearity. Why is it done after relu here?

[@roger_is_red]

so what is axis =1 I couldn't understand what you said? thanks

[@heejuneAhn]

I have a question on the slide around 4:00. Why do we need multiple and some parameter value after normalizing the value? The step will transform the value range. In term of the original papers, they say identify transform. In fact I wonder why we use 'identiy transform' which essentially makes no chnage to the input.