Building makemore Part 3: Activations & Gradients, BatchNorm

We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video.
Links:
- makemore on github: github.com/karpathy/makemore
- jupyter notebook I built in this video: github.com/karpathy/nn-zero-t...
- collab notebook: colab.research.google.com/dri...
- my website: karpathy.ai
- my twitter: / karpathy
- Discord channel: / discord
Useful links:
- "Kaiming init" paper: arxiv.org/abs/1502.01852
- BatchNorm paper: arxiv.org/abs/1502.03167
- Bengio et al. 2003 MLP language model paper (pdf): www.jmlr.org/papers/volume3/b...
- Good paper illustrating some of the problems with batchnorm in practice: arxiv.org/abs/2105.07576
Exercises:
- E01: I did not get around to seeing what happens when you initialize all weights and biases to zero. Try this and train the neural net. You might think either that 1) the network trains just fine or 2) the network doesn't train at all, but actually it is 3) the network trains but only partially, and achieves a pretty bad final performance. Inspect the gradients and activations to figure out what is happening and why the network is only partially training, and what part is being trained exactly.
- E02: BatchNorm, unlike other normalization layers like LayerNorm/GroupNorm etc. has the big advantage that after training, the batchnorm gamma/beta can be "folded into" the weights of the preceeding Linear layers, effectively erasing the need to forward it at test time. Set up a small 3-layer MLP with batchnorms, train the network, then "fold" the batchnorm gamma/beta into the preceeding Linear layer's W,b by creating a new W2, b2 and erasing the batch norm. Verify that this gives the same forward pass during inference. i.e. we see that the batchnorm is there just for stabilizing the training, and can be thrown out after training is done! pretty cool.
Chapters:
00:00:00 intro
00:01:22 starter code
00:04:19 fixing the initial loss
00:12:59 fixing the saturated tanh
00:27:53 calculating the init scale: “Kaiming init”
00:40:40 batch normalization
01:03:07 batch normalization: summary
01:04:50 real example: resnet50 walkthrough
01:14:10 summary of the lecture
01:18:35 just kidding: part2: PyTorch-ifying the code
01:26:51 viz #1: forward pass activations statistics
01:30:54 viz #2: backward pass gradient statistics
01:32:07 the fully linear case of no non-linearities
01:36:15 viz #3: parameter activation and gradient statistics
01:39:55 viz #4: update:data ratio over time
01:46:04 bringing back batchnorm, looking at the visualizations
01:51:34 summary of the lecture for real this time

[@mileseverett]

Andrej, as a third year PhD student this video series has given me so much more understanding of the systems I take for granted. You're doing incredible work here!

[@leopetrini]

1:30:10 The 5/3 gain in the tanh comes for the average value of tanh^2(x) where x is distributed as a Gaussian, i.e. integrate (tanh x)^2*exp(-x^2/2)/sqrt(2*pi) from -inf to inf ~= 0.39 The square root of this value is how much the tanh squeezes the variance of the incoming variable: 0.39 ** .5 ~= 0.63 ~= 3/5 (hence 5/3 is just an approximation of the exact gain). We then multiply by the gain to keep the output variance 1.

[@peace-wink]

Thank you : )

[@MonkeyKong21]

i hope they're using the actual value and just writing 5/3 in the docs as slang

[@Abhishekkumar-qj6hb]

Hi leopetrini could you check my comment above n answer if possible

[@Zoronoa01]

Thank you for the insight!

[@lennixplayzpokemon1239]

@leopetrini Can you explain how you calculated the integral?

[@Erosis]

This has to be the best hands-on coding tutorial for these small yet super-important deep learning fundamentals online. Absolutely great job!

[@thodorispaparrigopoulos8542]

I cannot fathom that this video only has 4k likes... He is literally explaining stuff that no one else goes through, because they simply don't know them, but they are crucial!

[@adamskrodzki6152]

thanks for reminding to gie a like ;)

[@nkhuang1390]

Every time another Andrej Karpathy video drops, its like Christmas for me. This video series has helped me to develop genuine Intuition about how neural networks work. I hope you continue to put these out, its making a massive impact on making these "black box" technologies accessible to the anyone and everyone!

[@parent5x]

Andrej you have a wonderful gift for educating others. I’m a self learner of NNs and it’s a painful process but you seriously help ease that suffering… much appreciated! Ty

[@Umar-Ateeq]

Great, I am also going through this same painful process. Can you suggest something that can help ease this pain?

[@hetanshpatel8521]

If you want to learn the theory. Try Soheil feizi . He is professor at UMD. Amazing teacher. And the course content is just top notch.

[@ragibshahriyear3682]

Thank you Mr. Karpathy. I am in love with your teaching. Given how accomplished and experienced you are, you are still teaching with such grace. I dream about sitting in one of your classes and learning from you, and I hope to meet you one day. May you be blessed with good health. Lots of love and respect.

[@chanep1]

I like that not even the smallest detail is pulled out of thin air, everything is completely explained

[@khuongtranhoang9197]

Turns out this should be the way to teach machine learning: a combination of theory reference and actual coding. Thank you Andrej!

[@american-professor]

Exactly. My DL course back in 2015 had a ton of obscure math and no coding. I had no idea how to train NNs after that course. I'm rediscovering and learning a ton of stuff from this video alone, way more than from my course.

[@vitorzucher435]

You have some much depth in your knowledge, Yet, you manage to explain complex concepts with such and incredible didactics. This is someone who truly understands his field. Andrej, thank you so much and even more for the humility in how you do it so. You explain how libraries and languages like python and pytorch work and dive into the WHYs on why things are happening. This is absolute priceless.

[@hungrydeal6154]

To put BatchNorm into perspective, I am going through Geoffrey Hinton's 2012 lecture notes on bag of tricks for mini-batch descent, it was when AlexNet was first published. Hinton was saying there was no one best way for learning method/gradient descent with mini-batches. Well, here it is BatchNorm. Hinton: "Just think of how much better neural nets will work, once we've got this sorted out". We are living in that future :)

[@amgad_hasan]

What did he mean by "Hinton was saying there was no one best way for learning method/gradient descent with mini-batches"? Did he mean initializing them?

[@IchibanKanobee]

These video series are exceptional. The clarity and practicality of the tutorials are unmatched by any other course. Thank you for the invaluable help for all practitioners of deep learning!

[@hlinc2]

This series is definitely the clearest presentation of ML concepts I have seen, but the teaching generalizes so well. I’ll be using this step-by-step intuition-building approach for all complicated things from now on. Nice that the approach also gives a confidence that I can understand stuff with enough time. Truly appreciate your doing this.

[@swarajnanda5990]

My mind is totally blown at the detail I am getting. Feel like this is an ivy league level course, with the content so meticulously covered.

[@1knmd]

The quality of these lectures is out of the charts. This channel is a gold mine!. Andrej, thank you, thank you very much for these lectures.

[@project-hq]

This whole series is absolutely amazing. Thank you very much Andrej! Being able to code along with you, improving a system as my own knowledge improves is fantastic

[@lkothari]

You're a great teacher Andrej. This has been by far the most interesting ML course/training I have come across. Keep up the good work!

[@CuriousAnonDev]

I really enjoy learning and listening to people like Andrej who love what they do and aren't doing it just for money. Shared the channel with my friends ☺️ Keep up the great work Andrej!

[@ptrckqnln]

This is filling in a lot of gaps for me, thank you! I especially appreciate your insights about reading a network's behavior during training; they gave me a few epiphanies.

[@minhajulhoque2113]

The batch normalization explanation was amazing! Thank you for your hard work and concise and clear explanations.

[@enchanted_swiftie]

So many small things, scrutinzers and how easily he has pointed them out one by one, step by step, problem to solution is just amazing. Love your work Andrej. You are amazing.

[@kimiochang]

I completed this one today, and I just want to show Andrej my gratitude. Looking forward to the next one. Thank you very much, Andrej. Thank you!

[@theusualcouple]

Thank you @Andrej for bringing this series. You are a great teacher, the way you have simplified such seemingly complex topics is valuable to all the students like me. 🙏

[@scottsun345]

Thank you, Andrej, amazing content! As a beginner in deep learning and even in programing, I find most materials out there are either pure theories or pure API applications, and they rarely come this deep and detailed. Your videos cover not just the knowledge of this field, but also so many empirical insights that came from working on actual projects. Just fantastic! Please make more of these lessons!

[@vpamula1]

A lot of material packed in this video; I envision understanding the mechanics of these networks to take many years, even with some prior experience.

[@eitanporat9892]

Thank you for delving deep into the nitty-gritty details. I appreciate the exercises!

[@tsunamidestructor]

Thank you so much for this, Andrej! Your series single-handedly revitalized my love for deep learning! Please keep this series going :)

[@eustin]

it's done the same for me. i'm excitedly going through each video. it feels good to be back!

[@rmajdodin]

1:33:30 The reason the gradients of the higer layers have a bigger deviation (in the absence of tanh layer), is that you can write the whole NN as a sum of products, and it is easy to see that each weight of Layer 0 appears in 1 term, of layer 1 in 30 terms, of layer 2 in 3000 terms and so on. Therefore a small change of a weight in higer layers changes the output more.

[@siddharth-gandhi]

It's so nice to have you back on KZbin! Thanks for teaching me Rubik's Cube back in the day and thanks for teaching us deep learning now!

[@gonzaloalbornoz7279]

You explain very simple things that I know and always give me a new perspective on it!. Your way of transmitting knowledge is incredible.

[@parent5x]

Exactly!

[@mrmiyagi26]

Thank you for the deep dive into batch normalization and diagnostic approaches! Really useful to see it explained from the paper with the code.

[@divelix2666]

Can't even explain how impactful this video for my understanding of nns... Thank you so much!

[@AboutOliver]

You're an incredible teacher. You really have a gift. Thanks for these lectures!!!!

[@fhools]

These are some of the best lectures i've ever seen. I love the explaination in the first part about tanh saturation. Really trying to get the viewer to develop intuition.

[@american-professor]

You are such a gold mine of knowledge, it's insane. I wish you were my DL professor during my PhD.

[@cktse_jp]

Your choice of visualizations as diagnostic tool is super insightful. Thanks so much for sharing your experience.

[@yonastesh7830]

Andrej, thank you so much for your tutorials here. You've no idea how much your videos helped me. Please keep doing more videos.

[@8eck]

Thank you Andrej, i have finally found time to go through your lectures. I have learn and understood a lot more than before, thank you.

[@MLwithAmir]

These videos are so useful, Andrej thank you so much. The parts when you wrap up the lecture, and then change your mind to add more content are my fav. 😄

[@philipwoods6720]

These videos have been incredible. Thank you so much for taking the time to make them, and I look forward to all the future ones!!!

[@1997benjaminvh]

This is awsome!! Thank you so much for taking the time to do this Andrej. Please keep this going, I am learning so much from you.

[@AbhishekSingh-ee2bo]

Thank you.. I am a self learner and your series has been a milestone for me.

[@juleswombat5309]

These lectures are an awesome gift to us mortals. Such a clear explanation on the principles of neural networks. I only need to be able to afford access to massive TPU cloud compute and huge corpus, but at least I can now gain insight and understand the principles of these technologies.

[@sanjaybhatikar]

I keep coming back to these videos again and again. Andrej is legend!

[@moalimus]

Thanks very much for this, please keep them coming, you are changing lives.

[@Lauren-qj6ti]

What an inspiration. Like others have alluded in the comments, I find Andrej's teaching so remarkably therapeutic.

[@greatwall2003]

Great material on the intricacies of how neural networks work. Until now, I hadn't paid attention to the distribution of values entering the activation layers, and as it turns out, this is an extremely important issue. Thanks!

[@ernietam6202]

Really enjoy your classes. I learnt a lot of tips for training and feel comfortable now. Will continue finishing this series.

[@the-hanhpham8950]

Thank you so much for the time and effort put into the videos of this series. Appreciate it very much.

[@vil9386]

I don't think any other book or blog or videos cover what Andrej has covered. Awesome insights. THANK YOU Andrej.

[@rafaelsouza4575]

oh man, this is top-notch content! Not sure if there are other available contents on these topics with so much clearness about its inner gears with reproducible examples. Thank you so much! You're a DL hero.

[@PureArtMV]

What a gem. Thanks for the lectures, Andrej

[@AlexTang99]

This is the most amazing video on neural network mathematics knowledge I've ever seen; thank you very much, Andrej!

[@styssine]

This is a great lecture, especially the second half building intuition about diagnostics. Amazing stuff.

[@dengzhonghan5125]

Your lecture is so amazing. Please keep updating, thanks for sharing and educating.

[@houbenbub]

following your lectures is a delight! Thanks for taking the time to make them :)

[@lucianovidal8721]

The amount of useful information in this video is impressive. Thanks for such good content.

[@dang242]

Thank you for explaining everything in such detail. It makes everything much more understandable

[@user-og6xo6iw9h]

love the short snippets about how to implement these tools in production.

[@yazanmaarouf48]

I have shot myself in the foot multiple times before these videos. Training big models are much more difficult than I initially anticipated. Time wasted sadly. But I have more confidence in myself thanks to these video. Thanks Andrej

[@kemalware4912]

My life is much better now because of your videos.

[@afbf6522]

Amazing explanation about the nitty gritty details of Deep Learning, the "dark arts" of the trade.

[@ShouryanNikam]

Thanks for making this! It's such an honor to be learning from you!

[@jeanchristophe15]

Thank you so much for your clear and thoughtful explanation Andrej!

[@ukaszlipniak2365]

I'm learning a lot. Thanks for these lectures!

[@mPajuhaan]

Very interesting how you've described the concept of pre-tuning NN.

[@alexandermedina4950]

Amazing video tutorial, thank you professor Andrej.

[@riponsaha8320]

This tutorial is so engaging and insightful.

[@punto-y-coma7890]

Awesome explanation Andrej! Than you very much for sharing your knowledge.

[@hole62]

As a former HTML nerd, I am forever indebted to the amount of precise calculations and their limits, as is expressed in this video…

[@sarai3538]

Thanks for the great explanation of activation,gradient as well as histogram.

[@adamskrodzki6152]

Amazing, knowledge that is hell hard to find in other videos and also, you have AMAZING skill in clearly explaining complex stuff.

[@tildarusso]

Nice to be lectured again after watching the Stanford CS231 multiple times!

[@8eck]

Wow, didn't knew that there is such content and from Karpathy himself. Thank you!

[@peterhojnos6705]

Zdravím, díky, že si sa dal na tvorbu videí pre širšiu verejnosť!

[@EmileAI]

You are an Angel sir The land of AI is blessed and the harvest is plenty. New AI warriors will rise from this Thanks

[@josephngari1659]

I absolutely LOVE this series!!!!

[@sauloviedo2677]

Finallyyy!!! I was nervious waiting for the new video! Thank you Andrej!!!

[@aureliencobb199]

Thank you for the lectures. I really appreciate the effort.

[@ianfukushima1316]

Thank you! You made me happy with these lectures :)

[@robertcowher]

If you decide to make more content, a video series like this with a focus on self-driving or RL for robotics would be awesome. Not that you don't have enough going on, but that's my wish-list item :) Thanks for putting an incredibly in-depth resource out here free on the internet.

[@JuliusSmith]

Thanks for the fantastic download! You have changed my learning_rate in this area from 0.1 to something >1!

[@ramboli4118]

I finally understand when Statistics come into play in machine learning. It's when you introduce the randomized weights(matrices)!

[@casvanmarcel]

I love these videos! Thanks for making them!

[@mikehoops]

Thank you Andrej, fantastic tutorial

[@fetulhakabdurahman4720]

Your lectures are the best

[@eduardtsuranov712]

incredibly cool videos, huge thanks, all the best!

[@karolduracz4008]

This is very helpful and interesting. Thank you.

[@ayogheswaran9270]

Thanks a lot Andrej!! Thank you for all your efforts❤.

[@user-kx9fi2vo6k]

you are really a wonderful teacher.... thank you..... i will meet you soon...

[@aayushsmarten]

Andrej's transformation between 15:14 and 15:16 was pretty quick 😉

[@venkateshmunagala205]

Thanks Andrej :) . It gave me a better understanding on the concept

[@4mb127]

Thanks so much for this series.

[@fabianandresvagnoni5057]

1:18:36 The "Okay, so I lied" moment was too relatable xD

[@billykotsos4642]

Dives straight in and kills the presentation…. Another banger…. Can make old papers fun to go through….

[@smithwill9952]

Better than study in University. Keep going A.K. Share your video for sure.

[@user-eb8ih5mm3w]

very beautiful Thank you Andrej.

[@starshipx1282]

Love you Andrej❤❤. I hope you keep going with this

[@TheMato1112]

Dakujem ti Andrej, je to vazne na inej urovni :)

[@escaraskaskosky9]

Thanks so much for this videos; they re amazing