Building makemore Part 3: Activations & Gradients, BatchNorm

[@mileseverett]

Andrej, as a third year PhD student this video series has given me so much more understanding of the systems I take for granted. You're doing incredible work here!

[@Erosis]

This has to be the best hands-on coding tutorial for these small yet super-important deep learning fundamentals online. Absolutely great job!

[@chanep1]

I like that not even the smallest detail is pulled out of thin air, everything is completely explained

[@nkhuang1390]

Every time another Andrej Karpathy video drops, its like Christmas for me. This video series has helped me to develop genuine Intuition about how neural networks work. I hope you continue to put these out, its making a massive impact on making these "black box" technologies accessible to the anyone and everyone!

[@leopetrini]

1:30:10 The 5/3 gain in the tanh comes for the average value of tanh^2(x) where x is distributed as a Gaussian, i.e. integrate (tanh x)^2*exp(-x^2/2)/sqrt(2*pi) from -inf to inf ~= 0.39 The square root of this value is how much the tanh squeezes the variance of the incoming variable: 0.39 ** .5 ~= 0.63 ~= 3/5 (hence 5/3 is just an approximation of the exact gain). We then multiply by the gain to keep the output variance 1.

[@peace-wink]

Thank you : )

[@MonkeyKong21]

i hope they're using the actual value and just writing 5/3 in the docs as slang

[@Zoronoa01]

Thank you for the insight!

[@lennixplayzpokemon1239]

@leopetrini Can you explain how you calculated the integral?

[@sanjaybhatikar]

Awesome, thank you!

[@khuongtranhoang9197]

Turns out this should be the way to teach machine learning: a combination of theory reference and actual coding. Thank you Andrej!

[@american-professor]

Exactly. My DL course back in 2015 had a ton of obscure math and no coding. I had no idea how to train NNs after that course. I'm rediscovering and learning a ton of stuff from this video alone, way more than from my course.

[@1knmd]

The quality of these lectures is out of the charts. This channel is a gold mine!. Andrej, thank you, thank you very much for these lectures.

[@vitorzucher435]

You have some much depth in your knowledge, Yet, you manage to explain complex concepts with such and incredible didactics. This is someone who truly understands his field. Andrej, thank you so much and even more for the humility in how you do it so. You explain how libraries and languages like python and pytorch work and dive into the WHYs on why things are happening. This is absolute priceless.

[@parent5x]

Andrej you have a wonderful gift for educating others. I’m a self learner of NNs and it’s a painful process but you seriously help ease that suffering… much appreciated! Ty

[@Umar-Ateeq]

Great, I am also going through this same painful process. Can you suggest something that can help ease this pain?

[@hetanshpatel8521]

If you want to learn the theory. Try Soheil feizi . He is professor at UMD. Amazing teacher. And the course content is just top notch.

[@IchibanKanobee]

These video series are exceptional. The clarity and practicality of the tutorials are unmatched by any other course. Thank you for the invaluable help for all practitioners of deep learning!

[@hlinc2]

This series is definitely the clearest presentation of ML concepts I have seen, but the teaching generalizes so well. I’ll be using this step-by-step intuition-building approach for all complicated things from now on. Nice that the approach also gives a confidence that I can understand stuff with enough time. Truly appreciate your doing this.

[@TheInGenList]

I remember you you are badmephisto i will always remember you. you taught me how to solve the rubix cube : ) i will never forget how far you have come from the teenager in his room making rubix cube videos and look how far you have come god bless

[@swarajnanda5990]

My mind is totally blown at the detail I am getting. Feel like this is an ivy league level course, with the content so meticulously covered.

[@tianwang]

This video makes you feel grateful about internet where you can learn from masters in this much depth for free. Thank you Andrej, this whole series greatly helped my understanding of NN!

[@thodorispaparrigopoulos8542]

I cannot fathom that this video only has 4k likes... He is literally explaining stuff that no one else goes through, because they simply don't know them, but they are crucial!

[@adamskrodzki6152]

thanks for reminding to gie a like ;)

[@dreamwalker1739]

Minute by minute, this course is giving us master-level knowledge. We're being molded into experts without even attending a world-class university! 🚀

[@project-hq]

This whole series is absolutely amazing. Thank you very much Andrej! Being able to code along with you, improving a system as my own knowledge improves is fantastic

[@enchanted_swiftie]

So many small things, scrutinzers and how easily he has pointed them out one by one, step by step, problem to solution is just amazing. Love your work Andrej. You are amazing.

[@ragibshahriyear3682]

Thank you Mr. Karpathy. I am in love with your teaching. Given how accomplished and experienced you are, you are still teaching with such grace. I dream about sitting in one of your classes and learning from you, and I hope to meet you one day. May you be blessed with good health. Lots of love and respect.

[@pwdrhrn]

I think your videos are the only ones I've come across that actually explain why you have a validation split, for the developers/data scientists to check and optimise the parameters/distribution. The ability to stop and replay is invaluable for me. Thank-you so much for these fantastic videos.

[@lkothari]

You're a great teacher Andrej. This has been by far the most interesting ML course/training I have come across. Keep up the good work!

[@minhajulhoque2113]

The batch normalization explanation was amazing! Thank you for your hard work and concise and clear explanations.

[@ptrckqnln]

This is filling in a lot of gaps for me, thank you! I especially appreciate your insights about reading a network's behavior during training; they gave me a few epiphanies.

[@dontwannabefound]

Seeing this built out with the code side by side is the most helpful thing you have done here. I think this is the biggest difference in this pedagogical style. When you see the code, then it is real and clear what is happening, it is not handwaving or imagination. So this is the most valuable piece. And your gentle personality is just the wonderful cherry on top of the whole thing. Many kudos to you I hope you can continue outputting more and keep growing your contribution to ML research going forward. Incredible.

[@wonderousw33]

this is my 3rd year in AI. very grateful i come across these videos.

[@scottsun345]

Thank you, Andrej, amazing content! As a beginner in deep learning and even in programing, I find most materials out there are either pure theories or pure API applications, and they rarely come this deep and detailed. Your videos cover not just the knowledge of this field, but also so many empirical insights that came from working on actual projects. Just fantastic! Please make more of these lessons!

[@hole62]

As a former HTML nerd, I am forever indebted to the amount of precise calculations and their limits, as is expressed in this video…

[@CuriousAnonDev]

I really enjoy learning and listening to people like Andrej who love what they do and aren't doing it just for money. Shared the channel with my friends ☺️ Keep up the great work Andrej!

[@Lauren-qj6ti]

What an inspiration. Like others have alluded in the comments, I find Andrej's teaching so remarkably therapeutic.

[@AboutOliver]

You're an incredible teacher. You really have a gift. Thanks for these lectures!!!!

[@american-professor]

You are such a gold mine of knowledge, it's insane. I wish you were my DL professor during my PhD.

[@hungrydeal6154]

To put BatchNorm into perspective, I am going through Geoffrey Hinton's 2012 lecture notes on bag of tricks for mini-batch descent, it was when AlexNet was first published. Hinton was saying there was no one best way for learning method/gradient descent with mini-batches. Well, here it is BatchNorm. Hinton: "Just think of how much better neural nets will work, once we've got this sorted out". We are living in that future :)

[@amgad_hasan]

What did he mean by "Hinton was saying there was no one best way for learning method/gradient descent with mini-batches"? Did he mean initializing them?

[@theusualcouple]

Thank you @Andrej for bringing this series. You are a great teacher, the way you have simplified such seemingly complex topics is valuable to all the students like me. 🙏

[@tsunamidestructor]

Thank you so much for this, Andrej! Your series single-handedly revitalized my love for deep learning! Please keep this series going :)

[@eustin]

it's done the same for me. i'm excitedly going through each video. it feels good to be back!

[@yazanmaarouf48]

I have shot myself in the foot multiple times before these videos. Training big models are much more difficult than I initially anticipated. Time wasted sadly. But I have more confidence in myself thanks to these video. Thanks Andrej

[@kimiochang]

I completed this one today, and I just want to show Andrej my gratitude. Looking forward to the next one. Thank you very much, Andrej. Thank you!

[@greatwall2003]

Great material on the intricacies of how neural networks work. Until now, I hadn't paid attention to the distribution of values entering the activation layers, and as it turns out, this is an extremely important issue. Thanks!

[@AbhishekSingh-ee2bo]

Thank you.. I am a self learner and your series has been a milestone for me.

[@binwangcu]

Here are some calculations to appreciate how good the prediction already is for Andrej's model. Perplexity = e^(CE), random guess gives a perplexity of e^(3.3)~27, you can think of to choose one out of 27 options. After the training, the CE is 2.0, perplexity =e^(2.0)~7, you only need to select from 7 options. Previously the bigram has a CE of 2.5 and that is e^(2.5)=12 options to choose from, this accuracy is really really good. There goes the inherent uncertainties in language as Andrej mentioned, you can start the first character with any character, so 7 on average is pretty good already.

[@joaovitormeyer7817]

this video totally opened my mind about this subject I've been obsessing about for over a year. Truly an amazing video, I feel it's the bare minimum to thank you for this.

[@vpamula1]

A lot of material packed in this video; I envision understanding the mechanics of these networks to take many years, even with some prior experience.

[@vil9386]

I don't think any other book or blog or videos cover what Andrej has covered. Awesome insights. THANK YOU Andrej.

[@juleswombat5309]

These lectures are an awesome gift to us mortals. Such a clear explanation on the principles of neural networks. I only need to be able to afford access to massive TPU cloud compute and huge corpus, but at least I can now gain insight and understand the principles of these technologies.

[@sanjaybhatikar]

I keep coming back to these videos again and again. Andrej is legend!

[@cktse_jp]

Your choice of visualizations as diagnostic tool is super insightful. Thanks so much for sharing your experience.

[@divelix2666]

Can't even explain how impactful this video for my understanding of nns... Thank you so much!

[@kemalware4912]

My life is much better now because of your videos.

[@fhools]

These are some of the best lectures i've ever seen. I love the explaination in the first part about tanh saturation. Really trying to get the viewer to develop intuition.

[@mrmiyagi26]

Thank you for the deep dive into batch normalization and diagnostic approaches! Really useful to see it explained from the paper with the code.

[@afbf6522]

Amazing explanation about the nitty gritty details of Deep Learning, the "dark arts" of the trade.

[@AliAlfadhel-l6b]

love the short snippets about how to implement these tools in production.

[@gonzaloalbornoz7279]

You explain very simple things that I know and always give me a new perspective on it!. Your way of transmitting knowledge is incredible.

[@parent5x]

Exactly!

[@rmajdodin]

1:33:30 The reason the gradients of the higer layers have a bigger deviation (in the absence of tanh layer), is that you can write the whole NN as a sum of products, and it is easy to see that each weight of Layer 0 appears in 1 term, of layer 1 in 30 terms, of layer 2 in 3000 terms and so on. Therefore a small change of a weight in higer layers changes the output more.

[@AlexTang99]

This is the most amazing video on neural network mathematics knowledge I've ever seen; thank you very much, Andrej!

[@1997benjaminvh]

This is awsome!! Thank you so much for taking the time to do this Andrej. Please keep this going, I am learning so much from you.

[@8eck]

Thank you Andrej, i have finally found time to go through your lectures. I have learn and understood a lot more than before, thank you.

[@EmileAI]

You are an Angel sir The land of AI is blessed and the harvest is plenty. New AI warriors will rise from this Thanks

[@ramboli4118]

I finally understand when Statistics come into play in machine learning. It's when you introduce the randomized weights(matrices)!

[@moalimus]

Thanks very much for this, please keep them coming, you are changing lives.

[@eitanporat9892]

Thank you for delving deep into the nitty-gritty details. I appreciate the exercises!

[@PureArtMV]

What a gem. Thanks for the lectures, Andrej

[@lucianovidal8721]

The amount of useful information in this video is impressive. Thanks for such good content.

[@siddharth-gandhi]

It's so nice to have you back on KZbin! Thanks for teaching me Rubik's Cube back in the day and thanks for teaching us deep learning now!

[@yonastesh7830]

Andrej, thank you so much for your tutorials here. You've no idea how much your videos helped me. Please keep doing more videos.

[@Buchilly]

These videos are so useful, Andrej thank you so much. The parts when you wrap up the lecture, and then change your mind to add more content are my fav. 😄

[@styssine]

This is a great lecture, especially the second half building intuition about diagnostics. Amazing stuff.

[@JavArButt]

He says 'Bye', but looking at the time, it seems too early [01:18:30]. Most people don't want lectures to be long, but I'm happy this one didn't end there.

[@robertcowher]

If you decide to make more content, a video series like this with a focus on self-driving or RL for robotics would be awesome. Not that you don't have enough going on, but that's my wish-list item :) Thanks for putting an incredibly in-depth resource out here free on the internet.

[@philipwoods6720]

These videos have been incredible. Thank you so much for taking the time to make them, and I look forward to all the future ones!!!

[@ernietam6202]

Really enjoy your classes. I learnt a lot of tips for training and feel comfortable now. Will continue finishing this series.

[@bensphysique6633]

Andrej, thank you ever so much. You are an inspiration, and thanks to you I have a better understanding of the concepts.

[@PrarthanaShah-nk1xh]

I feel bad that this video dropped an year ago and I am just now watching it.

[@rafaelsouza4575]

oh man, this is top-notch content! Not sure if there are other available contents on these topics with so much clearness about its inner gears with reproducible examples. Thank you so much! You're a DL hero.

[@billykotsos4642]

Dives straight in and kills the presentation…. Another banger…. Can make old papers fun to go through….

[@dengzhonghan5125]

Your lecture is so amazing. Please keep updating, thanks for sharing and educating.

[@tildarusso]

Nice to be lectured again after watching the Stanford CS231 multiple times!

[@the-hanhpham8950]

Thank you so much for the time and effort put into the videos of this series. Appreciate it very much.

[@ziangxu7751]

Finally, all those small techniques make sense to me. Thank you so so much!

[@doktoripartise]

Thank you for explaining everything in such detail. It makes everything much more understandable

[@peterhojnos6705]

Zdravím, díky, že si sa dal na tvorbu videí pre širšiu verejnosť!

[@jeanchristophe15]

Thank you so much for your clear and thoughtful explanation Andrej!

[@THOSHI-cn6hg]

You will be remembered❤

[@bazgo-od7yj]

i mean, he didnt die

[@JuliusSmith]

Thanks for the fantastic download! You have changed my learning_rate in this area from 0.1 to something >1!

[@houbenbub]

following your lectures is a delight! Thanks for taking the time to make them :)

[@MihaiNicaMath]

I believe you can calculate the gain by doing Gain = 1/sqrt(E[ f(Z)^2 ]) where Z is a standard Gaussian (so that Gain*f(Z) will have unit variance when Z is a standard Gaussian). If you do this for tanh you ~=1.592 which I guess is close to 5/3?

[@jonathansum9084]

Yeah! New video. I 🥰😍 love it. I have decided. After finishing all of your videos, I am planning to use your model as a starting point to solve the true AI problem, just like wright brothers. I want to try my way. I still think the idea of building a a brain simiulation of neuro network is wrong . And I think I have my way to solve the problem.

[@RaviAnnaswamy]

That is the way to go, @Jonathan Sum. Pick up a really hard problem, stay with it yourself, trying to solve them in many many ways, for years if needed. Whatever it takes. You can refer to others work as needed, but ONLY AFTER you have tried each subproblem on your own. I learnt this method of solving problems for oneself by seeing Newton's biography. That is how you create new knowledge.

[@leslietetteh7292]

Have you heard of Openworm? Or the FlyEM project? They might be worth a look.

[@adamskrodzki6152]

Amazing, knowledge that is hell hard to find in other videos and also, you have AMAZING skill in clearly explaining complex stuff.

[@ShouryanNikam]

Thanks for making this! It's such an honor to be learning from you!

[@8eck]

Wow, didn't knew that there is such content and from Karpathy himself. Thank you!

[@mPajuhaan]

Very interesting how you've described the concept of pre-tuning NN.

[@chonmon]

lol although I don't really understand what's going on here, but I'm just liking and commenting to support Andrej! Keep it up Andrej!

[@smithwill9952]

Better than study in University. Keep going A.K. Share your video for sure.

[@punto-y-coma7890]

Awesome explanation Andrej! Than you very much for sharing your knowledge.

[@ninjaturtle205]

you have to watch these videos twice. Once you will just watch the videos. The next time you will try to write the code Andrej is writing from memory or from your notes. You don't progress until you are stuck, and only as the solution you will play those parts of the video.

[@TheMato1112]

Dakujem ti Andrej, je to vazne na inej urovni :)

[@sarai3538]

Thanks for the great explanation of activation,gradient as well as histogram.

[@lrostagno2000]

Thank you so much Andrej, you are a real inspiration for me and I really appreciate you

[@fabianandresvagnoni5057]

1:18:36 The "Okay, so I lied" moment was too relatable xD

[@sauloviedo2677]

Finallyyy!!! I was nervious waiting for the new video! Thank you Andrej!!!

[@riponsaha8320]

This tutorial is so engaging and insightful.