Backpropagation Details Pt. 2: Going bonkers with The Chain Rule

[@statquest]

The full Neural Networks playlist, from the basics to deep learning, is here: kzbin.info/www/bejne/eaKyl5xqZrGZetk Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@TheLLMGuy]

i think you are one of the few teachers on the planet who understands that the secret of understanding is simplicity! Thank you

[@statquest]

Thanks!

[@Priyanshuc2425]

Thank yoy for amazing fantastic video. No words is enough to say how great your channel and specially Bam! Is 😊

[@maksims4669]

Throughout all my years of Bachelor studies I was avoiding computer sciences and statistics as much as it was possible, for I could not understand them. However, when I enrolled for Masters I had no other choice but to figure it out. Last semester I had a compulsory course in Computational Intelligence, so in order to understand the material I had to find some additional sources. That's how I encountered StatQuest. You explained everything so well that through the summer I was inspired to make an additional course in Machine Learning. This semester I took several courses in statistics and optimization, next semester I will certainly take more and now I really consider to connect my life to one of these fields. All this with help of your channel. I have no doubts that there are and will be much more people like me, to whom you became a guiding light in their studies. Thank you for your work, Josh, and keep up helping the curious ones in finding answers. Have a great year 2021!

[@statquest]

Wow! Thank you very much! Good luck with your career and happy 2021!!!! :)

[@MelonsIncorporated]

I'm surprised more teachers don't know about this channel, I'm not exaggerating when I say it's the MOST incredibly useful tool in helping me understand material in a class that makes no sense.

[@statquest]

Wow, thank you!

[@tensorsj]

Josh, I cannot stress how much of an empathetic teacher you are. I went through years and years of education in engineering and I tend to forget the low level math behind it all. Here I can rescue that with you. You are just an amazing human being :)

[@statquest]

Thank you! :)

[@Vanadium404]

This is the best explanation on YT. No fancy animations just pure calculation. Better than 3B1B and others.

[@statquest]

Thanks! I specifically tried to go deeper into topics that 3B1B only skimmed over because I wanted our videos to complement each others.

[@nandakumar8936]

every time you mention the soft plus activation function you show the toilet paper pic. Every time. That's commitment right there.

[@statquest]

bam! :)

[@mrglootie101]

Finally!, tbh you're really good in teaching Josh, simple but detail, keep it! Bam! From Indonesia

[@statquest]

Thanks! 😃

[@kanui3618]

mantap bangg

[@Nandeesh_N]

"Quadruple bam" is what I feel as soon as I learn something from your video. It's so amazing and when I get any doubt, first thing I do is to check a relevant video on your channel! Thank you Josh for these amazing videos!

[@statquest]

Happy to help!

[@seanfitzgerald6165]

There were so many BAM's in this video it made my head spin, and I love it.

[@statquest]

BAM!

[@rajathk9691]

I'm really confused right now because the quality of the content really makes me want to promote you but I also wanna keep this channel a secret to myself.

[@statquest]

I'm glad you like my videos! :)

[@bruceeugene7500]

A tip : you can watch movies at Flixzone. I've been using it for watching lots of of movies recently.

[@houstonabraham6181]

@Bruce Eugene yea, have been watching on flixzone} for since december myself =)

[@adrienmaverick8334]

@Bruce Eugene yup, I've been watching on Flixzone} for years myself :D

[@aakarshanraj1176]

@Rajath K "Share your knowledge. It’s a way to achieve immortality."

[@None_me_]

This is the best thing ever... it's like you help me understand the complex concepts in 3Blue1Brown in the simplest ways in these videos....you both are the epitome of Bam!......

[@statquest]

BAM! Thank you very much! :)

[@kalpaashhar6522]

Brilliantly explained. Not only is it easy to assimilate the information due to the colours, the style in which you have broken down a complex explanation is a skill not many teachers have. Keep up the awesome work

[@statquest]

Many thanks!!

[@tangh146]

the preschool vibe really does make everything much less intimidating. love u josh 🥺

[@statquest]

:)

[@wenyange2607]

Thank you for making this video. It's very clear, easy to follow, and super helpful for understanding the algorithms.

[@statquest]

Glad it was helpful!

[@brucebalfour1002]

I love your videos so much. So helpful. I always look forward to the BAMs, not only because they are fun, but also because it gives a sense of fulfillment. BAM from Switzerland.

[@statquest]

BAM from North Carolina! :)

[@zihaozhou1256]

Honestly, some of the illustration is better than the college professors.

[@statquest]

Thank you! :)

[@panku077]

A phenomenal example of teaching from first principles. Josh's brand of elegance and charisma kept me engaged and on track to mastering this topic.

[@statquest]

Hooray! :)

[@nabeelhasan6593]

Understanding Back Propagation always give me Panic Attack but your style and simplicity is beyond amazing. Thanks for making such a complicated topic to be so easy to understand

[@statquest]

Thanks!

[@siddharthmodi2740]

I dont believe , how can someone explain this weird topic with this level of simplicity.Hats off to your efforts. Thank you josh

[@statquest]

Thank you!

[@ramyav4689]

Dear Josh, Thank you so much for making this video. I have always been intrigued by the way you explain the mathematics of complicated concepts! It is just amazing.

[@statquest]

Thank you!

[@alejandrocanada2180]

I wish all the teachers in the world explain as good as you do. Thank you very much!

[@statquest]

Wow, thank you!

[@mathscorner9281]

I wish I could find videos on every topic in which I m having problem on this channel. Sir, you are really a fabulous teacher.

[@statquest]

Thank you! :)

[@nandankakadiya1494]

The best and perfect videos on Machine learning I have ever seen bammmm thank you ......

[@statquest]

Wow, thanks!

[@fengjeremy7878]

Three lectures about backward propagation. Finally understand what's going on in this fancy technique. Thank you sir!

[@statquest]

Triple bam!!! :)

[@TheGoogly70]

Awesome! It appeared daunting in the beginning, but at the end it was so easy to understand. Great job!

[@statquest]

Hooray! :)

[@jancvrcek1541]

Erudite, funny and free of charge (triple BAM!!) what more could we wish? Absolutely love your videos!

[@statquest]

Thank you very much! :)

[@parshanjavanrood368]

With all the hype around AI and machine learning, it's not easy to find sources that teach the statistics behind these subjects, and the ones that do teach the math behind it, make it very hard to understand(from the perspective of a second-year bachelors student), but what you do is awesome. Thanks for this great series.

[@statquest]

Thank you!

[@parijatkumar6866]

As always brings a smile along with great clarity...

[@statquest]

Thank you! :)

[@huidezhu7566]

This is the clearest explanation I can find. Amazing work

[@statquest]

Thank you! :)

[@hunter8831]

You are such a great teacher that you make machine learning seem so easy and motivate anyone to learn more about it

[@statquest]

Wow, thank you!

[@shivamgaur8624]

I know I'm enjoying a video when I like it even before it starts. Amazing work!!

[@statquest]

BAM! :)

[@tupaiadhikari]

Thank You Josh, for these valuable Videos. I am immensely grateful to you for making these videos. You are truly a Legend in the Data Science Community. Love and Gratitude from Kolkata, India.

[@statquest]

Thank you very much! :)

[@haozeli9660]

damn, the quality of the content of this channel is just over the top. Hope u can get more views because u definitely deserve it!!!!!

[@statquest]

Thank you!

[@lisun7158]

[Notes] 1:27 2:45 3:06 Because of the Chain Rule, we get the formula for the derivative of SSR with respect to w1. (w3 is used) 7:31 formula for the derivative of SSR with respect to b1 (w3 is used) 9:10 formula for the derivative of SSR with respect to w2 (w4 is used) 9:15 formula for the derivative of SSR with respect to b2 (w4 is used) formula for the derivative of SSR with respect to b3 = (d SSR / d Predicted)*(d Predicted/ d b3) [ref.13:24 kzbin.info/www/bejne/f3-ViaB4na5_qpY&ab_channel=StatQuestwithJoshStarmer] Based on formula above, (The best StatQuest gives the succinctest formula I've ever seen) I get the solution for question below, Q: why called "backward"? why dynamic programming? A: The backpropagation algorithm computes the gradient of the loss function with respect to each weight by the chain rule. The calculation of the gradient proceeds backwards through the network, computing the gradient one layer at a time, iterating backward from the last layer to avoid redundant calculations of intermediate terms in the chain rule. [ref. wikipedia] 11:00 use each derivative to calculate respective step size and each new value. -- optimize all parameters of the NN simultaneously

[@statquest]

Double bam! :)

[@avinashbabudebbati1095]

"THE CHAAAAIN RULEEEEEEE" gets me every time 😂

[@statquest]

bam! :)

[@azizmouhanna8996]

Thank you from the bottom of my heart, from a phd student who learns a lot from you 🙏

[@statquest]

Happy to help!

[@mitchynz]

I love this explanation so much.... It actually helped me intuitively understand the chain rule too. I just purchased your book on machine learning which is the perfect compendium to this series.

[@statquest]

Thank you very much!

[@powangsept]

Watching the videos in your channel let me love the statistics more and more! Thank you for the great videos!

[@statquest]

Awesome, thank you!

[@aakarshanraj1176]

you really explained it better than anyone on youtube. Thanks a lot, it was really helpful.

[@statquest]

Thank you very much! :)

[@elhairachmohamedlimam9640]

Thank you so much, I have wasted a lot of time to understand these things, but really, after watching your videos things become very easy, thank you a lot

[@statquest]

Thanks!

[@amyma2204]

You saved tons of confused souls by this amazing explanation.

[@statquest]

Thank you!

[@semenbondarenko3512]

I'm your fan, seriously you are the best teacher I have ever met. ~~~Triple Thanks~~~

[@statquest]

Wow, thanks!

[@praveerparmar8157]

As a matter of fact, I was going bonkers trying to understand Backpropagation until I watched this video. Now I'm stable 😁

[@statquest]

bam!

[@anilkumar-ki1xb]

Josh, You are a true saviour....

[@statquest]

:)

[@willw4096]

Thanks for the great video! My notes: 0:59 9:49 10:54 - 11:07

[@statquest]

Nice work!

[@lakshman587]

Million billion trillion thanks for this Neural networks videos Josh! BAM!!! You are really awesome!!!!

[@statquest]

BAM! :)

[@MADaniel717]

I'm back Josh! Exactly what I needed. Now I'm going to try to implement a neural network from scratch in Python using your videos :D

[@statquest]

Go for it!

[@MADaniel717]

@@statquest github.com/danielmarostica/simple-neuralnet/blob/main/neural_network.py I did it! Can't believe haha. Thanks Josh!

[@statquest]

@@MADaniel717 TRIPLE BAM!!!! Congratulations. That is awesome!!!

[@lin1450]

Thank you so much for your content. I will be forever grateful. They way you convey the information with such simplicity, step-by-step and humor makes it so fun to watch and builds up motivation. Concepts that seemed way out of reach for me are becoming something I'm slowely building trust to truly and deeply understand them one day! Thank you so much!

[@statquest]

Thank you for your support!! When I have time, I'll design some more merch.

[@tagoreji2143]

Educating along with Entertaining. That too for a complicated topic.Thank you very much, Professor

[@statquest]

BAM! :)

[@kostjamarschke4613]

Love the video and explanation, but I love the SoftPlus commercial every time you mention the activation function even more.

[@statquest]

bam!

[@franciscoruiz6269]

You're a master! You have gain all my respect.

[@statquest]

Wow, thanks!

[@magtazeum4071]

this channel is lovely..Thank you Josh...

[@statquest]

Thank you!

[@anasmomani647]

u literally should get Nobel prize for your videos

[@statquest]

bam!

[@yonasabebe]

You make it look so easy. Thanks for your effort and contribution.🙏

[@statquest]

Thank you!

[@TheGabytls]

Thank you, thank you, thank you thank you, thank you!!! I never thought I could understand this type of thing. If somebody asks me to explain this to them I'll say... THE CHAAAIN RULE!!! Thanks Josh! I'll donate part of my first salary to you eventually :)

[@statquest]

Hooray! I'm glad my videos are helpful. :)

[@pavapequeno]

Hi Josh, what you have made was for me the first definitive guide to advanced stats and ML that is accessible and understandable by anyone with a basic scientific background. Multiple Bam! I searched a while before stumbling on this treasure trove. Thank you so much! Will you also be releasing some videos on graph embeddings, GNNs/GCNs? I think you would have a very eager audience (certainly including me at least)!

[@statquest]

Thanks! I'll keep those topics in mind, but for now I'm working on LSTM and Transformer neural networks.

[@pavapequeno]

Thanks for the quick reply, Josh :-) I‘ll keep my eyes peeled for future videos. I guess graphs and graph embeddings is almost a playlist in itself! Once again thank you for opening my eyes to this wonderful world! All the best with the current new videos!

[@ek_minute_]

thanks for the toilet paper reference of soft plus now i will always revise the backpropagation Every time i go to toilet.

[@statquest]

bam?

[@petercourt]

Fantastic work Josh! :)

[@statquest]

Thank you! And thank you for your support!

[@marccrepeau6853]

Thanks for this great series of tutorials! One thing I'm confused about: in this video some of the final parameter estimates (after the 450 gradient descent steps) end up larger than the original (random) parameter estimates, and some end up smaller. But the derivatives calculated for the first gradient descent step are all positive, so if you multiply them by a constant (positive) learning rate you will end up *decreasing* all parameters (by subtracting a positive value from them). Thinking about that parabola in your gradient descent tutorial, you would be starting with tangent lines (derivatives) all on the right side of the parabola. All derivatives are positive and thus all the tangent lines have positive slopes. Gradient descent will subtract positive step sizes from all parameters. All parameters will thus *decrease*. So how do some final parameter values end up greater than the original (random) estimates? (For example w1 is originally set at 2.74 and the final estimate is 3.34)

[@statquest]

I think the answer might just be that the 7-dimensional surface that we are trying to find the bottom of with gradient descent may have some complicated shape that includes local minima that we get out of because the step size is large enough to allow for that sort of thing.

[@marccrepeau6853]

@@statquest So if you step outside of a local minima then you might end up with negative derivatives in subsequent descent cycles? I think that makes sense!

[@statquest]

@@marccrepeau6853 bam!

[@pelocku1234]

This video was great and made me want to build this in R. One note to others that are trying to use the same starting values and get to the same optimal values, you will need to use a learning rate that is not .1 because this will find different optimal values that are not quite the same as we see here. Just wanted to drop this note in case someone else tried to recreate it like I did. Again, it was the teaching that inspired me to do it.

[@statquest]

Glad you were able to figure something out that worked for you! :)

[@twandekorte2077]

Great video. You are exceptional with regards to explaining difficult concepts in an easy and straightforward way :)

[@statquest]

Thank you, and thanks for your support!

[@ouche71]

I really liked this series, but I still feel confused what happens when we have more than one feature. Keep the great work!

[@statquest]

We'll get there soon enough.

[@yacinerouizi844]

best tutorials on machine learning, thank you!

[@statquest]

Glad you think so!

[@devharal6541]

You are best Josh!!

[@statquest]

Thank you! :)

[@dijkstra4678]

Any other videos always omit the most key details and only generally explain the concepts such that you come out of that video having learned absolutely nothing on how it actually works. The world needs more videos like yours which actually explain these concepts in mathematical detail without getting too difficult either.

[@statquest]

Thank you very much! :)

[@xinyuechang6062]

Not me clapping my hands watching that green line fit perfectly with data 🐕

[@statquest]

:)

[@tymothylim6550]

Wonderful video! Now I will always think of toilet paper when seeing the softplus activation function :)

[@statquest]

BAM!!! :)

[@user-ur2en1zq4f]

Sir, you are gold. Thanks

[@statquest]

Thank you!

[@simplifiedscience7497]

You are so amazing!! You really made me love machine learning!!

[@statquest]

Wow, thank you!

[@sreerajnr689]

This is such a fun to learn!! 😀😀

[@statquest]

Thanks!

[@Luxcium]

This was pretty cool to watch 😅but the more I watch the less remains 😢 which is so sad given how JS is so busy with the rest of his life and stuff 😮 I don’t know if he will ever have enough time to make videos awesome again MVAA

[@statquest]

:)

[@jiayiwu4101]

I like this beginning song a lot!

[@statquest]

bam! :)

[@1243576891]

Thanks for the video. This is awesome!

[@statquest]

Glad you liked it!

[@ignaciozamanillo9659]

Thanks as simple and good as always! Planning about a tutorial of Neural Networks in R / Python? Would be great

[@statquest]

That's the plan! :)

[@mischievousmaster]

Hey josh, could we expect videos where you write code in python and implement ML models?. I know you did one on decision trees and I really loved it. I know you gave us a conceptual understanding of Ml in all of your videos but what I'm requesting is to give examples for every popular Ml algorithm along with python code in three or two levels of complexity(basic, intermediate) etc. There are other channels who do this, but if we could learn it from your teaching approach it would totally be one of a kind. The reason I ask you this is because I genuinely think you're a teacher who puts himself in the shoes of a beginner's mind who is laboring to understand all the data science and statistics jargon, and tries to explain stuff so that it stokes up interest and ambition in them. It certainly worked for me. I was never good at math let alone stats. But your videos helped me tremendously and now I am a little better and thanks to you for that. I know I asked you a lot and you may not have the time for all that I requested, but hopefully one day. Thank you Josh.

[@statquest]

I've also done webinars on SVMs in python: kzbin.info/www/bejne/bnKafWN9qKecgrM and XGBoost in python: kzbin.info/www/bejne/faOtgWx8gbtmfKc and, in 2021 I hope to do more.

[@mischievousmaster]

@@statquest thank you!

[@bhavikdhandhalya]

Thank you for wonderful videos.

[@statquest]

Glad you like them!

[@LEELEE-dg3xd]

This video really helped me a lot!

[@statquest]

BAM! :)

[@yeyuan4235]

Thanks Josh for another great video! Do you plan to introduce RNN and CNN at some point? Looking forward to your clear explanations on these topics.

[@statquest]

I hope so.

[@nikolatotev]

I have a question about backpropagation: (edit after finishing writing, actually 2) In a real implementation when performing backpropagation do the weight values get after each layer is reached or does the algorithm go through the whole network, saving how each weight & bias should change, and then after reaching the start of the network all of the values get updated? And a question that is related to a more complicated version of neural networks - In Convolutional neural networks that use Skip Connections, during the forward pass results from the previous layers gets concatenated with a deeper layer. My question is - when performing a backpropagation are the skip connections used to pass the gradient directly to a layer closer to the start of the network or are the skip connections just ignored. I'm not sure if anyone else is struggling with backpropagation in CNNs, if there are more people a video on the topic with your teaching style would be amazing!

[@statquest]

My understanding is that, for each iteration of backpropagation, there is a single "forward pass", were the data is run through the neural network, and this is used to calculate the loss (in this case, the sum of the squared residuals), and then it does a single "backwards pass", where all of the parameters are updated at once. If the parameters were updated one at a time, then we would have to do a bunch of extra forward passes, one per parameter that we want to update, and that would probably take a lot longer. As for your question about CNNs, I don't know the answer, but, believe it or not, 2022 is the year of the neural network for StatQuest, and I plan on making a lot more videos about them, so I might get to this question before too long (Unfortunately I can't promise I will!).

[@ArifDolanGame]

glad I found this channel! very helpful 👍

[@statquest]

Glad to hear it!

[@sattanathasiva8080]

Best videos for stat

[@statquest]

Thanks!

[@juliank7408]

Thank you very much! Appreciated!

[@statquest]

You're welcome!

[@harishbattula2672]

Thank you for the explanation.

[@statquest]

You are welcome!

[@SuperYkf]

OMG This is amazing! 😭

[@statquest]

bam! :)

[@pratyushsinha374]

This is goldmine

[@statquest]

Thanks!

[@namanjha4964]

Thanks! Loved it!!!!!!!

[@statquest]

Thank you!

[@justinwhite2725]

Thank you for the animation at the end. I've been purplexed why my network with 2 hidden layers seems to find a 'happy medium' where all the outputs are muddled values around the median. Your graph showed this is normal. I suspect this means I need more time to train the deeper layers towards the inputs.

[@statquest]

Good luck! :)

[@engindenizalpman]

Amazing as always 😎👍

[@statquest]

Thank you!

[@rameshmitawa2246]

This guy is just too awesome :)

[@statquest]

Thanks!

[@jitpackjoyride]

Thank you for this! Love learning from statquest Hope there’s more videos for CNN

[@statquest]

More to come!

[@jakhongirkhatamov3694]

You the man!

[@statquest]

Thanks!

[@superk9059]

throughly, clearly, amazing, awsome!!! Thounds BAM!!!

[@statquest]

Wow, thanks!

[@suryatejakothakota7742]

Bam from India 😍

[@statquest]

Thanks!

[@joehsiao6224]

awesome! so it doesn't matter which weight to start with!

[@statquest]

We do them all at the same time.

[@MrDemultiplexer]

I think this series is better than 3Blue1Brown's

[@statquest]

Thank you very much! I tried to include details that he did not.

[@durrotunnashihin5480]

Question please: Why we use backpropagation and forward propagation in training process, while only forward propagation could possibly find the optimal parameters? Is it faster than only forward propagation? Or any other reason? Anw, I watched a lot of videos from your channel, it is very interactive and make the complexity simpler.. thank you for the effort :)))

[@statquest]

I'm not sure I understand your question. Why do you think it is possible to find the optimal parameters using only forward propagation? I'm pretty sure that would be impossible, but maybe you know something I do not.

[@durrotunnashihin5480]

@@statquest let's say we train the data with forward propagation. At iteration 500, we get all the gradients of the weight < 0.001. Is this not possible? of course, in practice, it could be difficult to get the optimal parameters with a few training data only

[@statquest]

@@durrotunnashihin5480 Again, I've never heard of anyone training a neural network only using forward propagation. Can you provide me a link to a reference where this is done?

[@Luthea]

And this is the first step in my adventure to create an earthquake early warning system. BAM!

[@statquest]

Let me know how it turns out! BAM! :)

[@krizroycetahimic4087]

You made me realize what AI truly is. THANKS!

[@statquest]

bam! :)

[@kanui3618]

Keep it up, josh🔥

[@statquest]

Thanks!