Backpropagation Details Pt. 1: Optimizing 3 parameters simultaneously.

The main ideas behind Backpropagation are super simple, but there are tons of details when it comes time to implementing it. This video shows how to optimize three parameters in a Neural Network simultaneously and introduces some Fancy Notation.
NOTE: This StatQuest assumes that you already know the main ideas behind Backpropagation: • Neural Networks Pt. 2:...
...and that also means you should be familiar with...
Neural Networks: • The Essential Main Ide...
The Chain Rule: • The Chain Rule
Gradient Descent: • Gradient Descent, Step...
LAST NOTE: When I was researching this 'Quest, I found this page by Sebastian Raschka to be helpful: sebastianraschka.com/faq/docs...
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
If you'd like to support StatQuest, please consider...
Buying my book, The StatQuest Illustrated Guide to Machine Learning:
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
KZbin Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
0:00 Awesome song and introduction
3:01 Derivatives do not change when we optimize multiple parameters
6:28 Fancy Notation
10:51 Derivatives with respect to two different weights
15:02 Gradient Descent for three parameters
17:19 Fancy Gradient Descent Animation
#StatQuest #NeuralNetworks #Backpropagation

[@statquest]

The full Neural Networks playlist, from the basics to deep learning, is here: kzbin.info/www/bejne/eaKyl5xqZrGZetk Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@sahanamd707]

In neural network, does the gradient for parameters are calculated parallel? For example: when I start with finding gradient for all the 7 parameters, do I calculate all 7 parameters simultaneously by taking the previous iteration values or, first I calculate the bias gradient and get the new bias, then calculate predicted value by new bias and then calculate gradient for w3 ? And so on till w1 ?

[@statquest]

@@sahanamd707 Everything is done at the same time.

[@sahanamd707]

Thank you

[@chaitanyasharma6270]

the way you explain things,so patiently and in depth, i feel 200% more confident in the topic afterwards

[@statquest]

Awesome! :)

[@KenJee_ds]

I wish I had this when I was first learning backpropagation! Can I "work my way backward" with this knowledge haha

[@statquest]

BAM! :)

[@romanrandall2106]

Pro tip: you can watch movies on flixzone. Me and my gf have been using it for watching lots of of movies lately.

[@amoszahir7346]

@Roman Randall Definitely, have been using Flixzone for years myself :)

[@rajeevradnair]

haha good one !!

[@eduardbenedic9844]

@Roman Randall and @Amos Zahir are bots but nice one

[@magabosc2451]

BAM !!! I'm doing my PHD in this field, and it is the BEST serie of videos that I have watched since the bigenning of my study ! Thank you so much for that :D

[@statquest]

Thanks and good luck!

[@joserobertopacheco298]

I'm writing from Brazil. This channel's playlist about neural networks is a masterpiece.

[@statquest]

Muito obrigado! :)

[@KzrLancelot]

join a cartel

[@arindammitra2293]

Triple BAM (Explanation)+Triple BAM (Animations)...... You are a very great teacher Josh Starmer :) :)

[@statquest]

Wow, thanks!

[@Vanadium404]

This NN series is so underrated just 124K I mean come on

[@statquest]

Thanks!

[@nonalcoho]

BAMMMMMMM! I like the animation in the last part and the music with Fan~cy notation lol

[@statquest]

BAM! :)

[@TheAkiller101]

I really like the medieval guitar sound you added when you said "fancy notation" , you effort can really be seen in the little details

[@statquest]

Thanks!

[@shafir360]

I am watching all of these eventhough i already graduated with masters with focus on machine learning and deep learning. its actually amazing how much I am learning even as a intermediate student.

[@statquest]

bam!

[@victorreloadedCHANEL]

We all should buy his book, he deserves it given the quality of these videos!!

[@statquest]

Thank you!!! :)

[@peki_ooooooo]

yes!

[@MultiSamarjit]

​@@statquest Hey man, just bought your book,will be arriving in a few days via amazon.All these topics are covered right?

[@statquest]

@@MultiSamarjit The basics of neural networks and backpropagation are covered. The other topics are listed here: statquest.org/statquest-store/

[@erenplayzmc9452]

OMG THE HAPPINESS I WAS FEELING WHEN I UNDERSTOOD EVERYTHING, you seriously are a really good teacher.

[@statquest]

Thank you!

[@averagecandy2581]

The details are just out of this world. Amazing. Breath-taking and short of words.

[@statquest]

Thanks!

[@voyam]

Had to watch 17:09 at least ten times. But now I get the most dificult part: the orange and blue curves, represent... the orange and blue curves. Without that, I would be completely lost 😆. Thanks for the hard work. Amazing series!

[@statquest]

I'm glad you figured it out! :)

[@mattduchene66]

Despite the simple explanations, these videos continuously make me doubt my mathematical abilities for about 15 minutes. But without fail, there’ll be a DOUBLE BAM! out of left field and suddenly everything’s clear in my head. Thank you! You’re doing God’s work.

[@statquest]

Bam! :)

[@ericchao3017]

Really loving these videos, thank you so much for your work Josh

[@statquest]

Thank you!

[@boxiangwang]

Mega BAMM!! I really love the explanation. Awesome!

[@statquest]

Thanks!

[@vusalaalakbarova7378]

Thanks Josh for these videos, I passed my data mining exam by watching your videos, now preparing for the ML exam. Your explanation is brilliant, I learn topics of 3 lectures with these 18 minutes videos. Please continue to publish such valuable content, you save lives of many people like me.

[@statquest]

Thank you and good luck with your exam! Let me know how it goes.

[@vusalaalakbarova7378]

@@statquest Josh, are you planning to make a video about batch normalization?

[@statquest]

@@vusalaalakbarova7378 Not soon. Currently I'm working on a series of videos about how to build neural networks with pytorch and pytorch_lightning.

[@mortyk182]

woah this was some amazing teaching skills sir, you're totally gifted with that

[@statquest]

Thanks! 😃

[@anshuljain2258]

Such hard work. Thank you Josh, you are helping generations with this + all your videos. Step by step learning with examples is the right way to learn anything !

[@statquest]

Thank you!

[@georgeshibley9529]

One of these days I'd love to see you do a NN to watch the process you produced on these videos get lined up with some code, maybe python or R. It's incredible work you do, hell you are helping me survive my masters program. If you put it up, I'd trust the content. Thank you for all your hard work

[@statquest]

Thank you! And good luck with your masters degree.

[@vladimirfokow6420]

Thank you for your clear explanations with the simple example! Great work, and very useful.

[@statquest]

Glad it was helpful!

[@wliw3034]

You are One of the Best Content Creator I have ever Seen.

[@statquest]

Wow, thanks!

[@mohammadhaji2191]

That was the best explanation I had ever seen. Thank you very much.

[@statquest]

Thank you! :)

[@DharmendraKumar-DS]

How the heck do you have this much understanding in each concept...you are irreplaceable.

[@statquest]

Thanks!

[@maayanmagenheim441]

I'm a student for CS at the Hebrew University of Jerusalem, study right now IML course. Your lectures so help me and my friends, and I really want to thank you. You're a great & funny teacher and your lessons are a perfect example to how to teach at the 21 century. Tnx again

[@statquest]

Wow! Thank you very much! BAM! :)

[@rohanmishra3115]

What a great explanation to such complex topic. I can't imagine the amount of effort you put in to create such detailed videos along with spoken text. One of the best youtube channel I have ever come across ! Hats off to you .. Don't BAM me :)

[@statquest]

Wow, thank you!

[@joserobertopacheco298]

I agree 100%

[@KayYesYouTuber]

This is simply beautiful!. You are the best.

[@statquest]

Thank you!

[@ayushipal7605]

Hats off to you Josh!! So nicely explained ❤

[@statquest]

Glad you liked it!

[@tagoreji2143]

A Brief Indepth Explanation.Thank you Sir

[@statquest]

Glad you liked it

[@girmazewdie8366]

Thank you so much for sharing your knowledge, it is really so increadibly helped me understand the basics of the NN.

[@statquest]

Glad it was helpful!

[@anashaat95]

Great explanation as usual. Thank you very much.

[@statquest]

Thanks again!

[@Tapsthequant]

You make this stuff so accessible, well done!

[@statquest]

Thank you!

[@AnBru]

amazing video, thanks for all your hard work on this.

[@statquest]

Glad you enjoyed it!

[@alinadi9427]

this playlist is excellent

[@statquest]

Thank you!

[@amarnathmishra8697]

Well you actually make complex things super easy.Hats off and of course BAAA...M!!!

[@statquest]

Bam! :)

[@user-np8mg5yg9n]

I am on vacation in Hawaii but I am watching your neural network video. This video is so entertaining to watch :) Tai

[@statquest]

BAM! Have a great vacation! :)

[@user-np8mg5yg9n]

@@statquest thank you! you too. have a nice day

[@edphi]

Thanks. Great video again and again.

[@statquest]

Thank you very much! :)

[@nidakhan1412]

thank you so much sir for clearly explaining everything

[@statquest]

Thanks!

[@snp27182]

You're a legend Doctor Starmer.

[@statquest]

Thanks!

[@user-zj5os2zl8w]

BAM! Great work!

[@statquest]

Thank you!

[@quantummusic2322]

I love you Statquest

[@statquest]

:)

[@ilducedimas]

God bless this Good Man.

[@statquest]

Thanks!

[@Viezieg]

thank you so much for these videos. i hated math back in high school, but now in my mid 20's i would rather do math than play video games. all thanks to your tutorials

[@statquest]

Wow! That's awesome! Thank you!

[@emkahuda776]

As usual, your videos are totally awesome, I like them much and easy to understand. I wonder if you will make a video about spatial transcriptomic analysis please since you uploaded the scRNA three years ago considering the spatial analysis is now more famous?

[@statquest]

I'll keep it in mind.

[@cairoliu5076]

great content!

[@statquest]

Thanks!

[@starkarabil9260]

that was exactly what I needed. It would be great if you could 'also' do an application through one of Python libraries in order to show a real application by scripting with using this knowledge.

[@statquest]

Thanks! I would like to do that.

[@samuelpolontalo6882]

Best channel ever

[@statquest]

Wow! Thank you! :)

[@omkarghadge8432]

YOU ARE THE BEST!

[@statquest]

Thanks!

[@gf1987]

very informative ty

[@statquest]

:)

[@abhijeetmhatre9754]

This is just awesome. I had started learning machine learning algorithm from multiple sources until I found your youtube channel. And now I don't have to check for any other source for understanding any ML algorithm. Looking Forward for more deep learning videos as my area of interest is deep learning. Could you help me with a good book for deep learning? And thanks for such wonderful videos.

[@statquest]

This series ends (for now) with Convolutional Neural Networks, so just keep watching to learn about deep learning.

[@hamidfazli6936]

You are amazing!

[@statquest]

Wow, thank you!

[@Ruhgtfo]

Yeaaaa finally new episodde

[@statquest]

:)

[@parijatkumar6866]

Hey Josh, great video as always!! Can you also please point to some source with examples (with answers) which we can practice on our own? I know there are tons of them on internet, but you know, your selection will be really helpful as always!!

[@statquest]

I don't have anything yet, but I will create a "how to do neural networks" video soon.

[@mikhailbaalberith]

Hey Josh, this is dope. Hope you could do some videos about the Hessian and Jacobian matrices, Thanks.

[@statquest]

I'll keep those topics in mind.

[@bhagyeshgaikwad6964]

BAM! that was good!

[@statquest]

Thanks!

[@jordiwang]

AMAZING BROOOOOOOOO

[@statquest]

Thanks!

[@SM-xn9bv]

I can not thank you enough!

[@statquest]

Thanks!

[@fndpires]

THIS MAN IS AN ANGEL! :D QUADRUPLE BAM!

[@statquest]

Thank you! :)

[@robertdavis2855]

I love you man! You have a sense of humor about you that is rare in deez parts lol

[@statquest]

Thank you!

[@killer-whale864]

i hate stats, and i hate statquest. But i keep finding myself on this channel again and again

[@statquest]

noted

[@sattanathasiva8080]

Many many thanks for your videos.

[@statquest]

Glad you like them!

[@lisun7158]

[Notes] 6:44 Notation for activation functions 2:50 Initialize weights using standard normal distribution. Q: Why N(0,1)? -- A: Just one of many ways to initialize weights. [ref. 9:50 of kzbin.info/www/bejne/fXy9oIJ-jayWgtE&ab_channel=StatQuestwithJoshStarmer] Initialize bias with 0 since bias terms frequently start from 0. 4:33 4:48 plot SSR with respect to b3

[@statquest]

bam!

[@soraf583]

Thanks for your great video as always! I have a question though after watching this video and the other SGD video you've made in the past. When calculating the gradients for each parameter with regular gradient descent, we are plugging in all of the samples into the derivative of the loss function w.r.t the current parameter; versus we will just randomly pick one sample in the same process with SGD being used. If that's the case, then what will be the purpose of looping through all the samples (with regular GD) in a complete epoch if we are already using all the samples when calculating the gradients? Thanks in advance!

[@statquest]

I'm not sure I fully understand your question. The difference between "regular" and "stochastic" gradient descent in this context has to do with the summation. In "regular", the summation goes from 1 to 'n', where 'n' is the number of samples. In "stochastic", the summation goes from 1 to m, were 'm' is < 'n' and is the number of samples randomly selected for the iteration. Does that help?

[@soraf583]

@@statquest Thank you for the quick reply! Yes that’s helpful and I think I’m understanding that part. I was mixing the concept of Gradient Descent with epoch/batch numbers, but I guess whether the GD is stochastic or not has nothing to do with the general epoch/batching concept when running a neural network, as we would still need to go over all the samples in a full epoch.

[@salihylmaz4694]

So underrated

[@statquest]

Glad you think so! :)

[@roberthuff3122]

The nested chain rule.

[@statquest]

:)

[@tinacole1450]

Hi Josh! Love the videos. Do you have any posts for building models in R/Rstudio on neural networks? Thanks,Tina

[@statquest]

Not yet!

[@willw4096]

Notes: 2:31 6:14 15:57 the "y"s are calculated based on other weights (w1 and w2)

[@statquest]

:)

[@harshmittal63]

♥

[@statquest]

:)

[@danielo6413]

Hi Josh, great video as always. One question, if I want to speak in epoch and batch terms for this video, is it correct to say that this video shows one epoch, which includes one batch that contains all 3 data points we have (Batch Gradient Descent process)? Thanks a lot !!!

[@statquest]

Yes, that is correct.

[@highelojungler]

you are insane thank you so much

[@statquest]

:)

[@kousthabkundu1996]

Sir, one question I have. when you said we randomly select w3 and w4 from standard distrib in the first iteration, that is any values from standard distrib table or we select no's w.r.t. given dataset?

[@statquest]

In this example I selected random value from a standard normal distribution. This is a normal distribution with mean = 0 and standard deviation = 1 and is completely independent of the data.

[@elmoreglidingclub3030]

Great video and explanation. But I'm missing something simple. The blue and orange lines are added to render the green line, right? It appears (I'm squinting) that, after convergence, the middle dose (the 1/2 dose; actually, just to the left of it) value is 1 but the intersection of the blue and orange lines is at about -.5. Adding those together gives -1, not 1. What am I missing??

[@statquest]

You forgot to add the bias term.

[@gero8049]

Im gonna make a AI agent that create youtube bots that promotes your channel. You really deserve all kudos.

[@statquest]

Bam!

[@dodosadventures7593]

Hi Josh ! Love your videos, could you please explain why normal distribution is used to initialise w3 and w4 or else if you have already uploaded a video on normal distribution can you tag it

[@statquest]

It's just a standard way to do it. However, you can use uniform distributions or other distributions if you would like. One thing people like about the normal distribution is that changing the standard deviation for each hidden layer can make it easier to train deeper models (models with lots of hidden layers).

[@akaBryan]

Hey just a question! Around 14:00, why are you taking the derivative of SSR with respect to w_3 and w_4 rather than y_2,i and y,1_i? What is the logic between choosing taking the derivative with respect to the weight rather than the functions themselves?

[@akaBryan]

Ah nevermind, its because you want to optimize the weights w_3 and w_4, so you just take their derivative to get step size and so forth... im so dumb haha! Im assuming that in the next part then you will optimize the weights w_1 and w_2 by also connecting them to the derivative of the loss function with respect to the weights, so itll be a huge bonkers chain rule in action

[@statquest]

Yes! It will be totally bonkers with chain rule action. :)

[@debadridutta]

The God! Please do NLP also

[@statquest]

Noted

[@NoNonsense_01]

I think for the sake of clarity and rigour, it should be noted that all of the differentials are partial. Otherwise, some people may wonder why implicit differentiation wasn't used in such cases where W2 was differentiated with respect to W1 or vice versa.

[@statquest]

noted

[@_epe2590]

BAM!! I finally understand but.... Am I correct to say that if I was optimizing 3 weights and biases at the same time i would do gradient descent in a function with 3 dimensions (1 for each weight and bias)??

[@statquest]

Yes

[@chicagogirl9862]

OMGGGGG, Is that you who sings at "big bang theory", S12, E24???!!!!!

[@statquest]

I wish! :)

[@GuidedTrading_]

basically, taking derivatives of losses with respect to unknown terms to find how quickly the loss is changing if we change the parameters is the essence of this whole Machine learning thing.

[@statquest]

yep

[@madghostek3026]

Small question: since we fiddle with all (or part) of the parameters at once, and for example bias is dependent on weights on the graph, does that mean they fight with each other? Can something be done about it? Like we calculate the derivatives for current forward pass, ok, but then changing all parameters at once to what the think is optimal might throw off everything, since they can't communicate in any way, how does it not explode?

[@statquest]

In my video on gradient descent, I show how to optimize two parameters at the same here: kzbin.info/www/bejne/qXXZZZlqqJeGeJo In that video, we're trying to fit a straight line to some data points and are using gradient descent to find the best values for two parameters, the y-axis intercept and the slope. If you watch, you'll see a fancy graph, where one axis represents different values for the y-axis intercept and another axis represents different values for the slope. When we optimize both at the same time, we take a step towards a better intercept on that axis and take a step towards a better slope on that axis, which is different, and doesn't affect the one the intercept is on. So the parameters don't fight each other because each one gets its own axis to work on. That being said, we can still get stuck in a local minimum, but it's like progress in one parameter can be negated by progress in another.

[@madghostek3026]

@@statquest Ah, this makes a lot of sense now, I think I know why it was misleading for me - in the end all you see a numerical value, the error, but behind the scenes the partial derivatives take apart the loss function in their own domains, so it's not just one number. Thank you for very descriptive response!

[@statquest]

@@madghostek3026 bam! Your question is actually a very good one and maybe one day I'll make a short video that explains it for everyone.

[@creativeo91]

Please make a tutorial on Gaussian mixture model and EM algorithm

[@statquest]

I'll keep that in mind.

[@creativeo91]

@@statquest thanks.. It will be really helpful 🙂

[@puppergump4117]

13:35 Do you mean the derivative of observed - predicted? Wouldn't that be a derivative of a single number? Or does it always just come out to be -1?

[@statquest]

To get a better understanding of how we determine this derivative, check out the StatQuest on The Chain Rule: kzbin.info/www/bejne/rZ2Unqyup9mEfrM It will explain exactly where that -1 comes from.

[@puppergump4117]

@@statquest Oh the derivative of the negative intercept? ok thanks

[@alexfeng75]

In "d SSR/ d Predicted", is Predicted a single value like Predictedi (with index i ) or a collection of values as i can range from 1 to 3?

[@statquest]

A collection of values. You can tell if you keep watching the video and see how it is used.

[@alexfeng75]

@@statquest thank you for the prompt reply, Josh! you are the best!

[@dahirou_harden]

Just wanted to clarify. Is the output given at the end of each pass an actual function or just a set of 3 points (summed from y1 and y2)? Thanks!

[@statquest]

What time point, minutes and seconds, are you asking about?

[@dahirou_harden]

@@statquest Basically I'm just confused about if the final curve approximating the 3 points is a "curve" as in a polynomial, or just a set of 3 points. Because when we add the two activation functions, you talked about adding them at each point as if we were adding the equations for the lines themselves, in order to get the final line. But it seems like instead we're just adding the y values at each input (the 3 given inputs) rather than a line itself..?

[@dahirou_harden]

@@statquest At 4:03 for example.

[@statquest]

@@dahirou_harden The adding is done for all possible x-axis coordinates (or input values), and thus, we are adding the lines themselves, not just the 3 points. The points (or circles) on the lines are just to illustrate the concept of adding y-axis values, and do not to limit the adding to just those points.

[@shubhamkumar-nw1ui]

My regards to the friendly folks of the genetics department of University of North Carolina at Chapel Hill

[@statquest]

Thanks!

[@84mchel]

Dw_3 = (observed-predicted) * y1. The output is also a softplus activation. Why isn’t this derivative in the chainrule? Thank you!

[@statquest]

We include the derivative of the SoftPlus activation function in the next video (part 2), when we optimize all of the weights and biases, including the ones to the left of the activation functions: kzbin.info/www/bejne/fXy9oIJ-jayWgtE

[@akshaynn4651]

when i plug the value -1.43 into the equation log(1 + e**x) i get 0.093. should I use the base 10 for log or a different one?

[@statquest]

In statistics, data science, machine learning and almost all programming languages, the default base for the log function is 'e', and that's what I use here.

[@akshaynn4651]

@@statquest Thanks, this was very helpful.

[@kiranchowdary8100]

i dont know how u feel but i feel very bad that your are not recognized as much as you should be (from india )

[@statquest]

Thanks!

[@kiranchowdary8100]

@@statquest urs is the first channel I purchased member ship and God I am the most conjuze(resistant. In spending money)person I ever saw

[@statquest]

@@kiranchowdary8100 WOW! That is awesome!!! Thank you so much for your support. It means a lot to me that you care enough to contribute.

[@zer995]

Triple BAM!!! That's what I said when I knew my girl, married her and got children :)

[@statquest]

:)

[@cagatayerturk3595]

[@statquest]

:)

[@arnavm2003]

BAM!!! Reply to every comment!!

[@statquest]

:)

[@rafibasha1840]

@4:45 ,Hi Josh why sum of squares residual used classification problem

[@statquest]

Because it works just fine in this simple example. However, if you keep watching the series, you'll see how to do backpropagation with ArgMax and SoftMax and Cross Entropy. Here's the whole playlist: kzbin.info/www/bejne/eaKyl5xqZrGZetk

[@rafibasha1840]

@@statquest ,Thank you Josh …I am watching your videos daily …Please make videos on RNN GAN LSTM and NLP ..

[@statquest]

@@rafibasha1840 I plan on making those in the spring.

[@rafibasha1840]

@@statquest ,Thank you Josh

[@giorgosmaragkopoulos9110]

So what is the clever part of back prop? Why does it have a special name and it isn't just called "gradient estimation"? How does it save time? It looks like it just calculates all derivatives one by one

[@statquest]

Backpropagation refers to how the gradient is calculated. Gradient Descent is how the gradient is used.

[@axolotl6814]

soft plus

[@statquest]

:)

[@xianchen1935]

17:08

[@statquest]

bam! :)

[@macknightxu2199]

Hi, how to understand back? not forward or other direction? I mean the video is nice, but didn't explain to clear why backward is important. Why not forward?

[@macknightxu2199]

got it. At the back point, the derivative is much simpler than the derivatives at the front. So, as we would like to go from simple to hard, we'd choose from back to front. That's why it's backpropagation, which is discussed in the next video. BR

[@statquest]

bam! :)

[@karrde666666]

The right way to learn, textbooks and lectures should be obsolete

[@statquest]

bam! :)

[@396me]

If there are only 3 points in the inputs, how it’s possible to get 5 points for getting orange or blue curve😢, please help me to understand

[@statquest]

What time point, minutes and seconds, are you asking about?

[@396me]

@@statquest 11:13

[@statquest]

@@396me Since the range of possible input values goes from 0 to 1, we can just plug in numbers, from 0 to 1, to see the shape of the curve that the neural network is using for this dataset.