Backpropagation explained | Part 1 - The intuition

Let's discuss backpropagation and what its role is in the training process of a neural network. We're going to start out by first going over a quick recap of some of the points about Stochastic Gradient Descent that we learned in previous videos. Then, we're going to talk about where backpropagation comes into the picture, and we'll then spend the majority of our time discussing the intuition behind what backpropagation is actually doing.
We're going to be building on concepts that we covered in a couple of previous videos. These two videos covered what it means to train an artificial neural network, and how a network learns. So if you haven't seen those yet, go ahead and check them out now, and then come back to this video once you finish up there:
• Training a Neural Netw...
• How a Neural Network L...
🕒🦎 VIDEO SECTIONS 🦎🕒
00:00 Welcome to DEEPLIZARD - Go to deeplizard.com for learning resources
00:20 Agenda
01:20 Recap stochastic gradient descent (SGD)
04:40 Backpropagation intuition
08:28 Backpropagation summary
10:19 Collective Intelligence and the DEEPLIZARD HIVEMIND
💥🦎 DEEPLIZARD COMMUNITY RESOURCES 🦎💥
👋 Hey, we're Chris and Mandy, the creators of deeplizard!
👉 Check out the website for more learning material:
🔗 deeplizard.com
💻 ENROLL TO GET DOWNLOAD ACCESS TO CODE FILES
🔗 deeplizard.com/resources
🧠 Support collective intelligence, join the deeplizard hivemind:
🔗 deeplizard.com/hivemind
🧠 Use code DEEPLIZARD at checkout to receive 15% off your first Neurohacker order
👉 Use your receipt from Neurohacker to get a discount on deeplizard courses
🔗 neurohacker.com/shop?rfsn=648...
👀 CHECK OUT OUR VLOG:
🔗 / deeplizardvlog
❤️🦎 Special thanks to the following polymaths of the deeplizard hivemind:
Tammy
Mano Prime
Ling Li
🚀 Boost collective intelligence by sharing this video on social media!
👀 Follow deeplizard:
Our vlog: / deeplizardvlog
Facebook: / deeplizard
Instagram: / deeplizard
Twitter: / deeplizard
Patreon: / deeplizard
KZbin: / deeplizard
🎓 Deep Learning with deeplizard:
Deep Learning Dictionary - deeplizard.com/course/ddcpailzrd
Deep Learning Fundamentals - deeplizard.com/course/dlcpailzrd
Learn TensorFlow - deeplizard.com/course/tfcpailzrd
Learn PyTorch - deeplizard.com/course/ptcpailzrd
Natural Language Processing - deeplizard.com/course/txtcpai...
Reinforcement Learning - deeplizard.com/course/rlcpailzrd
Generative Adversarial Networks - deeplizard.com/course/gacpailzrd
🎓 Other Courses:
DL Fundamentals Classic - deeplizard.com/learn/video/gZ...
Deep Learning Deployment - deeplizard.com/learn/video/SI...
Data Science - deeplizard.com/learn/video/d1...
Trading - deeplizard.com/learn/video/Zp...
🛒 Check out products deeplizard recommends on Amazon:
🔗 amazon.com/shop/deeplizard
🎵 deeplizard uses music by Kevin MacLeod
🔗 / @incompetech_kmac
❤️ Please use the knowledge gained from deeplizard content for good, not evil.

[@danluba]

This video deserves to be on television at dinner time.

[@RaghavendraBoralli]

Yes i want to know on the math behind backprop

[@kareemjeiroudi1964]

Me too!

[@piyushchauhan343]

Jan ke kia karo ge

[@husseinwassouf2743]

this course is so nice you are simplifying complex ideas in a really nice way, I think this series should be binge watched like a netflix series to refresh your memory on all these concepts XD

[@maedehzarvandi3773]

your videos are actually great in expressing the concepts in possibly quickest time . i havnt seen your other playlists yet. but this one really helped me so much.

[@tymothylim6550]

Thank you very much for this video! It was fantastic that I could finally understand how SGD uses back propagation to calculate the gradients to minus off!

[@yzyangliang]

This is an amazing video and I understand the deep learning instantly!

[@parthasarathybalaraman1800]

I have just completed watching all the 27 videos in the list. All are excellent videos!! Thanks for uploading. Please continue the good work!! Thanks!

[@deeplizard]

Hey Parthasarathy - That's great to hear that you've gone through the entire playlist! I'm glad you're enjoying them!

[@nicholasscotto3712]

after watching sooooooo many videos on this topic yours was EASILY the most helpful by far. It is accurate, simple, and just all around perfect. the image is very simple. I love your voice and the explanation is so good.

[@RohanPaul-AI]

Just can not THANK YOU enough for this Greatest of videos on the topic. Just Brilliant.

[@hkrishnan91]

Bless you for making the world a better place. Keep up the good work!

[@javierCi]

Thanks you very much for making this great videos. And I want to know more about maths

[@travel7517]

great..explaination...simple and effective

[@qusayhamad7243]

thank you very much for this clear and helpful explanation. Words fail to express my gratitude.

[@MsStoCa]

Hey! What's going on everyone? :D Great content!

[@justchill99902]

There is no limit to the awesomeness of her explanation!

[@luna-kr3zc]

there's a whole team behind this she is just the voice (but she does a great job at it)

[@ThePRINCEBARPAGA]

I cannot thank you enough. I don't know how many videos I watched to clear this concept, nothing helped but your video. You explained it in a very detailed and clear way. Thank You so much!

[@flamboyanz]

Thank you so much for this! Its concise and well explained.

[@deeplizard]

You're very welcome, Parvez!

[@TheMeltone1]

You are truly a fantastic teacher :)

[@zw9423]

Omg Thank You! All these books make it so complex and include a bunch of unnecessary equations before a noob like me can actually understand the process. I am interested in the math as well but first I have to know the concept, otherwise I often find myself understanding these complex details and be like, why exactly are we doing this? This video actually makes sense and now I can dive deeper into it. Nice animations too. Props to you guys

[@deeplizard]

You're welcome, Zi! Glad you found it helpful. The math explanations are in the following videos of this playlist.

[@rajuthapa9005]

yes math too please.

[@samaryadav7208]

Wow that was exciting Hi. Thanks for sharing these videos. And Also we want to see the maths.

[@richarda1630]

such an awesome super simple explanation! and how you bring up the math, it's almost like a teaser :) now I want to see the Math! =)

[@DM-py7pj]

These are excellent videos. I do worry about biases not being discussed in later videos as these are also being updated.

[@robertoooooooooo]

Amazing, love your videos, will recommend it to all my friends

[@deeplizard]

Thank you, Robert!

[@Arcaerus]

I would greatly appreciate if you could make a series on how to take the derivative of a bunch of layers! Thank you for these videos!

[@loganmay2105]

Never thought I'd hear the words "chain rule" outside of my old high school AP calc class

[@deeplizard]

🤓

[@deepaksingh9318]

Yess.. Adding a specific Playlist with Maths behind all the functionality with an example (Like uh showed for max pooling which was visible to see hoe things are working with input) would be really great.. Bur still appreciate your efforts in helping students by such a great videos..

[@deeplizard]

Thanks, deepak. The next four videos in this playlist show the full math behind backpropagation. Let me know how it goes as you progress through them!

[@ismailelabbassi7150]

YES YES YES YES YES YES WE NEED MATH TOPICS

[@paragjp]

Hi, Thanks for super clean way to explaining basic concepts. Keep it up. Request you for following video series 1. Complete Maths and Stats for ML/Deep Learning. 2. Pl have additional series on calculus on backpropogation. 3. Make complete new series on ML/Deep Learning Practical projects rather than Housing Price Prediction, Titanic, iris, Hand written digits ....etc

[@mohamedmahdy969]

I already know the math behind back propagation; however, I will watch your videos in order to see how you are going to present it. Your way of giving the information is awesome. I want to see how it will work with the math complexity.

[@deeplizard]

Thanks, Mohamed! Let me know your thoughts after you finish the following math videos.

[@mohamedmahdy969]

@@deeplizard To be honest, as I mentioned earlier, I already know the math behind the back propagation, yet your videos were a good refresh to me. You used the same mathematical notation and the same methodology of my teacher. he started with the last hidden layer; after that, he generalized to any hidden layer. Thanks a lot for your videos.I already finished the math videos and I will finish this list today. In the future, I am going to watch tensorFlow.js series

[@deeplizard]

Great to hear, Mohamed! Thanks for letting me know. Would love to hear how your progression is going in the TensorFlow.js series as well once you start!

[@1sankey2]

Great video I wanna see the math. Thank you for uploading this one.

[@deeplizard]

Great :D It's in the following episodes. Starting with this one: deeplizard.com/learn/video/2mSysRx-1c0

[@VISWESWARAN1998]

Thank you so much

[@cedricrogers8102]

This was a very clear explanation of what happens during back propagation. The next step is to provide the math. Thank you for your effort. Great Job.

[@deeplizard]

Thanks, cedric! The full math for backprop is covered in the videos following this one! :) They are currently #28 - 31 in this playlist: kzbin.info/aero/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU

[@cedricrogers8102]

deeplizard I will watch them thanks again

[@vantongerent]

great video - yes I want to see the math! 🙂

[@faraazmohammed3693]

please make a series on videos explaining math behind machine learning and deep learning in particular. Thanks for Amazing videos.

[@deeplizard]

Hey Faraaz - You're welcome! Did you see the next four videos in the playlist that cover the full math for backprop?

[@xeroxSoldier]

Love your videos, really helping me with my master thesis! :) You're talking about SGD but there are also other optimizers. I'm interested especially in Adam, and how it differs from SGD, maybe you got some paper or article recommondations? - In my understanding Adam is doing the exact same thing as SGD just using another algorithm, is that correct? - You're talking about backpropagation referring to SGD, is Adam also using backpropagation? In my understanding, backpropagation ist just the general term of changing the weights after every forward propagation.

[@GirlKnowsTech]

00:00 Intro 01:23 Recap stochastic gradient descent (SGD) 04:42 What is backpropagation, the intuition 08:28 Summary

[@deeplizard]

Added to the description. Thanks so much!

[@GelsYT]

Hello :D Is the output of the activation equal to the weight ? i mean is the activation the weight? of every neuron firing? right? cause i think that's what I remember about your video on activation function so basically the output of the activation function is going to be the weight? thank you so much great tutorial ! you explain it at the simplest way :))

[@rohtashbeniwal9202]

great videos,,saveed,subscribed ,

[@deeplizard]

Awesome, thank you!

[@krishnasaibiradar889]

Thanks for the awesome video on backpropagation Yes i want to know the math behind backpropagation as the whole logic lies in and around math.So kindly make a video on the math behind the Backpropagation.

[@deeplizard]

You're welcome, krishnasai! There are 4 episodes following this one that explain the math :) And they all have corresponding blog articles along with the video! The next one starts at the link below, and the following 3 are directly after. deeplizard.com/learn/video/2mSysRx-1c0

[@nourelislam8565]

Amazing... Before asking about the math courses for the backpropagation. Is it essential to study the math behind deep learning in order to work with Keras or any other APIs? Thanks for these fruitful videos!

[@sanketkachole9639]

yes we need it

[@deeplizard]

You got it - Math starts in the next episode :D kzbin.info/www/bejne/aJ62qqaIrZJkmZI

[@mjain2172]

Yes, please explain math behind of this calculation, thanks for explaining deep neural network concepts

[@deeplizard]

Great :D The math is included in the following episodes. Starting with this one: deeplizard.com/learn/video/2mSysRx-1c0

[@omargonalfa]

Awesome video. please a brief example of the math behind.

[@deeplizard]

The math is shown in the following four videos of the series :D

[@deeplizard]

Backpropagation explained | Part 1 - The intuition kzbin.info/www/bejne/jnaWnKWcaKiEotU Backpropagation explained | Part 2 - The mathematical notation kzbin.info/www/bejne/aJ62qqaIrZJkmZI Backpropagation explained | Part 3 - Mathematical observations kzbin.info/www/bejne/fWbFZZ2Id7CBrtk Backpropagation explained | Part 4 - Calculating the gradient kzbin.info/www/bejne/kKOYp5x3j6yhmqc Backpropagation explained | Part 5 - What puts the “back” in backprop? kzbin.info/www/bejne/rnTPfJKVeNaNpLM Note, at 7:44, I misspoke when I stated that the updated values we get for the weights are the the corresponding derivatives of the loss function with respect to each weight. Actually, the updated values themselves are *not* the derivatives. Rather, after calculating the derivatives, the weights are updated to their new values, which are calculated *using* the derivatives we obtain. This process of updating the weights is covered in more detail in the following video. This particular detail is mentioned at 1:26: kzbin.info/www/bejne/lX-YnKOJgqmZatEm26s Machine Learning / Deep Learning Fundamentals playlist: kzbin.info/aero/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU Keras Machine Learning / Deep Learning Tutorial playlist: kzbin.info/aero/PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL

[@jabrams3643]

i'm confused about the difference between the gradient and derivative? Is it called a gradient of the loss function when you average out all of the derivatives of the loss wrt to a certain weight? Meaning that its called a derivative when its just dloss/dweight for a single sample, and its called a gradient when its the average dloss/dweight for all of the samples? thank you for this amazing series btw it is the best on the internet

[@dan7582]

At 8:00 you say that the update values of the weights are equal to their derivatives (gradient). But since we are talking about a loss function, they should be equal to de NEGATIVE gradient instead?

[@krishnaik06]

Math too please

[@avibcci1297]

interesting

[@fritz-c]

I noticed a broken link in the article for this video, near the words: "shown in an [earlier post]." It looks like part of the url got duplicated.

[@deeplizard]

Fixed, thanks Chris! :D

[@GelsYT]

sooo gradient descent is the one who's updating the weights? thank you soo much

[@deeplizard]

Yes it is!

[@generalzeedot]

would this backpropagation technique, therefore, only be usable in supervised learning?

[@abail7010]

Something isn't clear tome. Do we update the bias for each neuron or is this example without using bias? Apart from that, I love your videos and the way you explain things! Please keep up since this is a highly interesting thing. :)

[@deeplizard]

Thank you, Xi! I've left bias out of this example, but they get updated at the same time and in the same fashion as the weights.

[@abail7010]

Thank you for the reply! I've not seen your video to the bias which cleared my question when I commented. :)

[@jesilmohammed7926]

What is the default height and width of a conv_2d filter ?

[@stydras3380]

I can only agree with the other comments, knowing the math is neat :D Is there some sort of follow up video of yours?

[@stydras3380]

Ok, I have found the videos discussing the math :D

[@aamir122a]

same here.

[@richardme9928]

Any links to videos that do the complete math calculations ?

[@deeplizard]

Hey richard - They are the following videos in this playlist. Check out the full series in order here: deeplizard.com/learn/playlist/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU

[@fahimmahmud3115]

{ "question": "Since the derivative of the loss function is calculated with respect to the weights of the model during backpropagation, what characteristics should the loss function have?", "choices": [ "Continuous", "Discontinuous", "Constant", "Zero" ], "answer": "Continuous", "creator": "Fahim Mahmud", "creationDate": "2020-02-12T04:10:35.701Z" }

[@deeplizard]

Thanks, Fahim! Just added your question to deeplizard.com/learn/video/XE3krf3CQls :)

[@meetayan15]

hi, do you have any lecture on SGD ?

[@deeplizard]

Hey Ayan - The following two videos (in the order listed) talk about SGD: 1. kzbin.info/www/bejne/qYuknYZplMmhoZI 2. kzbin.info/www/bejne/lX-YnKOJgqmZatE Note that SGD uses backpropagation during training, which is where most of the "grunt work" comes into play. So after generally understanding what SGD is doing from the above videos, this 5-part backprop series, starting with the video we're currently commenting on, gives all the details for what backprop is doing during the training process.

[@herohari27]

In which video you were talking about SGD?

[@deeplizard]

Hey Hari - These two: deeplizard.com/learn/video/sZAlS3_dnk0 deeplizard.com/learn/video/_N5kpSMDf4o

[@torbjornstorli2880]

Is the backpropagation being applied only once per epoch? So, if you have 20 epochs you will perform backpropagation 20 times, once per epoch ?

[@deeplizard]

Hey Torbjorn - It occurs at each batch. The details for this implementation are covered in the backprop videos that come after this one in the playlist.

[@rajuthapa9005]

r u bringing RNN tut too?

[@deeplizard]

Hey Raju - Yes, I have RNNs on my list to cover in future videos.

[@aliasgarzakir4779]

Yes please, math is fun.

[@deeplizard]

All the math is in the episodes following this one :D

[@torbjornstorli2880]

That is per batch per epoch

[@zrmsraggot]

Why cant we just change the weights from pink to blue nodes since we just want activation in blue nodes to change to be efficient

[@ojasvinnagpal9472]

I have a similar question: Why can't we simply change the weights between the blue and the yellow nodes and arrive at the optimal values for the hidden layer units?

[@ojasvinnagpal9472]

Is it the case that in order to change the nodes in each layer, we need to change BOTH the nodes in the previous layer as well as the weights between the previous layer and the current layer?

[@ahdm1319]

please explain the maths behind this in seperate videos

[@deeplizard]

The math is shown in the next four videos after this one. Let me know what you think!

[@saranshtayal2526]

i want to know about the math

[@deeplizard]

It's in the following episodes after this one :)

[@josegregorioperezmagallane3211]

We want to Sep the math

[@deeplizard]

The math explanation is in the following four videos of the series :D

[@danluba]

And yes, I could go for some math.

[@deeplizard]

The math for backprop starts in the next video in the playlist! Here are the full details for the backprop series: Backpropagation explained | Part 1 - The intuition (this video) kzbin.info/www/bejne/jnaWnKWcaKiEotU Backpropagation explained | Part 2 - The mathematical notation kzbin.info/www/bejne/aJ62qqaIrZJkmZI Backpropagation explained | Part 3 - Mathematical observations kzbin.info/www/bejne/fWbFZZ2Id7CBrtk Backpropagation explained | Part 4 - Calculating the gradient kzbin.info/www/bejne/kKOYp5x3j6yhmqc Backpropagation explained | Part 5 - What puts the “back” in backprop? kzbin.info/www/bejne/rnTPfJKVeNaNpLM

[@danluba]

Yeah - I found it. Awesome stuff. Thank you!

[@deeplizard]

Awesome, you're welcome!

[@urvashidang6083]

i want to know the maths behind

[@deeplizard]

Great! It's in the following episodes. Starting with this one: deeplizard.com/learn/video/2mSysRx-1c0

[@wesleyrademaker5167]

math please!

[@deeplizard]

Hey Wesley - The math is in the following four videos after this one in the playlist!

[@SnoopyDoofie]

Anyone with an electrical engineering background would realize that this is nothing more than a negative feedback loop.

[@yvesa6959]

give maths pls

[@deeplizard]

The math comes in the episodes that follow this one :D

[@sprajapati2011]

7:17 'ie' is pronounced as 'that is' not i e itself

[@lando6583]

i don't want to know about the math.

[@deeplizard]

Warning: Don't watch parts 2 - 5 of the backprop videos 😜

[@MrFranciscoooooo]

A video for this a video for that!! Just teach or do a brief sum of what this is, because if I want to search more I will do it from other videos not just yours.

[@akshatsahu2637]

The background is so bad. The image and background blend

[@deeplizard]

I agree the image and the background don't have enough contrast with each other. The background has changed in later videos.

[@akshatsahu2637]

@@deeplizard rest all the videos are very very good. The way everything is explained is awesome!. Better that course era. One last things, are there any videos in which the programming part is explained or the dimensions of the error or Delta, and the dimensions of theta?

[@jesilmohammed7926]

What is the default height and width of a conv_2d filter ?