What is backpropagation really doing? | Chapter 3, Deep learning

What's actually happening to a neural network as it learns?
Help fund future projects: / 3blue1brown
An equally valuable form of support is to simply share some of the videos.
Special thanks to these supporters: 3b1b.co/nn3-thanks
Written/interactive form of this series: www.3blue1brown.com/topics/ne...
And by CrowdFlower: 3b1b.co/crowdflower
Home page: www.3blue1brown.com/
The following video is sort of an appendix to this one. The main goal with the follow-on video is to show the connection between the visual walkthrough here, and the representation of these "nudges" in terms of partial derivatives that you will find when reading about backpropagation in other resources, like Michael Nielsen's book or Chis Olah's blog.
Video timeline:
0:00 - Introduction
0:23 - Recap
3:07 - Intuitive walkthrough example
9:33 - Stochastic gradient descent
12:28 - Final words
Thanks to these viewers for their contributions to translations
Italian: @teobucci
Vietnamese: CyanGuy111

[@rbm0307]

You might think that your videos are fodder for university students boning up on a subject, or mathematicians/engineers in the early stages of their careers - basically that you cater to a younger audience. I'll have to prove you wrong. I'm in my early 60's and have been involved with information technology in some form or fashion my entire career. I enjoy learning; always have. I've viewed many of your videos only because they interest me and have you subscribed on my KZbin account so as to get notifications of updates. I find the topics about which you speak fascinating and am a bit jealous of those university grads today who now have access to this material at their fingertips. I wish with all my heart that I was able access these videos back when I was in university. It would have made life SOOO much easier for me back then. Your pedagogic skills are astounding, demonstrated by your ability to communicate difficult subjects precisely, concisely and simply. The animation format is integral with the presentation, adding to the delivery of the material. I salute you!! Please keep these videos coming.

[@zottasi]

I could not have said it better.

[@johncoleman1930]

That is very inspiring to hear you say I am a student rn, not in university though, but I love learning and hope to continue to foster a love of learning.

[@jinalin9351]

Well said.

[@richardsleep2045]

This 65 yr old ex s/w developer completely concurs!

[@landwarrior348]

And great thing about democratizing this learning is that kids in Africa or Europe or Asia have access to same content to what kids in America have. This makes me wonder if learning becomes easier or easily accessible, what the kids of future will be learning?

[@Niki_0001]

I can't claim to have understood everything from the first watch-through of this series, and I will watch these videos again with pen and paper in hand, but even this first viewing has made neural networks go from pure witchcraft and wizardry to something that actually makes sense in my head. I can't possibly thank you enough for posting these videos.

[@wajdanali1354]

and I thought everybody else here except me is a genius

[@joshdominguez2765]

"...has made neural networks go from pure witchcraft and wizardry to something that actually makes sense in my head" I died coz I've never related to something so much xD D:

[@Dr_Neo_Cortex.uka_uka]

I thought that I was the only one who did it.

[@GabrielCarvv]

I did not understand a single word. Please help me.

[@fillthedao]

@@GabrielCarvv If this video did not explain this to you then nothing will I'm afraid... Don't get me wrong - it's not about you - its a tough subject but this is by far the most approachable presentation I have seen out there. I can't even imagine how one would present it to the fellow human being in a more approachable way... ;-) PS. Have you watched the previous 3 vids from his series?

[@kevinconnolly6450]

This series is totally brilliant. I am 73 years old and used to teach mathematics. I am still learning stuff and with the help of sites like yours it makes it so much easier. Have you thought of doing any videos on the really complex subject of real analysis/. Keep up the good work. Kevin Connolly

[@damian_smith]

Cool! I'm mid forties and about to do a CompSci MSc. Gotta keep my own neurons going!

[@yeimarsoto3196]

​@@damian_smithSame! Im 31 getting my PhD in CS! Hopefully i dont need to be in a retirement home at 35.

[@ThePRASANTHof1994]

Anyone else smiling through all of his videos because you're understanding so much so well like never before?

[@makingmediocremachines2133]

yup

[@jehanbhathena6270]

Yes!!!!!

[@angelbythewings]

yeah man!

[@vladimirbosinceanu5778]

Every single time.

[@pranavprameshgandhi9392]

Hell yeah

[@TheSimonvdp]

I disable adblock for this

[@3blue1brown]

+Simon van der Poel Funny, because I disabled ads for this :)

[@BrianFaure1]

What's the first rule of adblock?

[@BrianFaure1]

TIGuardian - Apologies, I was talking to Simon, the first rule is you never talk about adblock.

[@ntwede]

I tried, but my adblock uses neural networks. It decided it would rather re-enable itself.

[@krakenmetzger]

Bro there's ads in the video

[@TopGunMan]

These visualizations are spot-on. Only a few people in the entire world need to make a great explanations backed by powerful visualizations about a topic - the rest of the world just needs to discover these. So much time wasted by learners trying to locate easily-digestible information, among all the inferior presentation methods out there. Glad to have found one of the best for this topic.

[@user-qj6hl5xb8q]

It is challenging to go through books and lectres from authors and professors trying to sound smart.

[@srirams0071]

Great explanation, your team is awesome. "A drunk man stumbling aimlessly downhill,but taking quick steps" is the best analogy ever for Stochastic gradient descent. :-)

[@youtubeviolatedme7123]

what the deuce?

[@PeterGriffinLovesLois]

stewie

[@ivoryas1696]

​@@youtubeviolatedme7123 Nah, it's actually not that uncommon. My material science professor even used it when discussing some Quantum processes! 😁

[@xanthoptica]

"A drunken sailor walking a plank" is often used to describe fixation of an allele by genetic drift in a small population. So next time you've had one too many, explain that you're feeling a little too stochastic to drive.

[@MySkittlesRainbow]

First time I caught myself having a moment of awe while watching educational content. The production value is incredibly high... the way the connections twinkle and move to represent adjusting the weights, the small animations, the descent into various shapes, how the little arrows indicating the desired change move and change size. Beautifully put together! Thanks a lot!

[@brianevans4]

Your animations are amazing!

[@ShanmeiLiuCS]

How are the animations made? It is great!

[@florianro.9185]

He has built an animation "engine" in python, which you can find on his GitHub

[@ShanmeiLiuCS]

Thanks!

[@thescientist7753]

Brian Evans a

[@chinmaydutta3783]

github.com/3b1b/manim

[@dinub8414]

I find it astonishing how well you convey some of the intricacies here, way better than most of ML practitioners who teach the public, whether as youtubers, online instructors or public speakers who end up on video on the internet. I very much resonate with the way you frame things, the metaphors you choose, your visualizations and of course your evident love for understanding and sharing thereof. Your work is a great gift to all of us - students, engineers, researchers, philosophers, random viewers from all walks of life. Thank you.

[@bikrammajhi3020]

Thank you for what you are doing. I have just started learning ML and I find your video really helpful. Thank you again from bottom of my heart for everything you are doing. I just wanted to let you know that you are really making changes and inspiring youngsters like me.

[@shiladitya7739]

No form of words can express enough what magic you're creating! I don't know how much you actually think you impact us.. but let me tell you Grant, your effect on my life is immeasurable! And the fact that you learnt it the hard way, and made it so simple for us, so that we don't have to go through the same, makes me respect you even more and more every single day.. Thank you so much.. 3B1B is undoubtedly the best channel on KZbin..

[@makebreakrepeat]

I just want to say that I love that you're moving into the mathematics of ML. The visualizations convey the concepts so well!

[@adityashukzy]

Note to myself: Aditya, if you're having trouble understanding, read this. Scroll to 06:17. Listen to what he's saying, "In a sense, the neurons that are firing while seeing a 2, get more strongly linked to those firing when thinking about a two." Now, pause the video and listen. All this is is just a fancy way of saying: when we show our model a picture of a handwritten 2 and we tell it, "hey listen up this thing is a 2" (like in our training set we have labels for our input pictures) and then we find out the activation units that have a say in influencing the hypothesis value of the 2 label, ie, the activation units which can heavily increase or decrease the value of the output unit for the label 2, we tell those activation units "hey guys when you see something that resembles this thing, fire up the 2 label, ie increase the hypothesis value of the label for 2. Basically, we assign those activation units greater weights (or parameters) which influence the hypothesis value of the 2 label more (so that they can actually give us the right answer and say "oh look this is probably a two". I hope this helped and didn't complicate it further. If you don't get it, go over the video a few more times and review Andrew Ng's notes on this in the ML course on the Backpropagation lecture in week 5. Cheers, bro. Love you.

[@hzmuhabbet]

Thanks dude, it really helped. I am trying to follow Ng but after a point it sounds me like an Indian language. Sometimes it might be a bit difficult to visualize it in mind, and Ng is really a bit 'bad choice' about that part. I am not saying that he is a bad teacher but sometimes I feel like I am surfing through the universe...

[@adityashukzy]

@@hzmuhabbet I understand. Ng gets caught up in notation quite a lot so it's difficult to follow his intuition. I actually just noted this down for myself for future reference but I'm glad that you and other people are finding it helpful in understanding! I wish you well on your ML journey!

[@HARIHaran-ks7wp]

I finished the Andrew NG course except for the programming assignment part. I felt his explanation wasn't quite reaching my head and now watching this series has blown me away at how easy it's to understand things when they are visualized. Thanks for sharing your note snippet, pretty cool!

[@nicolasstencel7775]

nice

[@SamuelJFord]

I have just found your videos. I have enjoyed watching science educators on youtube for many years, but these videos are the best examples of complex ideas being explained clearly whilst still being entertaining. I can't believe I had never heard of this channel before.

[@jacqueskirstein9833]

Thank you so much for all the time and effort that you put into making a visual understanding of Neural Networks. I've been trying (admittedly on and off) to get a good intuition on Neural Networks for at least a year now. This is by far the best fundamental video I have come across to get an understanding of what the theory behind the scenes is for. I know that this will increase my (and many others') learning curve by a large amount. Thanks again!

[@mark.fedorov]

Man, the clarity and the animations make your videos masterpieces

[@gmish27]

I've been reading the book for quite some time but your explanations using animations have pushed my understanding to new levels. So many thanks to you. PS: Keep the background music. Holds our concentration for a long time.

[@paskymail]

I had to stop seeing your video for a moment because I needed to write down my admiration for your work. It is outstanding the amount of intelligence, communication skills, careful graphic design and passion in very single frame of your videos. Congratulations and thank you for your contribution to maths education. See you in patreon!

[@donniegoodman8679]

All of the videos shown on this channel have been written so well. Even I can understand, mathematics was a complete mystery for me in school. I have a sixth grade education and I feel really smart after watching one of these programs. Thanks so much. I'm really grateful that you have taken the time to educate the people that have a hard time understanding but still want to learn. What you're doing is just as important as any other volunteer or charity work. I'm excited I found this .

[@kemsekov6331]

I must say it. I am 100% seriously learnt English just to be able watch your videos. This is the content that will help everyone wandering grow stronger in their favourite subjects. I thank you for your work from the bottom of my heart ❤️❤️

[@trevinsmall9069]

All I can say is thank you! Every 3b1b video that I have watched is simply incredible. The chronology of ideas introduced throughout a video makes complex topics easy to follow, and the animations are BEAUTIFUL. I have never been able to visualize math so well before. You are an amazing teacher and have contributed invaluable knowledge to society! I'm still in shock that educational resources this profound are available for free. I truly appreciate the work you are doing!

[@vgdevi5167]

Hello, I'm impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

[@soheilsepahyar1437]

Thank you so much for the simplicity in your explanations. You try to explain everything not by their definition, but with an understandable, simple way. Thank you!

[@cvm7549]

This is the best video on machine learning I have seen so far. Thank you for making this complexity as simple as possible.

[@cameronadams4366]

You are a wizard when it comes to animations and understanding

[@vgdevi5167]

Hello, I'm impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

[@brendawilliams8062]

You would like Quaternions.

[@AishwaryaAR0013]

I've watched these videos 3 times and everytime I watch them, I feel a bit smarter. It starts with understanding little to progressively understanding more and more and finally seeing the big picture. Can I say I've understood everything? No. I am getting there and I'll be coming back for these again. For anyone who's feeling discouraged, I can assure you that you'll get there. Thank you so much for creating quality content that brings the driest, most theoretical concepts to life! You're a hero.

[@vgdevi5167]

Hello, I'm impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

[@TheVilivan]

Thank you for saying this, I was pretty confused by some things in this series and I'm glad to hear that it gets better the more times you watch it. Learning some fundamental stuff like linear algebra and calculus would probably help as well, I really should get onto that

[@thismakesnosense]

​@@vgdevi5167 Hey, this might be a little late but there are some links to learning about Neural networks/Backpropagation in the description if you haven't looked at them already.

[@erikaambrioso8966]

Im taking a Neural networks course at my university right now and I'm finally starting to understand this stuff after watching this video series! Thank you so much!

[@davidjurelius5967]

This series is the best I've seen on neural networks. Thank you for making it. ❤❤❤

[@Kaixo]

You are amazing!! Best explanation I've seen on the internet yet!!

[@vgdevi5167]

Hello, I'm impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

[@lilysu2529]

I love how you enunciate the words as if you are truly interested and into the topic. The art direction and the animations are also on point. Kudos to the animator!

[@juliangonzalez28jg]

Implementing a neural network from scratch for a class project and this series helped me so much to get started. Thanks!!!

[@yuewang2186]

Am constantly in awe of how intuitive this is.

[@saitaro]

Grant, you're a diamond.

[@General12th]

no hes a human

[@-long-]

you must be fun at the party

[@timangar9771]

@@-long- I actually found it funny. Also, THE PARTY sounds dramatic, am I invited?

[@duykhanh7746]

@@timangar9771 you gotta admit that we have weird sense of humor for watching this video for fun

[@user-vt4bz2vl6j]

no he's a pi creature

[@kushh7550]

I never understood neural nets like this before... Thanks!

[@minerscale]

Really well done. I'll probably have to go through the entire series once again, but that's to be expected. There is so much content in so little time. I'll probably actually get into designing a neural network from the ground up.

[@krikkiteer]

your videos are simply amazing.. awesome visualisation of what actually happens...i imagine a huge effort went into making those.... kudos - i almost never comment in that manner... greatest thanks and respect for that!

[@CedricChee]

So far, I think this is the best intuitive intro to backprop I've seen. This channel means a lot to me. Because of its teaching style, I managed to get back to learning and grok maths while I was studying machine learning last year, 13 years since doing my undergrad in CS. It's almost the end of 2017 and I still keep hearing from the people I talked to that they fear/hate maths because they think it's a tough subject to tame. May be this example tell us why our education system (internationally) is still broken? As an aside, David Perkins, in his book "Making Learning Whole" also touch about this widespread diseases of the educational system, namely "elementitis". I think we can do better. Grant is doing great work to lower the barrier and making math more accessible to everyone. This is not an easy feat. I think we need some sort of concerted effort for encouraging more people to teaching maths or any subject through intuition. Visualization is one way to improve the teaching methodology. We can also distill intuition from stories, feelings, situations, etc. More examples, see: distill.pub/ colah.github.io/

[@ollerich32]

Cedric Chee spot on! And thanks a lot for those links.

[@shreyanshdarshan3199]

You have made life so much easier.

[@angelanikolova7354]

I am writing my internal assesment for school about neural networking and thank god your channel exists cuz everything about this subject is now clear to me. You explanations are perfect!

[@abhinavgarg5611]

I have watched first 3 videos of this series and i have gained a lot of insight about what basically happens within the black box. Thanks a lot 3Blue 1Brown for these wonderful video illustrations. It took me more than 3.5 hours along with note making. I have still not completely understood NN, but its ok. I am looking forward to video number 4 of these wonderful series. Thanks a ton once again :))))))

[@manjunathsastry7540]

Oh dear! Absolutely brilliant. This is one heck of video. How would you consider creating an exhaustive ML and Neural Network video series? I'm sure there are a lot of curious folk out there, waiting for it!

[@SkyTitans]

Thank you soooo much! This channel is pure *GOLD*!

[@joshisushant]

your videos are insanely amazing, and perhaps THE best source to learn any mathematical concept. We are extremely grateful that you take so much time and effort to create such a fantastic content.

[@prithvidhelia3516]

This is definitely one of the best educational video series I have seen. Explanation is concise, abstract when needed and visualizations are phenomenal

[@marcotroster8247]

I love the statement "the most confusing part is the notation". This is so true 😂 It would be really, really nice if you did a last video on how to program this network in a comprehensive way 😄 IMO the inventors of DNN chose all those activation functions and matrice multiplications really carefully to make computation feasible. But all this beautiful simplicity is gone once these unnecessarily complicated math formulas come into place. Honestly, you did an amazing job with your videos on DNNs and backpropagation. The first time I seemed to understand it.

[@ObitoSigma]

Now to watch all three parts at once!! Great video 3b1b, keep up the great work man! δ.δ

[@vgdevi5167]

Hello, I'm impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

[@Archonch]

I cannot overstate how much I wish I had these videos during my machine learning course a few years back

[@jimmieboboyo]

Normally, I never comment on youtube videos, but I'm making an exception for this one. Even though you might not see this, I want to thank you for your contribution to everyone's education. And for FREE access to it too. Amazing videos with amazing presentation, structure, and explanations. I'm 3 years late, but please do keep up the great work!

[@Michael-vs1mw]

* eagerly waiting for a video about convolutional neural networks *

[@3blue1brown]

+Michael Incog It'll be a little while. The next few videos won't be for this series, so I'll probably return back to this in a few months.

[@UtsavMunendra]

Computerphile did some videos on convolutional neural network

[@wanderingrogue3039]

3Blue1Brown Please please make one soon

[@alberttamazyan]

@@3blue1brown hasn't the few months passed? The whole community is waiting for a video on convolutional neural networks.

[@ciherrera]

@@alberttamazyan Ikr :(

[@UtsavMunendra]

Kurzgesagt and 3Blue1Brown in the single hour! I am in heaven.

[@grainfrizz]

Utsav Munendra ikr?!?!?!

[@zbzb-ic1sr]

Utsav Munendra I was about to comment this.

[@shreyashervatte5495]

So true!

[@Martinmarshallmargella]

this is fav youtuber so farrrrr

[@AlexiLaiho227]

haha me too

[@TheOnlyKeksSuchti]

You're saving my presentation! I didn't know how to explain neural networks to my fellow students but now I know how to do it! Thanks mate!

[@PunmasterSTP]

All of the other commenters beat me to it, but I can't express enough how insanely high the quality of these videos is, and how much they help teach people new things. Thank you so much for making all of them and sharing some of your vast amount of knowledge!

[@sohamharnale2878]

This man is the eternal fountain of knowledge straight from the heavens

[@CyberCookieMonster]

I am a “seasoned” systems engineer, inventor, and IT professional. IOW, I’m old (40+), lol. Like @Robert MacKinnon has said, you may think that your wonderful videos are only for those new in the field or younger folks in general. This couldn’t be further from the true. Your ability to explain things with clear diction, amazing graphics, and compelling story telling is phenomenal. They help me to not only refresh to get spun up quickly, but also serve as a point of reference for me to direct those newer in my teams; as your ability to run through this at just the right level and pacing far exceeds mine. I would go so far as to suggest that you freelance as a guest speaker or instructor for hire for larger firms having folks wanting to get spun up quickly. Apologies if this is old news and you’re already pursuing this. I simply feel compelled to tell you as we often don’t understand how awesome we are and need to be reminded by others. Consider this your reminder ;) Thanks again for your wonderful videos where you ask for nothing. I would pay for a Patron page if you get one to be able to ask for more content or explanation on topics such as the vanishing gradient problem prone to recurrent neural networks and how LSTM addresses it. I’ve been looking for a good one that does this graphically with intuitions. I haven’t found a good one yet, but I did come here first to check ;) All the best, CCM.

[@Spektrob]

This is by far the best educational content i've ever seen on youtube. Thank you and go on with it.

[@ammararif7524]

Half the time I'm in awe of the quality of the content. The other half I'm learning. Thank you so much man.

[@chaohongyang]

3blue1brown > Indian guy on youtube >>> my CS teacher

[@chriswangux]

Came for the knowledge, stayed for the Animations. Here I'm thinking how it can be done in After Effects, and turned out it's custom engine!

[@grossersalat578]

This series is really well done and so important - thank you so much!

[@shukkkursabzaliev1730]

wow, I am surprise anyone knows this sophisticated concepts to explain them clearly, and yet you also accompany them by amazing visuals which I believe not easier to do. Thank you!

[@lianmccc]

This series is better than my in total 6-hour lecture from school.

[@TheLouisandKyleShow]

This video is literal ART. Grant is my favorite artist.

[@Hgkbukk]

You are great at teaching! So many gets lost in the mathematics of it, not caring if their audience understand it or not. You make it incredibly simple! Great job!!

[@raver8056]

I love these moments in maths or other sciences, where you have this one singular moment of pure enlightenment. You got me at 7:37 ....I got goose bumps when the first column of "+" appeared. This is just brilliant.

[@JemimaGoodall]

This should be art. I don't normally comment on videos but I feel I must express my gratitude and appreciation for your amazing explanations and animations! Visualisation is so important and helpful and you have nailed it beautifully!

[@grainfrizz]

9:00 Eureka moment. I cried.

[@ChopinFlutist]

I actually got touched for real from these videos. A great example of the beauty of the math.

[@adiadiadi333]

Felt obvious to me 😕

[@Kiros37100]

aditya sai You are so intelligent! Wow!

[@sohamharnale2878]

Hahaha I feel you mate

[@milanstevic8424]

@Kiros37100 You're so bitter and sarcastic! Wow!

[@mattiasli]

extremely well made high quality material, every animation, every transition perfectly timed with the explanation. priceless

[@neilmcfarlane]

Amazing series; the best I've been able to find on explaining such a complex topic. Thanks!

[@TheBukkitArea]

AAAAAAAAA YEEEESSSS I'VE BEEN WAITING FOR THIS VID

[@markdashark3443]

I wish I had the option to pay my college tuition to you instead of my university. Well done mate!

[@terencechengde]

You have no idea how useful this video is to me. I learnt deep learning at uni but honestly this has just helped me through some hard time!

[@hdee5615]

Your animations are so good!! And when you do recaps and reference to what you have showed us..this is to awesome!

[@veratsien2014]

This is a lifesaver for completing week 5 of Andrew Ng's Machine Learning course!! Been stuck at backpropagation for hours and finally found some clarity in this video. I love your channel! ❤️

[@Endothermia]

Haha, I'm going through exactly the same thing!

[@veratsien2014]

@@Endothermia lol, high five and hang in there mate. :)

[@bricechivu8573]

I might misunderstand but maybe not. At 7:38, we want to reduce the activation of the neuron responsible for 3 right? So we should decrease the activation of the neurons that have a positive weight to 3, no? In the video, it's actually the opposite. For example, the first neuron has a positive weight with the neuron responsible for 3, so we should decrease its activation right? Can someone help me on that please?

[@nickgardner5641]

I had the same question. It seems like the color-coding of the video at this point is inconsistent? (it applies the same procedure for adjusting the preceding layer of neurons to all the other output neurons as it does to the output neuron for 2, when really the signs in the procedure should be reversed for all the neurons besides 2, since we want their activations to decrease?)

[@AyurvedaGyan]

your videos have made it incredibly easier to grasp fundamental concepts of ML. Great work!!

[@Zitlezot]

Thank you for these great tutorials! You make something complicated seem logical and fun, definitely fuels my interest for learning more!

[@toolegittoquit_001]

You guys just gotta stop blowing my mind ....💥

[@toostoned42069]

I would expect the training data to also include a bunch of random images that the network could classify as "not a number" (not hotdog 😉) That way, instead of just taking its wildest guess at which number it's detecting, it could also intelligently say "this doesn't look like a number to me"

[@kalebbruwer]

Nick V Yes, this would probably force it into more logical methods than what it turned out to be last video.

[@erilgaz]

it could also intelligently say "This doesn't look like anything to me." FTFY

[@skyacaniadev2229]

That's a good idea. Also I wanna see what will happen if alphabet and numbers are trained together.

[@allenwalker4703]

From what i understood i think you could just program it to say that whenever all the values in the last layer are either negative or really low without changing the back propagation process. Not sure tho.

[@TiagoTiagoT]

You need to be sure your "random non-number" is actually a lot of different things, otherwise the "not a number" result would have similar chances of being triggered as the number results when for example instead of alphabetical characters you present the NN with white noise it didn't train on.

[@balub326]

Beautifully explained! The video and narration cannot be easier to understand given the complexity of the mathematics of stochastic gradient descent. The artistry is very evident (and exciting!) Thank you.

[@thevoodooninja]

I've never clicked on a video this fast

[@assaissa]

What for did you clicked on the video?

[@samanrahbar8088]

Hey! Just want to shout you are my HERO! I'm an AI and ML researcher and I LITERALLY cry watching those awesome explanations! YOU ARE MY HERO!

[@Kishimita7204]

Ngl man one of my Goals, once I get a good enough job. Will be to be able to live contribute to your patreon. Your videos have been so entertaining, inspiring, and helpful that I hope the move that I give allows you to continue with this channel.

[@davidswygart7472]

I am a neuroscientist, and all your comparisons of actual neural networks to artificial neural networks seem pretty spot on to me. This is a great video series.

[@m0rjc]

"This is where we were in the 80s/90s" - I did my degree in the 90s, so this is the level I learned to. The more up to date stuff would be interesting.

[@maxsmith1733]

Hello, I'd like to start by saying your videos are amazing and beyond intersting. One thing I'm still confused about is how to update the biases through backpropagation? Thanks! and keep it up!

[@ybahman]

This!

[@just_a_duck3371]

I also don't get it what these "nudges" actually mean. Do I simply add value from computed gradient to the weight?

[@rossdemirel1365]

Impressive way of presenting complex information. Your teaching is really an art. Thank you!

[@genericnamethingy]

Always impressed by your videos! You have no idea how much they helped with my EE degree over the years. Eternal thanks

[@TheFadime123]

I think it'd be extremely beneficial if you were a complete lecture series for deep learning. Your way of teach is billion times better than this somewhat popular Andrew Ng lecture series, and also a lot more intuitive. Please consider this:) Best wishes

[@atul6147]

Sanderson>>> Ng

[@twiggy_witch]

At 12:25, he labels Fermat as a tease 😂

[@alectoperez1383]

#FermatsLastTheorem

[@masonhunter2748]

And Albert Einstein as a genius

[@plingvivin3032]

@@masonhunter2748 what

[@SuperReminou]

Amazing work !! Such a pleasure to understand those underlying notions ! Thank you so much

[@hlbusiness4432]

Fantastic work at explaining "difficult mathematical formulas" in very telling simple images! Great job, you are an excellent teacher!!! Thanks for your work.

[@conoroneill8067]

I feel like I'm missing something with the functionality of backpropogation. So, I get that you adjust the weights between the last two rows to adjust how the second last row feeds into the last row in the optimum way, and then you change the second last way, and then you backpropogate the second row of weights away from the end, etc. But, what I don't understand is that if you change the second last weights, that's going to change the second last row of inputs, which means that the tweaked weights between the last hidden layer and the output will be wrong again. I'm trying to work out what I'm missing. Is it that all the layers of weights are tweaked simultaneously in order to be accurate? If so, I'm still not sure I understand how that works (I'll probably try and watch the video again in the morning when my brains properly awake.)

[@3blue1brown]

+Conor O'Neill Good question! Backpropagation is just giving the list of adjustments you will make, but you don't make them until all are computed. And of course, since this is part of (stochastic) gradient descent, that's a process you repeat multiple times.

[@alectoperez1383]

You bring up a really good point. The next video, which will cover the actual mathematics of how backpropagation works, does take that into account. If you've taken calculus before, you've heard of the chain rule, which describes how to differentiate an expression like f(g(x)). The chain rule states that d/dx f(g(x)) = f'(g(x)) * g'(x). The chain rule is general enough that it's applicable to pretty much all situations where you can express stuff in terms of an f and g, even when f and g take multiple inputs. The important thing is that it allows you to only analyze one input at a time, making it easy to compute stuff. What happens in the case of backpropagation is that you're taking the derivative in terms of each of the weights of the cost function, and the chain rule accounts for how changes affect other changes. The math is too complex to put in a youtube comment (mainly because youtube doesn't have mathjax, so I can't write a lot of the symbols I'd need, like summations), but the next video should explain all of that!

[@conoroneill8067]

Thanks! That helps! Thanks for a great video - I love your work and the effort and care you put into these videos. Truly incredible. Thanks again.

[@ashirizly]

Consider that the changes merited by each example are quite small (because each example alone doesn't carry enough weight to merit making big changes), so the changes of weights based on these values are still accurate. It doesn't matter if you do a tiny step after each example, or a larger one after a thousand examples, the over all effect of each example is still like a single tiny step in the direction it points to.

[@lucianodebenedictis6014]

For me the most difficult part about maths is ALWAYS notation, as they expect me to know what exactly that greek letter stands for in every situation. Or maybe this is what's it's all about

[@Skironxd]

I find it really confusing when there's notation cross-over. Especially in the world of physics and engineering, there's only so many defined symbols that can be used for so many more concepts and topics.

[@screwhalunderhill885]

It's pretty natural for a mathematician. That's a problem when subjects, like this one, overlap with other research areas.

[@lucianodebenedictis6014]

I think we are all meaning the same thing

[@lucyfrye6723]

Thank you so much for this video. I used this and a few other resources to write a few c++ classes from scratch so I could be sure I really get the basics right in my head. It turned out to be trickier than I had assumed, filled with subtle pitfalls, not easy to debug and a few blatant mistakes. Not sure if reinventing the wheel was time well spent, most people should probably go straight to some off-the-shelf code but I got there in the end and there might very well be some kind of reward down the line for doing things the hard way. Thanks again, your video was invaluable.

[@raulguarini2342]

Absolutely great video and playlist as a whole. A great head start!

[@shengchuangfeng227]

At 7:37, shouldn't the activations associated with the digit 0 neuron (whose activation should be decreased) change the opposite direction? For example, if the neuron at the very top of the second-to-last layer increases activation, with a positive weight, the digit 0 neuron's activation would increase. Is that correct?

[@ludd8489]

Same question here...Someone has an answer for that...?

[@MrDivyanshu33]

I'm just starting out with Deep learning this quarantine and man...it's a bit difficult to get my head around these concepts.

[@JonesDTaylor]

Me too man! I am doing the course by Andrew Ng and things are getting much clearer.

[@MrDivyanshu33]

@@JonesDTaylor Me too 😁!

[@parthvasoya3562]

@Winston Mcgee hey..why??

[@plasmasheep4098]

@@JonesDTaylor same!

[@RitikSharma-pc5yj]

Include me too in your category...

[@LakTheShadow]

It is stunning how much better and clearer you can explain all this than my professor at university, and you are even a lot faster! Thank you so much for producing all those brilliant videos. Some of some help me understand what my professors are trying to tell me, and some of them are just fascinating. Keep on going!

[@vgdevi5167]

Hello, I'm impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

[@user-fg6ng7ej6w]

this author has an amazing talent to explain complex things in a simple intuitive visual way. that what most of academia lacks. thanks

[@tejas8211]

Dude how are you making these animations

[@screwhalunderhill885]

he makes them in python

[@shaileshupadhyay3920]

Dr tenma how are u