The spelled-out intro to neural networks and backpropagation: building micrograd

This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school.
Links:
- micrograd on github: github.com/karpathy/micrograd
- jupyter notebooks I built in this video: github.com/karpathy/nn-zero-t...
- my website: karpathy.ai
- my twitter: / karpathy
- "discussion forum": nvm, use youtube comments below for now :)
- (new) Neural Networks: Zero to Hero series Discord channel: / discord , for people who'd like to chat more and go beyond youtube comments
Exercises:
you should now be able to complete the following google collab, good luck!:
colab.research.google.com/dri...
Chapters:
00:00:00 intro
00:00:25 micrograd overview
00:08:08 derivative of a simple function with one input
00:14:12 derivative of a function with multiple inputs
00:19:09 starting the core Value object of micrograd and its visualization
00:32:10 manual backpropagation example #1: simple expression
00:51:10 preview of a single optimization step
00:52:52 manual backpropagation example #2: a neuron
01:09:02 implementing the backward function for each operation
01:17:32 implementing the backward function for a whole expression graph
01:22:28 fixing a backprop bug when one node is used multiple times
01:27:05 breaking up a tanh, exercising with more operations
01:39:31 doing the same thing but in PyTorch: comparison
01:43:55 building out a neural net library (multi-layer perceptron) in micrograd
01:51:04 creating a tiny dataset, writing the loss function
01:57:56 collecting all of the parameters of the neural net
02:01:12 doing gradient descent optimization manually, training the network
02:14:03 summary of what we learned, how to go towards modern neural nets
02:16:46 walkthrough of the full code of micrograd on github
02:21:10 real stuff: diving into PyTorch, finding their backward pass for tanh
02:24:39 conclusion
02:25:20 outtakes :)

[@georgioszampoukis1966]

The fact that this video is free to watch feels illegal. It really speaks volumes about Andrej. What a stunning explanation. It takes incredible skill and expertise to be able to explain such a complex topic this intuitively and simply. All I can say is thank you from the bottom of my heart that you offer videos like this for free. What an amazing man!

[@DrKnowitallKnows]

Andrej, the fact that you're making videos like this is AMAZING! Thank you so much for doing this. I will be spending some quality time with this one tonight (and probably tomorrow lol) and can't wait for the next one. Thank you, thank you, thank you!

[@2ndfloorsongs]

And thank you for your videos, Dr Know It All. Always appreciate them.

[@mattphorwich]

I was stoked to discover Andrej sharing the knowledge on these videos as well!

[@lonnybulldozer8426]

You made love to the video?

[@0GRANATE0]

And what happend? Do you now understand DNN?

[@fhools]

When I'm confused about deep learning, I go back to this video and it calms me. It shows that there is a simple explanation waiting for someone like Andrej to show the light.

[@FireFly969]

Yep I watched a 52 hours course on pytorch, I was learning how to build a neural network but not how neural network works, which is a stupid thing from me, and a good lesson, to learn how things works if you want to learn it.

[@nyariimani7281]

This reminds me of my college courses, except it's way better in three ways: 1) Here the speaker really does know what he's talking about. 2) I can stop and rewind, get definitions, and practice myself before moving on to the next step over and over, so I can get the most out of the next step because I actually had the time to understand the last step. 3) I can do this over several days so I can keep coming back when I'm fresh and present. You are a gem and I really, really appreciate you creating this.

[@peterdann643]

Simply stunning. I'm a 72 year old fiction writer with rudimentary computer programming skills whose son works professionally in this area. I wanted to gain a better understanding of the technology he's working with, and writes scientific papers about, and now I feel I've made a great start in that direction. Wonderful!

[@BarDots315]

You're a great father never change!!

[@hamdanalameri2885]

what an amazing father you are, My dad also tries to keep up with all the technologies just so he can understand and get to bond with his children. I want you to know that we appreciate you guys and love you.

[@ClayMole]

You're awesome! May I ask, how has it been to work as a fiction writer? Would you recommend it?

[@manishj5154]

All the people fawning over him, you do realize he started with saying he is a "fiction writer". Granted that it's pretty cool if this isn't fiction.

[@semtex6412]

@@manishj5154 he's likely backpropagating

[@robl39]

Finally… someone who understands it well enough to explain it to a beginner. This is hands down the best NN video on the Internet. Thanks a ton!

[@kemalatayev]

Just and FYI for those following at home. If you are getting an error at 1:54:47 you should add __radd__ into your Value class similar to __rmul__. It will allow the order of addition to not matter. I don't think it was shown in the earlier sections.

[@adamderose9468]

ty, i needed this at t=6422 in order to sum(wi*xi for wi, xi in zip(self.w, x)) + self.b

[@CarlosGranier]

@@adamderose9468 Thanks Adam. This had me stumped.

[@karanshah1698]

Underrated comment...

[@lidiahyunjinkwon7138]

OMG, thank you so much. It was driving me nuts.

[@jamesb43]

That’s comforting. I thought I just missed it. Good on you for sharing this

[@waldof86]

I've learned more following this along in some hours than I've learned in a year's worth of classes. Thank you for being so open with your knowledge

[@GregX999]

OMG! This is the first time I've ever TRUELY understood what's actually going on when training a NN. I've tried to learn so many times, but everyone else seems to make it so unnecessarily complex. Thanks for this!

[@ophello]

It’s spelled “truly.”

[@khaldrogo9451]

@@ophello giggity

[@pooroldnostradamus]

@@ophello Excuse his poor training dataset

[@veganath]

@@ophello an example of backpropagation, you have no doubt adjust Greg's waits...lol

[@sidg11]

@@veganath it's spelled weights ...

[@imtexaspete]

"remember back in your calculus class?...." nope. I'm subscribing anyway whenever I need a humble reminder that I don't know anything and there are people way way smarter than I am.

[@omkarajagunde4175]

W O W Same realisation 🙌🙌😔😔😔

[@Forrest_dev]

It's never too late to learn.

[@vidbina]

The beautiful part of tech is the feeling of constantly being mind blown when realizing how little one knows and how much there is to learn. Studying micrograd has been on my list for a while thanks to George Hotz and this series is making the owning of this context so much easier. Loving it. ❤️

[@pastuh]

If someone can explain, means its simple

[@CheeseBae]

I took through calc3 and aced it twenty years ago. If you put even the most basic calc problem in front of me now I couldn't even tell you how to start.

[@lawrenceadu-gyamfi4179]

Just wanted to say a big thanks to you Andrej and the team working on this. Truly amazing, the clarity with which you explain these things is impressive and inspiring! Looking forward to see the remaining videos and even more. Thanks again!

[@kerwinmarkgordo3458]

Thank you so much for doing a step by step simulation on how gradient descent works. I am grateful for the passion and effort you make in order to teach. These lessons are very essential as we continue to dive deep into learning.

[@gabrieldornelles9310]

I'm really inspired by you as an educator, and I'm very happy to see you sharing your knowledge in a lecture after a long time!

[@ThetaPhiPsi]

This is the single best explanation of backprop in code that I've seen so far. I've once implemented a neural network from scratch, except autograd, so Micrograd is a good fit and so clear and accessible. Thanks Andrej!

[@leslietetteh7292]

Actually true. And exactly the same, I've once implemented a neural network from scratch, and I broadly understood, but this is the best explanation of backpropagation I've seen. Excellent work.

[@mzlittle]

Revisiting this again and appreciating how much of your time you put into educating the rest of us. Again, thank you!

[@aayushjoglekarpersonal7392]

Thank you for making this tutorial! I have always been on a lookout for something like this. Normal videos either discuss super deep details or go on a brief overview. This was a perfect balance between depth and showing the actual usage of what we built. Bingeing your playlist now! :D

[@Qattea]

Thanks for this Andrej! I love the direction you are taking. I’ve been wanting to learn this and now I get to learn from the best

[@sanjay-89]

This was an exceptional lecture. Just wanted to say thank you for taking the time to make this. I have spent time in university courses, reading books, doing assignments and yet, I truly understood more out of this single lecture than from anything else prior.

[@manupatet]

I was looking for some intuition on backprop and this is it! Thanks Andrej for taking time out of your schedule to share this precious knowledge.

[@nyariimani7281]

You are really fun to watch. It's so nice to learn this from someone who really understands how everything works.

[@nkhuang1390]

It takes real talent, dedication and complete mastery of the subject matter to breakdown difficult technical topics so clearly. Its also clear that Andrej is having fun while he elucidates. This is simply the most amazing series of educational videos on the internet on these topics. I hope you continue to put out more material like these.

[@bergonius]

Great teacher with great background and expertise. We're lucky to have him spending his time to share his knowledge with anyone who wants to learn, for free. Looking forward to more videos.

[@notkamara]

He's great! He even has an old KZbin cubing channel (Badmephisto) and his tutorials there are awesome too!

[@alisaad679]

@@notkamara omg i learned how to solve a rbx cube from him decades ago thats crazy, now im learnning neural networks, crazy how the world works

[@alirezamogharabi8733]

I have been watching educational videos about neural networks for years, but no one had ever taught me like this. I know you since 2016 with your machine learning class videos. Thank you very much for these wonderful tutorials

[@ichidyakin]

Andrej, thanks for your NN video series! It’s really amazing how you explain in so simple terms how it works under the hood! Wish you all the best on your new position and hope you are going to continue making such great content!

[@hefwilliams5400]

Andrej, Great to see you on KZbin - you're knowledge is incredible and an asset to the community

[@2ndfloorsongs]

Thanks, really cleared up a few confusions for me... And added new ones. Perfect. I'm looking forward to your future videos.

[@mahmoudabuzamel7038]

This is amazing! You've just simplified the mathematics behind neural networks to an extreme extent. Thank you Andrej.

[@Expateer]

I'm in awe of your rare and great gift for explaining complex things in a way to make them easy to understand. My mind has many possibilities I hadn't considered before this video. I can't thank you enough for doing this.

[@karthikbhaskar974]

Wow Andrej! Big Fan of your work. Looking to learn more from you about Neural Networks and you being on KZbin teaching is going to reach billions of people who wants to get into AI and learn from the amazing instructor the world has ever seen. Really loved your Stanford Computer Vision course now looking forward to more awesome content here in KZbin

[@jamesdavison6290]

Andrej, I am a huge fan of your previous videos. Thanks for making this investment of your time to bring us closer to these amazing concepts!

[@irwindennis]

Wow!! I am watching this tutorial half way through and I am so stoked to learn the fundamentals of neural networks with such clear explanations. I don't remember in any of my university studies to have received a learning session as this one for a subject like this. If you are curious to learn the basics of modern ML and inner workings of neural networks, this is definitely the video to watch!!:) Thank again Andrej!

[@FabioAngela]

It's really easy to see people who master stuffs and are actively into it, because they love what they do/talk about and are still excited to talk about it, I feel the same on the subject I'm focused on. Keep up the wonderful job you've done here!

[@treksis]

This video reminds me of my old numerical analysis professor who forced us to draw every interpolation problem by hand as an assignment. We drew all the tangent lines with a ruler and a protractor like a kid in primary school. We were complaining because that was just a few lines of code in Matlab, but in the end, that really helped us to develop true intuition behind it. Thanks for the intuitive video.

[@caseyleemiller1]

This is an excellent tutorial not only on neural networks but python and Jupyter notebooks as well. Lost sense of self for 2.5 hours and learned a ton! Thank you.

[@aamerabbas]

This was an incredible video. I wish I would have had this when I very first tried to learn about NNs. I would have been able to start my journey with so much more intuition on how things work. Thank you for making this - I will whole heartedly suggest this to anyone who wants to start learning about ML.

[@CarlosGranier]

Hey Andrej, thanks for the very detailed and understandable explanation. I went through the NNFS book last year and your video was the perfect complement to it. The graph plots were particularly helpful in visualizing the architecture. Looking forward to watching the rest of the series.

[@krebul]

I'm a traditional dev and I have tried a lot of different guides and tutorials on neural networks. This is the first time I have been able to understand it. I'm about 1/3 through the video and it's 2am. Thanks for your excellent breakdown!!!

[@0GRANATE0]

Did you finish the Video? Do you now understand it? Are you able to read papers in this area and implement them?

[@brenok]

@@0GRANATE0 Is this some kind of suggestive question?

[@ChrisOffner]

Amazing tutorial, very cool! I love that you patiently walked through a lot of manual examples - too often educators get self-conscious about showing simple steps more than once and then yada yada yada their way through it, which helps nobody. Love your teaching style and hope to see more.

[@gnorts_mr_alien]

what a legendary piece of video. I knew 80% of this stuff but it was very hard to get there to begin with. if only someone explained all this to me like you did when I was confused about all this stuff. you have a very calming presence btw. thank you for having a go at this from the first principles, so very helpful. I'll binge the other videos now.

[@user-nl3od6qk7z]

Андрей, огромное спасибо! То, что ты делаешь обучающие видео просто невероятно! Спасибо!!!!!👏👏👏

[@raphaelkalandadze9691]

What an astonishing lecture, the best explanation of backprops, and the whole cycle is so intuitive and easy to understand. I wish I had a teacher like that. I would know everything 100 times better than I do now. Somebody is still saying that a child will learn everything on his own, but I bet everyone will be a genius if Andrej teaches them and all of you are happy you attended his lectures at Stanford. I wish I had such an explanation skill one day I'm glad to see you on KZbin, and I hope you continue this series in the future I have a lot more to say, but I hope to tell you in person one day thank you 100 times

[@agehall]

When I took my AI course some 20 years ago, people were pretty depressed because things were too hard to compute and there was very little future in this type of thing. Awesome to see the simplicity in this and how powerful it is. We’ve really come far in the field of AI.

[@DrRosik]

This video is by far the best and most educational I've seen regarding "how NN work", and I have looked very long. For me, even though I thought I knew how NN work and operate, at least to the point that I could use the tools out there to build a simple one. I never fully grasped the details and was always frustrated why I didn't know WHY I had to use a specific setup for my NN, I just "knew" that this is the way to do it. This video explains the basics in such a simple and logical way. Thank you Andrej! And please keep up the good work.

[@NehadHirmiz]

This is a brilliant lecture. Thank you very much for taking the time and putting this together. I love the bloopers at the end :)

[@chris_piss]

Many others have already said it, but thank you so much for making this. I've been trying to learn machine learning for many years now (in several short-lived attempts), and this lesson was a huge missing piece for me. Understanding the calculus behind it all, and how to really grasp how the weights and biases affect the output, really made back propagation and the learning flow click for me

[@keikaku9298]

CS231 was life-changing for me. You are a fantastic educator. I hope this new endeavor works out for you!!

[@antweb9]

I would like to thank you from the bottom of my heart for making this. I'm a developer myself but this new advent in AI seemed unapproachable. Thanks, for making it clear that no subject is tough if you have a great teacher. I am seriously going to consider this area as a thing I want to do next.

[@user-zb8rc3kn5j]

Thank you for implementing backpropagation and automatic differentiation in such an elegant and easy to understand way. This is the most detailed and in-depth beginner's course I have ever seen.

[@OlabodeAdedoyin]

The weird sense of accomplishment I felt when I visualised (draw_dot) the all the operations in its full glory is unreal 😂. Thank you for this 🙏. I'm a software engineer that has been trying to truly understand neural nets for a year now. I'm not sure you understand how much this means for me. I really appreciate you 🙏

[@chingizabilkasov6625]

Thank you very much for that lecture Andrej! It really helped to understand and to combine different pieces I learned separately into one structured concept. I especially appreciated that you left the part with the bug on gradient zeroing and made an explanation for it at 2:10:24. Making mistakes and learning from them is so effective and undervalued imo. Thanks a lot!

[@JellyfishJNM]

Awesome Video. I did not only finally understand back propagation in a way that I could explain it to someone else, but also realized again that math can come across super complex when explained poorly, or nearly obvious with a different approach. Thank you so much for putting in the effort to record the videos in that much detail. I loved that you took the time to explain the main topic and also all the math around it.

[@Themojii]

I subscribed right after I saw you are creating videos on Neural Network. I have huge respect for Andrej that shares his valuable knowledge with us for free. I am starting today watching this first one and can't wait to watch the other videos and the upcoming ones. You are changing lives Andrej

[@daverothery9713]

Thank you so much Andrej. I'm good at coding and bad at maths and this is the first time I've been able to properly understand a lot of this stuff... I just needed someone to explain it in my language :) I went along with the first half in detail, then in the second half I built the same MLP using pytorch tensors just to prove to myself that I understood how this all applies to pytorch, and it all worked great :)

[@necbranduc]

I already know this is going to be good. I remember watching and enjoyin your Stanford lessons on KZbin.

[@alexanderoros6484]

Thank you so much for your insights into the field. I try to wrap my head around NN for so long (2001). I am so glad to finaly found a teacher that gets down to the core with the least amount of clutter and teaches me the fundamental basics. Looking foreward to the following videos where it gets more complicated and you show the history of ML with accessible examples.

[@asatorftw]

Watched this again, and its still blowing my mind how useful this tutorial is. Thank you Andrej!

[@punto-y-coma7890]

By far, the best neuronal networks introduction and tutorial ever made on KZbin. Thank you Andrej for sharing your valuable knowledge.

[@luismonge8720]

I have carefully listened to lots of explanations about Backpropagation and every time I understood a little more but this was SO clear and easy to visualize. Thanks Andrej!

[@hjr834]

Great lecture Andrej, you were AMAZING over it, it really broke down several complex topics on a very understandable form, thank you very much for giving the lecture!

[@markonjegomir8714]

Nice to see Karpathy going back to making some educational content! 🙂This is a must-watch!

[@mohit9920]

That was incredible. Never has anyone been able to simplify Neural Networks in this manner for me. Please keep making such videos, you're doing gods work. By god, I mean the imminent AGI :)

[@liamroche1473]

Prescient. ;)

[@ittaig]

Greate lecture - simply and thoroughly explained with neat and clear code. One of the best lectures I have ever heard. Happy to have found this treasure. Thank you, Andrej!

[@zhenli8674]

So educational and very clear to understand. I never expect to understand the whole nn framework/fundamentals in 2.5 hours. Thank you so much!

[@anatolianlofi]

This is probably the simplest, most well-paced explanation of back-propagation I've seen on KZbin. I wish everyone would break down information in this way. Thank you.

[@Kobe29261]

Its really sad; we don't pay the smartest people enough to be teachers. You think about it and its atrocious; the problem is actually worse - people like Andrej are coopted by big corporations where their expertise and research can be hidden behind a wall of NDAs

[@sam.rodriguez]

This is fantastic. Thank you Andrej

[@rohanshah9593]

It feels good to learn from someone well known in the industry. Thank you for sharing. I have learned a lot from a practical perspective and look forward for more. Really appreciate these videos!

[@cxdimarco]

Incredibly detailed and well explained tutorial, Andrej. Thank you - you are without a doubt a master of the topic.

[@angelsancheese]

You’re an amazing teacher. Thank you for the video!!

As a dev, sometimes I follow along some A.I. course so I'm looking forward to follow this one! Thanks for sharing your knowledge.

Just watched the last bit. The python notation got really intense when Pytorch was introduced. As a non-regular Python dev, I'm going to work out some code snippets to fully grasp the example. But dang, so all the trouble of the gradient descent is "just" to know the direction of the optimization for each node? Very insightful! Is the mathematical capability of computing the local derivative the root cause of the local maximum trap?

[@arsalanzabeeb6467]

what an amazing lecture , no words for it . thank you so much spending so much time and efforts . everyone watching and learning from would be praying for you man .

[@6Azamorn9]

This is enormously valuable knowledge and I'm grateful for your insight and how exceptionally well you are at teaching the fundamentals. Thank you Andrej

[@harrypotter6505]

Wow the op nodes you demonstrated this through made it so damn easy to intuit, its crazy, I have no math recall from school and didn't do any advanced math after school, nor coding and I had to watch this whole video in 3 or 5 passes to completely grasp what was happening, it was such an amazing journey to be absolute intimidated with the length of the video, the code, the math... I knew neither, I don't even know python to begin with, yet I was able to derive exactly all the concepts necessary to understand this video thank you so much Andrej!

[@galactic_dust42]

"Derive", yes, i think that's the word ! haha

[@user-kc3vh4xd9h]

Our professor highly recommended us to use your video to learn more about back propagation. You explained it so well. Thank you so much for making this video, this video really helped our study and understanding!

[@SrikarDurgi]

You've got a good prof. Many feel insecure and don't recommend anything good.

[@ZlatkoJoncev]

One of the best videos Ive seen in a while. Perfect level of explanations for someone who understands object oriented programming to expand to more complex topics. Greatly appreciated 🙇

[@fjg9657]

Absolutely the most informative and satisfying explanation and illustration of back-propagation and NN frameworks- and perhaps educational videos in general, that I've seen. Thank you!

[@amkamath]

This video is such an unexpected treat! Thank you so much for taking the time to do this. I have followed your Twitter account since a long time but completely missed this repo. I appreciate your approach of starting with the basics, and building up an intuitive understanding from the bottom up. Your class at Stanford took a similar approach (another fantastic resource!), but this is a speedier introduction 😊

[@Mutual_Information]

Goes without saying, but you're going to blow up on KZbin. Awesome to see you in the KZbin edu space. Maybe I'm a little optimistic, but I think the quality could approach that of Wikipedia one day. You joining is a good sign of that.

[@2ndfloorsongs]

Not without clickbait titles and the hint that there might be an exposed body part or two. Though he does have a picture of his face making an expression, so maybe there is some hope.

[@anasdev1553]

@@2ndfloorsongs Dude... who hurt you?

[@2ndfloorsongs]

@@anasdev1553 Haha, I have nine cats so your guess is as good as mine. 😸

[@anveshicharuvaka2823]

Andrej! You are one of the best teachers our there. You are doing incredible service to a whole bunch of people. Please keep these videos coming. We need more teachers like you.

[@Dipsuuu]

Great tutorial! I was looking for something that was practical with python to delve into the basics of neural networks. This definitely is among the best ones. Thanks Andrej!

[@NFT2]

I've been working with Python for years and never implemented classes with those operator overrides. Its never too late to go back to the basics. Great video man.

[@edwinmontufar5423]

Fluent Python is a good book if you'd like to understand these overrides, otherwise known as dunder (double-underscore) methods.

[@notkamara]

Amazing how you taught me how to solve rubiks cubes all those years ago when I was 12 and now at 22 I'm back here again learning backpropagation. You're doing God's work!

[@aqgi7]

Same! I learnt F2L all those years ago and now backprop!

[@rakotism]

Usually, watching long educative videos gives me headaches, but this time I didn't even noticed how time passed. Thank you! Also, direct connection of the loss function to the neural net was a fascinating insight! Looking forward to watch other videos!

[@IchibanKanobee]

This is an amazing tutorial. Seeing all steps of the development and the subsequent loss convergence feels like magic.

[@TheAIEpiphany]

58:41 "As long as you know how to create the local derivative - then that's all you need". Ok Karpathy. Next paper title "Local derivatives are all you need". Nice to see you on KZbin! :))

[@siddharthdhirde]

I appreciate that you did not edit out your mistakes in the video. It helped me to understand the common pitfalls in building neural network.

[@arisu2718]

Thank you, Andrej! That was just PERFECT!!! I've decided to dive in neural networks yesterday and today I've successfully done it thanks to you! Your explanations are very clear, and the whole fun vibe of the video is awesome!

Awesome! Finally finished walking through this in my own notebook. Doing it myself and being able to play around with it really solidified my understanding of building a simple neural net with forward pass and back propagation. Thanks so much for the learning opportunity!

[@curious_carbon]

Andrej, thank you so much for doing this 🙏. If all the 10X Engineers out there would do stuff like this, I think it would really boost our collective human intelligence by a significant lot. I appreciate your effort, and looking forward for more content 🤞. Also I hope that you do get to return at Tesla so you can contribute to Optimus👍. Looking forward to your future accomplishments 🚀

[@yashsurange7648]

It is amazing to see an AI leader sharing knowledge here

[@cezarmocanu5043]

Honestly, I have no words. This is an amazing presentation, in terms of code, math, logic. Can't wait to continue with the other videos. Just amazing. Thank you so much for taking the time, and sharing your knowledge

[@manug4604]

Amazing work, you truly inspire me to keep om learning Andrej. I Been Waking up early just to follow and test all you showed here. Its so beautiful when you see this whole structure coming to life in front of your eyes!

[@beathoven70]

Thanks so much for that video Andrej! While watching it i reimplemented micrograd in Javascript (the horror!) and with a few minor tricks got it to work quit nicely, even stuffed the actually train into the MLP class! I've done the Coursera course from Andrew Ng a few years ago and managed to get a full score there, but it still left me wondering a bit at the end how it all really ties together. With your 2 1/2h lecture here it finally clicked for me and now i really understood how it works under the hood!

[@AndrejKarpathy]

awesome!! :)

[@sven-0]

Cool, did you publish your code anywhere?

[@margaretesulzberger2973]

it is a very dense and inspiring lecture on scalar valued Neural Nets. I saw lectures from universities which also start with scalars and computational graphs but have a strict separation between theory and exercises for which they often use proprietary programming environments. This combination of theory explained/programmed simultaneously with commonly accessible python packages is unique. Your explanations were sometimes a bit short on python “magic methods” like “__radd__” but mostly I found hints with google. But not so for “ lambda: None” or “def _backward(): … “ and then “out._backward = _backward”? I’m waiting for a continuation with vectors and matrices.

[@TylerMeester]

As a new CS grad trying to prepare for my career and job interviews, it was a real pleasure following alongside you in this video! I had no idea how integral (pun intended) calculus was to neural networks and backpropagation! Mind = BLOWN!

[@kozer1986]

Andrej, this is one of the best videos on the topic I've ever seen. I like to understand what is going on under the hood, so your video helped a ton, especially that you took the time to show what's happening inside pytorch and the gradual building of everything. I'd love to analyze more complex stuff, like attention mechanism, and other state of the art stuff which I find a hard time to follow. Also if you know if this kind of analysis already exists for those concepts, I'd love to know! Thanks again for your work!

[@roddlez]

As someone who took Stanford's CS231n back in 2016 purely through watching the lectures and working though each assignment, this definitely strikes me as getting back to your educational roots. I do wonder how difficult it will be for others who do not have the background to approach this material. Conceptually, thinking about the training process of forward pass, calculating the loss, backprop, nudge on modern NNs with billions of parameters: from a computation standpoint, this requires many simultaneous reads from and updates to memory to/from the CPU/GPU, even for a single pass of data. Thinking from first principles, it would seem advantageous to assemble a custom computer architecture that would allow the entire NN math function (with billions of parameters) to remain in computational memory (registers) while doing forward and back propagation, thus saving time trying to load/unload/store updated values for weights and biases? Is there a company that's attempting to accomplish such a feat?

[@AndrejKarpathy]

your insight is exactly right. current computer architectures spend most of their time and energy shuttling data to/from memory, through the "von Neumann bottleneck", and calculating little pieces of the neural net at a time. this is not how it should be laid out and it is not how the brain works either, and yes many people are aware and working out various improvements. basically, "classical software" and "neural net software" need very different hardware for optimum efficiency and all the neural nets today run in "emulation mode".

[@roddlez]

@@AndrejKarpathy Amazing. Super excited for the future of ML.

[@elonfc]

@@AndrejKarpathy do you have elons phone number?😂

[@The_Special_Educator]

@@AndrejKarpathy You never responded with Elon's phone number. If you want to maintain your credibility you must post Elon's personal phone number, address, and underwear size in the KZbin comment section.

[@unchaineddreameralpa]

Joking

[@yuyangzhu]

Andrej spend 10 hours making 1 hour of content, and the 1 hour content actually worth 10 hours to go through multiple times

[@SydneyPanda2016]

These are amazing Andrej. Beautifully explained, logical, easy to follow. Thank you so much for generous knowledge sharing and time you put into creating the content.

[@carlosmunuerajavaloy3963]

I discovered you channel thank to your interview with Lex. This is just an incredible video, invaluable information. Thank you so so much for this.

[@TeslaFix]

Hey Andrej! I’ve watched the first 8 min so far! Super stoked to watch it even if I’m not a programmer. 😊 Would love to have you on my TeslaFix Podcast! ❤️