How to Create a Neural Network (and Train it to Identify Doodles)

Exploring how neural networks learn by programming one from scratch in C#, and then attempting to teach it to recognize various doodles and images.
Source code: github.com/SebLague/Neural-Ne...
Demo: sebastian.itch.io/neural-netw...
If you'd like to support me in creating more videos (and get early access to new stuff), you can join my patreon here: / sebastianlague
The data is from:
Digits yann.lecun.com/exdb/mnist/
Fashion github.com/zalandoresearch/fa...
Doodles github.com/googlecreativelab/...
Cifar10 www.cs.toronto.edu/~kriz/cifa...
Chapters
0:00 Introduction
2:39 The decision boundary
3:49 Weights
5:42 Biases
6:45 Hidden layers
7:45 Programming the network
9:57 Activation functions
12:42 Cost
15:07 Gradient descent example
18:22 The cost landscape
19:55 Programming gradient descent
21:10 It's learning! (slowly)
23:21 Calculus example
27:34 The chain rule
29:50 Some partial derivatives
33:14 Backpropagation
39:25 Digit recognition
43:56 Drawing our own digits
47:37 Fashion
48:25 Doodles
52:00 The final challenge
Music:
Cosmic Waves - Michael FK
Amber - The Stolen Orchestra
Beyond the Horizon - Sounds Like Sander
Air - Assaf Ayalon
Purest Form - Sounds Like Sander
Hear Wide Open - Sounds Like Sander
Universal Wonder - Moments
Roman P - Moments
All In Good Time - Shimmer
It Will Come Back - The Stolen Orchestra
Frontier - Shimmer
New Moon - Cloud Wave
Sunflower - Cody Martin
Inner Peace - Moments
Enchanted - Cody Martin
Just Around The Corner - Shimmer

[@SebastianLague]

Hey everyone, at 20:17 it should be -= costGradientW (the minus sign is missing). I somehow managed to delete it while formatting the code for the recording! Thanks for letting me know in the comments.

[@multiarray2320]

this video is even a better explanation of NN than the video from 3b1b and i thought his video was already perfect. thanks for creating this video because now i'm able to understand it much better.

[@shahzadansari849]

I was Always Waiting for Your Next Video , Finally Something to Watch and Explore ! #thanksForVideos its really helpful explanation with real examples

[@lucasgaperez]

youre so underrated buddy

[@hiihkumar9633]

@@multiarray2320 ez

[@SaleemKhan-qc1lh]

Fac card so c

[@niceguysayshi5765]

As someone who has some experience with machine learning i can say this has to be the most intuitive explanation i have ever seen

[@simonsemmler9804]

indeed. this is amazing

[@arivanhouten6343]

That is exactly what I was going to say

[@ProXicT]

As someone who has zero experience with ML/NN, I second this. What a great intro into the NN topic!

[@benamende5897]

The only thing missing is matrix math (I find it much more intuitive for some reason but I may be weird)

[@beraulgd3662]

Completely agreed! He actually walked through EVERY STEP (I can think of right now)! Which is great! One long video is soooo much better than scowering for dozens of bad ones (from experience)!

[@rienkthegamer5422]

I’ve never had someone explain calculus so intuitively. The quality of this content is absolutely incredible.

[@milanstevic8424]

He's not underrated, his content is stellar and known in the community for quite some time. He has almost a million subscribers, and everybody knowledgeable in Unity forums knows about his channel. Besides, in my opinion, after years of seeing all kinds of videos on game dev, Sebastian's are still one of the best, even old ones. And I'm not talking about just the style of presentation, I mean this strictly technically, his advice and his code are next level. Also I'm not young (as a human or as a game developer, the age is similar), and if I can learn a thing or two from him, and occasionally I do, that's instant five stars from me. Maybe he could become even more popular (though popularity comes with its own pressures and costs), but he's definitely not underrated.

[@johnhadden3998]

its pretty surprising that they don't teach differentiation by first principles in some places

[@rienkthegamer5422]

@@milanstevic8424 I am very sorry for abusing underrated in this manner, it’s just that, in my personal opinion (which is often wrong), concepts in his videos are often explained so thoroughly, yet he makes is so easy to understand, that I don’t see why more people love his content. And I know that CS isn’t for everyone, but *just* (I’m using just very loosely here, this is no small accomplishment) 1 million subscribers for videos unrivalled for this amount of quality is, as I said earlier, in my opinion, underrated. Sorry and I hope you understand, I might still be wrong Edit: to avoid confusion, I have removed underrated from the original comment

[@milanstevic8424]

@@rienkthegamer5422 Maybe I'm just picky with the words. This channel is definitely not as popular as, for example, Brackeys (although that channel is officially dead for a year or so). But to say it's underrated is something else. Maybe you thought like underappreciated by the general population / not recommended enough by the KZbin algorithm, and I can agree to an extent. But let's ponder instead, whether such front-page popularity a good thing. Those who do got to see his content, would definitely recommend it, and in that sense it is not underrated. His channel is constantly growing, regardless of the algorithm. I've seen people recommend Seb's videos to people who cannot grasp even the most fundamental concepts, so that's why I've said that popularity isn't always the best thing. His videos and experiments are very smart and sometimes beautiful, but you still want the audience to understand the beauty and the effort behind it. If you cultivate a wrong culture here (for example, children screaming that they get errors for the most mundane things and nobody understanding a word of what he says) his videos and his enthusiasm would certainly be affected by this. So, with all that in mind, maybe "underrated" is a good thing? I definitely would not call 1M a bad subscribership. That's gold on KZbin. Btw, now that you changed your message, I agree 100%. I'm probably just picky with the words.

[@khiemgom]

Well then prob u dont watch enuff maths related vid, there are other talented maths teacher on youtube, saying like that is an injustice for them

[@wlockuz4467]

I am a software engineer and I've always wanted to learn machine learning, being able to code is not a problem for me but being terrible at maths and statistics makes it hard to get accustomed to all the terms and concepts. I've tried multiple courses from different platforms and instructors and all of them try to teach you "what" to do instead of "why" to do it. I personally find learning more intuitive when I know why am I doing something instead of blindly following steps. This video is exactly the type of introduction to ML that I've always wished for, The explanations are highly intuitive and most importantly visual, there are no assumptions and no brushing over concepts, Nothings being done just for the sake of it, Everything is explained in simple language. I admit I'll still have to watch this video a couple of more times to make full sense of it because its jam-packed with so much information. You'll probably miss this comment in the sea of other comments (although I hope note) but I genuinely want to thank you, This video has relit my once dead interest in ML, I would love to see more videos from you on this topic or least get some recommendation on where I can learn more.

[@SebastianLague]

Thank you, that’s wonderful to hear!

[@AkshatSinghania]

Exactly what i wanted to say , This is the first time , I actually got to learn about the math behind machine learning and not just working with some ml library and get to know the true essence behind it. Great videos as always

[@hepcat93]

The story of not liking doing something just in the sake of doing this very something without thorough explanation is literally my school story with trigonometry and derivatives :D I couldn't understand it back then, but no one cared to explain instead just telling me to focus on the process itself.

[@pascalschopper2817]

Exact same situation here. Coding is fine. Maths....not so much. Certainly a bit of a lack of talent but school wasn't too helpful with the usual attempt of explaining the what without the why. @Sebastian, your videos are by far the best source for many of the topics i'm engaging in.

[@pixboi]

My exact sentiment also!

[@CYON4D]

The production quality is off the charts. As a software developer I can't even imagine the amount of hard work, research, technical knowledge, expertise, patience and determination this must have required. Hats off to you :)

[@SebastianLague]

Thank you!

[@TAP7a]

"I'm bad at naming things" There are only 2 hard problems in computer science: cache invalidation, naming variables, and off-by-one errors

[@bingusdingus8239]

Segmentation faults 💀

[@rube9169]

I see what you did there XD

[@NStripleseven]

dang

[@hhhpestock951]

Truth.

[@Temari_Virus]

0, 1, 2. Thank goodness, I thought there was an off-by-one error there

[@mangoalias608]

everything else aside, can we all appreciate the incredible visuals all of Sebastian's videos have? the animated graphs, the visualizations, the explanations, its so pleasing :)

[@senftube2460]

Yeah, Its so smoooooth 😍

[@BlenderTimer]

I certainly appreciate it! Making good visuals is not easy!

[@BlenderTimer]

@@jitin4179 He shared a link to the source code on his community page.

[@To-mos]

I'm guessing it's a Unity3D framework he made for doing visualizations in the engine.

[@superturkey2458]

@@To-mos Only Losers call it unity3d.

[@Suburp212]

I wish somebody had explained calculus like this in school. Intuitive, descriptive, visual, simple, elegant. This content is marvellous.

[@thehollowknerd3858]

School's advertising is such a complex topic, but it's simply because they don't teach it well. I was able to learn it in eighth grade while I was struggling with 10th grade math. Schools are bad at teaching but don't let that hold you back from accomplishing what you want.

[@TheBoglodite]

Mfw Sebastien tricked me into learning calculus

[@JustinKoenigSilica]

We did get taught almost exactly like this in school. Most people didn't get it anyway.

[@tartarugabradipo6081]

Sebastian: "The End" Neural Network: "Oh, that's for sure a tractor"

[@saar5947]

You're probably not gonna see this message, but I want you to know you give me inspiration and motivation to learn, do and achieve more as i believe you do for many others

[@SebastianLague]

I’m very happy to hear that, thank you!

[@zeddoes]

True. He is the only channel I turn thr bell 🔔 on. It’s always worthy to watch the video

[@exildur]

@@zeddoes Same here, I'm subbed to close to 100 channels and this is the only one I have the bell notification on.

[@mulualemtekle6094]

my life would be very different if I hadn't came across his coding adventure playlist, I got into game dev and computer graphics because of Sebastian and I fall in love every day with the beauty of things in this realm

[@Malizma333]

I love how this channel focuses on quality over quantity, shame the youtube algorithm doesnt promote more of that

[@The14Some1]

25:18 OMG, and at this point I've completely realized the true nature of the derivative - why it becomes a slope function, why x^2 turns into 2x and so on. It was one of those mind-blowing moments of insight, which most of us have experienced at least once in our life. Thank you, Sebastian!

[@ET-yc4wb]

I haven't even learned calculus and this was my first introduction to it. To my surprise, I understood it perfectly and I'm probably gonna start self studying calculus before I study it in school.

[@lordleo8563]

same

[@To-mos]

The word you are looking for is epiphany.

[@The14Some1]

@@To-mos Ah yes, thank you.

[@n3ttx580]

I had the same when he was just going about adding and multiplying many values together, when it struck me; addition (and thus multiplication) is computationally A LOT less expensive than division, so adding/multiplying tens of numbers can be as fast if not faster than division. That's probably the reason behind all the ridiculous math - to speed up everything by doing what computers are best at: addition, so it can be optimized (in terms of algorhytms) furthermore without major performance hits.

[@blitzarsun]

This is the most intuitive explanation of machine learning I've watched. I hope you return back to it soon!

[@RabbleRousy]

As someone who watched a lot of these videos when writing my Bachelor's thesis on NNUEs (a specific kind of neural network for chess), I can safely say this is the best introduction to neural networks I've seen. Absolutely love all the visualization, how you start from the ground up but still include the calculus etc. Fun fact: my thesis was somewhat inspired by your Chess Engine video as well. I love your content, becoming a patreon now!

[@SebastianLague]

Thank you!

[@oinotnarasec]

I wanted to write my bachelor thesis on chess algorithms, but my professor told me it was too complicated… would you mind sharing your work? I would love to check it out!!

[@cvntdav]

I love every single video that you make, you make such amazing stuff! I wish you could bring back the "How computers work" series which made me discover my passion ★

[@SebastianLague]

Thanks, I'm happy you enjoy them! Might be a little while before the next computers video, but I do still have plans for that series!

[@cvntdav]

@@SebastianLague 😄

[@dkaloger5720]

I also really liked the how computers work series .It is definitely one of the fundamentals of computing I knew the least about ,along with many other people I assume .

[@MrMcbram]

In the mean while, if you havent already check out Ben eater's video's! He has some great explanations from transistor level logic gates, to how these can be used to create all building blocks of a basic processor and how it handles assembly code

[@oskar7063]

This video is looking awesome, do you have you're own animation library like manim or the one from aarthificial or Freya Holmer?

[@SebastianLague]

Thanks! I have my own library, although that’s a generous term to be honest, it’s more like a loose collection of scripts scattered across several projects (which I always have to hunt down and repurpose for whatever I’m working on). Really need to improve that some day! I have recently been experimenting with Freya’s (brilliant) Shapes plugin for rendering lines and points though, so some of the graphs and a few other things in this video are using that.

[@idophysics0113]

@@SebastianLague Been a huge fan for a long time! Would you be able to share more about your library? I've used manim a lot but I'd be very interested to see how you create your animations, even if you only show a small bit of it.

[@FlyingDominion]

This video has so much animation. Did it take longer to code the project itself or make the video?

[@nocturn9x]

I began loving this video the second I realized you had explained derivatives without actually mentioning them. I love practical approaches!

[@timothysnave]

Holy crap that car in the beginning even found like a line of best fit. Crazy.

[@hulmaji1695]

I cannot believe how lucky I am (how ALL of us are) that you sir exist. This quality, effort and precision put into these videos... Just wow... Thank you!

[@Kibito80]

It’s such a bittersweet moment watching your videos as they’ve recently come out because I know such great content with such level of detail takes so long to produce and it’s going to be a long and sad time until your next video comes out. I just love your videos man, everything you touch becomes gold, you make so many topics that are so boringly taught at uni seem sooooo exciting!

[@techyte8810]

Sometimes I forget this guy is real and not just a voice in my head teaching me everything

[@nrub]

Ok, the way you explained derivatives at 25:00 makes soooo much more sense. It perfectly presents why first derivative of x^2+ax+b is 2x+a. I'm now so angry this wasn't explained to me properly when I was in university. I also believe you are secretly a maths professor.

[@jamesmnguyen]

I would argue your Highschool teacher should've taught you derivatives. Unless, however, you took basic math in Highschool (which isn't a bad thing)

[@KnowledgePerformance7]

@@jamesmnguyen not everyone went to school in the same system you did friend

[@jamesmnguyen]

@@KnowledgePerformance7 I understand that.

[@jonsmit8134]

You can keep at most 4 new things in your brain at a time. Perhaps your lecturer explained tangents, secants, slopes, limits, derivative limit definition, specific derivative calculations, derivative dx notation, derivative prime notation, ect in too few lessons. Now that you are familiar with these things it is much easier to follow.

[@multiarray2320]

i dont think he is a math professor because he can actually explain it xD

[@avwie132]

Having worked extensively with neural networks some 10 years ago, I must say this is hands-down the best explanation I have seen for people who are new to it. Excellent visualisations and explanations. It is so great for someone to start working on the absolute basics (simple perceptron) and working up, instead of directly going to PyTorch of TensorFlow.

[@GumRamm]

Any reasonable person: “I will use one of the many existing Python libraries that implement backpropagation for me in an efficient and easy-to-use way” Sebastian: “I like writing C#” Jokes aside, very cool and impressive project as always. And of course presented in a stunning and intuitive fashion, keep ‘em coming!

[@wlockuz4467]

“I will use one of the many existing Python libraries that implement backpropagation for me in an efficient and easy-to-use way” I hate that its like 99% of online courses

[@weckar]

The reasonable person learned nothing.

[@notahotshot]

@@weckar The reasonable person learned you don't have to reinvent the wheel every time you want to build a wagon.

[@weckar]

@@notahotshot pretty good to know how a wheel is made for your first wagon

[@kazioo2]

@@notahotshot That's great until you need to heavily customize a wagon for a new kind of task, but those old wheels somehow don't work well with it and you have no idea why, because you never learned how these wheels even work. This is a very common problem in programming in recent years. There are so many programmers now who never learned basics (eg. they often don't even consider they operate on physical hardware with actual memory), new apps with the same UI and functionality we were using 20+ year ago start lagging horribly on a 10,000x faster computer than what we had back then. Oh and you better have a few GB of memory for those nice fonts...

[@sciencemathematics]

From just the first couple of minutes of your video, I was able to code my own working classifier. I love you build up everything from first principles and also show your first "naive" implementations before moving on to the more optimized versions. It really makes all the moving parts easy to understand. Excellent work!

[@jeremystott8188]

I've always wanted a great demonstration of programming in a similar way someone shows progress of machining something on a lathe/mill/forge/woodworking. Most other videos skip over "all the boring bits" when they get to some coding, but you have absolutely nailed it ❤

[@tz4601]

Might you be talking about "Stuff Made Here"? It's my only complaint about that channel. I mean, yeah, I'm a programmer so maybe I'm more interested in the programming bits than the average person, and I can also understand it can be difficult to coherently show code in an edited video. But a lot of his projects are something like 50% or more code, and he's a wicked smart programmer, so I wish he included more of it. On the extreme end of "demonstrating programming," Handmade Hero is a (very) long running series of originally-streamed videos where a professional systems game dev builds a modern, professional game/engine from scratch in C. (Really it's C++ but it's a very C-like subset of features.) Even if you don't watch the whole thing, the first ~20-30 videos are enormously enlightening w/r/t the fundamentals of lower level application programming. (e.g., I now know how you can use the Win32 API to open windows, handle keyboard/controller input, play sound, and so on.) Near the end of that first streak of Windows-specific videos he shows off some truly remarkable techniques for custom hot reloading and input recording/playback.

[@myavkat5954]

@@tz4601 661 videos with each video nearly 1 hour starting from 7 years ago. Just respect. I had the intention to build a game engine with C but then I realized it is much harder to write in pure C and I didnt wanna learn C++. I also realized that I was just motivated by all the youtube videos where people code their games/engines from scratch and had little idea how one works. So I dropped the engine idea for now (planning for later) and started to build a terminal text file editor with C. Even that simple project had gotten me so far (far in my own terms) that I had become comfortable with libraries, compilers, linking them, how to use them searching for old documentations etc. I dont know where I am going with explaining these lol. As for the recommendation you gave, thanks. I probably wont watch it because it is hella long and I like to learn hands on with trial and error, searching for problems on my own. But if I ever start to code an engine and I have a problem that I cant solve no matter what this will probably be my go to source.

[@wafje]

The series of 3Blue1Brown on neural networks is kinda similar and dives even more deeper into the calculus.

[@randy7894]

Sebastian :"Just open your favorite code editor, type in a few lines and there is your multiverse simulator with special effects" Always a treat to watch your "magic" man :)

[@gpazsilva]

It's awesome how, after even having classes in college about neural networks, I finally understood how they work *in practice*. I studied the theory and saw a lot of "for dummies" explanations about NNs, but people usually abstract the actual code from their explanations and this used to frustrate me a lot. Thank you so much for the explanations, Sebastian; your content is gold.

[@gsp_admirador]

best video on machine learning that I have ever seen, how is this even free? This is what the internet & youtube was made for.

[@ganymede242]

I'm in awe of how good this content is. Production values are superb: voice is really easy to listen to with clear diction and pleasing accent. Graphics clear, smooth, and helpful. Content is out of this world. The explanations are great. I've not seen someone cover the whole thing virtually from scratch and yet at no stage does it feel like we're getting bogged down.

[@YMandarin]

I love the visual explanations and the hand setting of the weights, its a really intuitive explanation for the networks

[@artiomboyko]

Calm voice, pleasing visuals, fascinating topic and intuitive explanations. What else would you need... I have actually just created my private playlist for videos I don't want to lose, with this video being the first one added there. Thank you so much!

[@JordanMetroidManiac]

Sebastian Lague, your videos might go down in history as the most well-produced educational videos on KZbin of all time. And it’s hard to even say that something beats 3B1B’s videos.

[@SamirPatnaik]

yes but there is another insane, underrated guy who, imho, exceeds Seb in terms of production value: braintruffle

[@jaideepshekhar4621]

@@SamirPatnaik What videos does he create?

[@codinghub3759]

@@jaideepshekhar4621 From what I got from a quick search, it seems to be focused on simulating stuff

[@bladekiller2766]

@@jaideepshekhar4621 he creates videos that no one understands even PhD levels struggle with his explanations, so it's everything about the production. His videos are very very high level Physics.

[@odomobo]

You're such a great educator/communicator. You explained gradient descent in a way that makes simple intuitive sense to me in about 2 minutes, whereas it's something I've only understood abstractly in over 15 years of trying to understand this stuff!

[@janjjd]

You´re such a gifted educator, I really appreciate the quality and the amount of work that goes into these videos, inspiring stuff!

[@fascher_]

I feel like I've learnt a lot, this must have taken ages to make!! I'm in awe of how you're able to visualise complex information, and explain it in simple language. Your videos are also super calming to watch. I think when it comes to maths, or learning new things in general, sometimes people can get a bit stuck or frustrated. So, a video that just states the information plainly can become a stress when someone is learning a new thing. I think your videos are amazing at both explaining the information and delivering it in a calming way that makes people receptive and curious to learn more. It's a skill and you do it very well, so thanks for sharing your passions!

[@charlieahill]

The caliber, quality and attention to detail in your videos is outstanding. I gained a better, and more fundamental, understanding about neural networks from watching this video that studying them for a whole semester at university. Thank you!

[@gregor-alic]

This is actually an amazing explanation of backpropagation, good job! I would recommend to anyone who wants to learn backpropagation to just take a piece of paper and a pencil, and derive those formulas themselves. Trust me, it helped me immensly when I tried to learn backpropagation a few months ago

[@stacklysm]

I couldn't use any other word besides perfect to describe this video. The visuals, explanations and overall progression were exceptional and on point, every concept contributed to the next one, nothing was rushed or under/overly exposed. I wish more people have the opportunity to find your amazing channel, because these videos are truly special.

[@proka1]

New to the channel, I see :) This guy is great.

[@ACorgi]

This is a really neat way of teaching this. I've already studied this stuff some, but I just love this approach. The way you're teaching this from the ground up, showing how to make little enhancements and the rationality behind them with each step. It's so cool.

[@RichardFredriksson]

This is a wonderful trip down memory lane; having learnt about this stuff at University. Your ability to visualize everything is amazing, thank you for making such lovely content.

[@vb0t429]

It's impressive what Sebastian can achieve with enough time and dedication

[@thephoenixsystem6765]

It's amazing what time and dedication can achieve... with enough Sebastian. ?

[@matthias916]

Don't underestimate yourself, you too can do these kinds of things enough time and dedication, the dedication is the impressive part

[@DynestiGTI]

Anyone*

[@aXYZGaming]

hello, V

[@raditz9676]

Whoever you are random stranger on the interwebs, I'm telling you: You can do the same! All you need is dedication, which helps you find the time.

[@conradrobinson7941]

You have never made a bad video. My favourite video in the notification box updates every time you upload, from your last video to this one. Anyway, point being, you make my favourite videos on this platform.

[@loganb2198]

This is the best explanation for neural nets I have ever seen. I have watched tons of videos on the subject and never learned anything, but after watching this video I now feel like I have a decent grasp on how they work. I am now inspired to go and try to make one myself. Thank you for your hard work Sebastian, your videos are amazing!

[@evanduffy1015]

This is far and away the best explanation of NN I've ever seen, and I took college courses on NN's. The included explanation of the calculus behind it was amazing as well

[@intangiblematter_misc]

Sebastian Lague is the only KZbinr who can make a 55 minute long video feel like no time at all

[@tagcopperlight]

Please continue to make hour long videos, it's so relaxing to watch !

[@ZaItan1]

I love these intuitive explanations, and appreciate the live demo testing. Drawing pictures of things distinctively *not* one of the labels might help identify issues to keep improving.

[@sibonyhugo5887]

Did I just watch an hour long calculus course and not only i think I understood everything and going to try on my own but wanted it to last 2 hours longer? My god this channel never fail to impress me by its quality.

[@Humberd01]

I wish I had seen this video when I was in an AI class back then. It would have been so much easier.

[@rodneydlouhy7409]

This Channel has been like Bob Ross for me. I may have very little idea how to code like him, but I love watching how excited he gets over his projects.

[@glumpfi]

Wow, this came perfectly in the right moment. I was just writing my own neural net in C++, but struggled on some points. You explained every single one of them in such a nice way :) Thank you so much!

[@Tommus1997]

That's a really nice way of introducing the neural networks. Explaining the different parameters and fiddling with them by hand + the visualization, before starting to explain how we can make our computer do the fiddling. Very cool idea!

[@hanzazazel412]

I never understood maths until I started attempting basic code. These videos are the first ones that actually make sense because I can properly visualize the maths.

[@philurname6575]

I've spent many hours learning about ML algorithms, with a lot of that time spent on MLPs. I've working with PyTorch and TensorFlow before. I thought I understood everything pretty thoroughly but just that first example of manually tweaking weights and biases and seeing how those affect the output graph showed me something new. You're videos are incredible and inspire a whole generation of programmers.

[@ryanchowdhary965]

And then there's me whos stuck on regressions for the past few days.

[@velocitygames9404]

This is seriously one of the best videos on neural networks I have ever seen. The amount of work going into these videos is incredible and inspiring! Thank you so much!

[@sammyboy1112]

One of the best tutorials I have come across for ANNs in the 2-3 years I've been working on them. Very intuitive, bravo. You must really know your stuff.

[@glorytoarstotzka2380]

I've been working on a basic neural network just because I wanted to learn this, I already knew the basics but it' so cool to see you working on these. This really helped cement the knowledge I learned.

[@skillplants]

I remember the poll you made about making longer videos. I really appreciate you listening to the fans, I absolutely love these longer videos. Keep it up!

[@balazskustos9166]

BTW, I think even the final challenge is "kind of realistic". Imagine if you were a newborn baby, who never really saw anything in this world, and doesn't know what things are, how things work. And imagine that you have a blurry vision like these low quality images. Also, you can only see static images, without any sound or context. I think this neural network makes a lot better job than any human being could ever do with the same information. Only it does it's learning/thinking a lot faster, so after a few seconds it's not really a toddler anymore, but you got the point...

[@RockuHD]

This video is fantastic! Most videos and tutorials don't explain it as well or as in much detail. You can really see how it started so simple and how it got to the end result! Thank you for clearing up this subject for me! I will try and write one based off the subjects in this video (not ripping the code ofc)! thank you

[@NamePointer]

This is by far the best explanation I have ever seen of Neural networks, not because it is particularly precise, but because it does a fantastic job at making the concepts of nodes, weights, biases, activation functions, gradient descent seem very intuitive, which is rare for such a theoretical subject. Outstanding job!

[@alexweinberger8925]

Props to you for not just using sklearn or PyTorch! You actually built the NN from nothing. That’s a dream of mine.

[@uncertawn]

Easily the best vid on neural network. I have been looking for a good explanation on haw neural networks work, but none of them explained it in such understandable way. THANK YOU SO MUCH

[@boklausen9583]

I've watched quite a few vids on NN creation and this stands out as just brilliant(!). I really applaud the exemplification through very easy to understand coding examples and learning demos. Combining this with the 3brown1blue vids on NNs is all I needed to get a good grasp of this topic (and will recommend to others). I, too, am a fan of (and subscriber to) this channel now! Thank you so much, Sebastian!

[@ET-yc4wb]

I know you're not gonna see this message, but I hope you know that what you do is amazing. I am still in highschool, haven't learned calculus yet, and even so, your explanation of calculus just made so much sense to me. Everytime I tried to learn calculus I could not wrap my head around what derivatives were or what it is meant to do, but I understood it in a few minutes just from your example! Thank you, Sebastian.

[@heyreefes]

Been learning a lot from you last two weeks. You are amazing

[@Sam_mcl]

This is one of my favourite videos on KZbin ever. Incredibly well-made and useful! Cheers, Sebastian.

[@experiment8230]

Truly a gifted educator, you had me captivated for an hour on a topic I already knew. Well well done

[@superkaboose1066]

How people are this smart just boggles me, I love how well you simplify things though, the little simulations are just perfect once again :)

[@BjarneSvanberg]

Lovely video as always! In your doodle of the helicopter problem, you might be testing it with a thinner line (or less opaque) than the training set. Thus if the intencity of the line is affecting the activation of the neurons, you might not get the correct result. This could be the case for the numbers as well. The training set with numbers seems to be drawn with a thicker line than the one you use to test it with.

[@pawepiat6170]

So, in theory, vectorization of the input would help?

[@bernardmcdonald5257]

I don’t often comment on KZbin videos, but I just had to say that you are one of my favorite KZbinrs. Your content is amazing, and you explain/visualise the concepts and code in such a compelling and entertaining way.

[@andrewkicha1628]

Just amazing visualizations that help to answer the most fundamental questions that popped up in my head. Thank you for this amazing video! I look forward to seeing more content like this on the channel.

[@dantelaviero7782]

i've had trouble for 5 years learning derivatives, dropped out of universities cause no one would explain to me why we use them, just how. Your channel is a blessing

[@polygontower]

Did you take the optional math course? It was probably taught there

[@tams805]

@@polygontower Did you go to the same educational institute as them? What's that? You don't know? Then who are you to make such a comment?

[@polygontower]

@@tams805 No. An educated guess. An educated guess. An educated guess.

[@ryanchowdhary965]

@@polygontower someone has learned something from grade 11 chapter 1 mathematics. Always take an educated guess.

[@RichConnerGMN]

@@polygontower i just want to know why you repeated that 3 times

[@Monkeylordz88]

Convolutional NNs are the key to creating good image-recognition networks (at least at the moment). It's pretty amazing how a couple of straightforward image transformations add so much information and allows CNNs to gain such an edge over traditional feed-forward networks. In fact, these convolution-like processes are even used by humans, such as line detection. Your videos and explanations are incredible, I would love to see you dive deeper into the amazing field of ML!

[@Flobyby]

I wonder if a simple luminance/chrominance encoding of images could already significantly improve the performance of a simple networks such as this one for the last exercise.

[@ruroruro]

@@Flobyby unlikely. Maybe, for really small networks it might, but in general feature engineering (manually preprocessing the data) is almost always useless for neural networks, because the network can learn the best transformations for each particular task on its own. Actually, that's kind of the whole point of deep learning - to avoid having to manually hard code the features.

[@stephenwestland942]

This is stunningly good. I love the visualisations.

[@ShikaIE]

omg this has got to be the best visual explanation of neural network. I have never seen anyone "touching" on how the training happen, it's like magic, but seeing how you have sliders to manually create boundaries is really mind-blowing!

[@vrcookingsimulator5458]

Hey man I think you would really find plate tectonics simulations interesting. It kind of fits well with your procedural terrain generation series and is extremely interesting to read about.

[@magic_pink_horse]

Even though I studied this for 2 semesters and most stuff was familiar to me, it was still great to watch such a masterfully, artistically made video. Well done!

[@michaeljburt]

I was expecting another standard coding video on setting up a neural network - but I was pleasantly surprised. This is an absolutely wonderful explanation of why we need biases and non-linear activations for neurons. Brilliant!!

[@ralph_d_youtuber8298]

This is exactly what I learned in my first year calc class. Without the fancy visualization. I feel like I understand it better now 😂

[@realryleu]

That graph renderer you made looks almost exactly like desmos with a dark theme enforces by the dark reader extension.

[@chanceslaughter3237]

absolutely fantastic video... As always has me wanting to go try some new programming projects i've been thinking about for ages... Thanks for keeping me inspired and entertained! (which by the way, you are the first person to ever do with calculus)

[@bumblinggimp]

A wonderfully clear introduction to machine learning. I've previously struggled to get my head round back propagation and partial derivatives, this video explains it beautifully. Thank you.

[@polimetakrylanmetylu2483]

It's a great video and I really appreciate the effort you put into visualizing those concepts :) I made a little note 6:50, you said that it doesn't make sense to change the size of input or output. For neural networks it would be impractical, yeah. However, for many other classifiers you can increase the number of inputs, that includes making logistic model to have a "bendy" decision boundary. You may for example add an input and make it be a nonlinear function of another input, like input one squared, sine of input two, euclidean distance between a data point and 0,0. Then you can train linear model on this augmented data and it will be able to have bendy decision boundary in input space. That's what eg. SVMs use and it's called a kernel trick - making a nonlinear problem a linear problem in nonlinear space

[@SebastianLague]

Good point, thank you!

[@WaluigiisthekingASmith]

The learn rate part reminded me of damped harmonic oscillation. For a damping coefficient that's too low it will start to oscillate around the steady state, if it's too high it will decrease but not very fast, but if it's critically damped it will return as fast as possible to the bottom.

[@samuelwerley528]

Yes these are very similar concepts

[@SuagulishusR]

Man, I love your voice it's so calming not only that the way you explain you're projects and videos is so spectacularly exciting and informative and I simply can't stop until I watch the video to the end. Or fall asleep due to the fact it is currently 1:00 AM :) please continue with these videos I enjoy them thoroughly and will continue enjoying them till the day I die.

[@KeygaLP]

This is BY FAR the best introduction to neural networks I've ever seen. The visualizations really helped. Thanks for the great content!

[@h4kku]

I like that you condensed a whole semester course of artificial intelligence at university into 55 minutes :D

[@randomizer495]

The way he slowly breaks the problems into smaller chunks was making even my small brain could digest it tho:D

[@aiden_3c]

I really really like how you start from a really simple example and start working up from that, it really makes a neural network a lot easier to understand

[@knicklichtjedi]

One of the best videos for explaining neural networks internal mechanisms without presenting them overly complex. Well done!

[@neelbanga]

Hey! I'm a huge fan, your videos have inspired the way I create content, my latest video takes a lot of inspiration from your style of content. I just wanted to say thanks for both your inspiration and great content!

[@Dysiode]

23:22 -ish I think it might be helpful to note that the derivative describes the rate of change (i.e. acceleration) of a function at every point, whereas the delta gives the rate of change at a single point. I'm at 31:37 and I had just remember that bit and found it interesting.

[@colinbrown7947]

Acceleration would actually be the rate of change of the rate of change, whereas velocity is the rate of change. And to be more mathematically precise, you're talking about the difference between the derivative at a point of a function vs the derivative of a function. A delta is always an approximation based on some small nudge.

[@AdrianHereToHelp]

This video is legitimately incredible. I genuinely believe this is one of the best explanation videos I've seen, across this whole site.

[@tapiomakinen]

This is the most comprehensive neural network video on KZbin. I have been tinkering with toy networks just for fun, and I am interested mostly in whys and hows of these systems. In this single video you gave me all the answers I have been looking for. Thank you very much.

[@dcode1]

That feeling when you spent months creating an AI that can drive a car (here is the video kzbin.info/www/bejne/g6iZeqp-Yttph7s ) and then Sebastian comes in and just casually drops in the intro ( at around 1:01 ) that he wanted to do "something simpler" and shows his version of an AI that can drive a car 😅

[@ashpun5998]

Awesome.(I haven't watched it yet though.)

[@ashpun5998]

And if it was uploaded a few seconds ago, how are there comments from 15+ hours before?

[@Extex_]

Was marked as private and he shared the link with some people before releasing it. Probably his Patreon

[@Prodrozer11]

@@ashpun5998 he probaly posted it already and set it on private after, probaly to edit corrections and stuff etc

[@oriyadid]

@@Prodrozer11 don't think so, his Patreon members get early access to videos

[@ashpun5998]

Thanks guys.

[@isaacgreen9495]

This is absolutely incredible. I wish I had had this video when I was learning to build neural networks. Definitely the best I’ve seen by far. Your visuals are incredible!

[@tobytheepic844]

I loved that you explained where the equations of the derivatives came from, amazing video. I just finished a course on this and this video covered almost everything

[@jonathanjiang6975]

This is probably the best (99.5% confident, 0.02% confident that it is second best) video/tutorial on neural networks I have seen. Your intuitive explanations coupled with showing the naive to optimal approaches is very effective. Definitely showing this at my school's computer science club in September!