12a: Neural Nets

*NOTE: These videos were recorded in Fall 2015 to update the Neural Nets portion of the class.
MIT 6.034 Artificial Intelligence, Fall 2010
View the complete course: ocw.mit.edu/6-034F10
Instructor: Patrick Winston
In this video, Prof. Winston introduces neural nets and back propagation.
License: Creative Commons BY-NC-SA
More information at ocw.mit.edu/terms
More courses at ocw.mit.edu

[@tommytan8571]

Rest in peace , professor . He died in 2019 , let us remembered him by watching this again and again.

[@omgcyanide4642]

No way

[@jeffreyanderson5333]

Let's push this to a million view

[@marcogelsomini7655]

The great explainer

[@avibrarbrar]

Maybe understanding it and doing something worthwhile with it.

[@sshaikh8104]

What a shocking news i just read, its feeling like some one from my own professor 😕 I am extremely sad

[@NeuralxAi]

I am From a village in Kashmir. We Don't Have Teachers That Can Explain Things on this Level And i Totally depend on These Great Teachers in MIT. Lot's Of Love Sir, I wish I could Get You Subscribers from my Whole University. I Can Only Say Thank You So much for Quality Educations .

[@chakibafraoucene397]

he passed away :( last month

[@NeuralxAi]

@@chakibafraoucene397 RIP 😓

[@sozejigar1326]

Koshur here too. Tuhund comment os on top of the list.

[@ahmedmoneim9964]

Thanks MIT for making these lectures publicly available, it is simply great!!

[@vinayreddy8683]

Ahmed AbdelMounem don't built a bomb with the base of this lecture

[@StingBolt]

@@vinayreddy8683 i wonder how idiots like you came here

[@OttoFazzl]

This professor is amazing! His explanation of SVMs was one of the best and clear I could find on the Internet.

[@gaurav63105]

I also started with SVMs and then decided to see his other lectures,he's so crisp

[@alexm5914]

I'm watching SVMs right now, and I think I might do that too...

[@binoruv]

Me too!!!

[@magnumalba]

It is not "This Professor". It is one of the fathers of AI.

[@ankitasahoo668]

I too agree

[@muhammadhamzahm1204]

May you live in peace professor Patrick! You're a giant in field of machine learning. Your these lecture are biggest asset that beginners can use to climb. Thanks

[@pandatobi5897]

rest in peace* now. he's dead.

[@psrajoria]

"All great ideas are simple. How come there aren't more of them? Well, because frequently, that simplicity involves finding a couple of tricks and making a couple of observations. So usually, we humans are hardly ever go beyond one trick or one observation. But if you cascade a few together, sometimes something miraculous falls out that looks in retrospect extremely simple." - Prof. Winston

[@juliogodel]

This is just great MIT. How I wished you could upload All classes from prof Winston.. I could keep watching them for days. Clarity and straight to the point. Marvelous!

[@dr.mikeybee]

I really like this course. When a Professor understands the material. it can be clearly explained, and Professor Winston really understands the material.

[@maffixwilliam5471]

very true

[@kutilkol]

8:55 disclaimer . there exists also neurons connected directly without synaptic gaps as proposed by Camillo Golgi. so both Cajal and Golgi were right. RIP prof. Winston, beautiful classes, thank you sir

[@EranM]

Patrick writing on the blackboard is ASMR to my ears :>

[@bradjones2071]

I agree. Everyone always assumed MIT professors will just leave you with there intelligence and not be able to connect with the average lay-person but that is an incorrect assumption. I can basically understand alot of what he's talking about and am glad for the video.

[@RobBarter]

Just happened upon this youtube video and begun watching it as have a passing interest in Neural Networks.....then realised I recognised his name. Looked up and pulled down a book I bought back in 1992 (not opened in years), Artificial Intelligence by Patrick Henry Winston. Sorry to hear we've lost him.

[@Nestorghh]

world-class professor and lecture.

[@jvanrs4928]

Thanks MIT, initiatives like this can truly spark innovation

[@adityanarendra5886]

Prof Winston, your explanations of AI have always fascinated and inspired mw in to the field. Rest in Peace professor.

[@soulysouly7253]

Holy shit everything is so clear. I also frickin love when he explains very simply why we use a that one specific function, why we square this, why do we divide that, where does that coefficient come from, etc... and it all makes so much more sense than the gibberish written on the slides that I have to decipher every lecture.

[@shatandv]

I'm loving this course

[@redaelhail9877]

Thank you professor Patrick ! you had an extraordinary simple explanation for complex principles ! Thank you MIT for sharing this incredible content.

[@irazt]

I wish I could have taken these courses in person. Thank you for sharing your knowledge to the world professor

[@OhhBabyATriple]

Winston is the best AI lecturer

[@OttoFazzl]

This.

[@Schroeder2424able24]

wow, had me fooled, he's so lifelike

[@guywithaname5408]

thelastphysician underrated reply

[@JG_1998]

Rest in Power Dr. Winston.

[@balllaktomas]

It's sad that in our school we had lecture for this and I was lost but I think teacher was too. And than this guy comes with all elegance and no arrogance providing you this information and let it share too people around the world. WELL PLAYED.

[@montserratcano2389]

Thanks for sharing MIT! Excellent teacher!

[@rustycherkas8229]

Great lecture! Lucid with moments of humour and humanity. Thanks MIT.

[@SharathPunreddy]

Loved it, thank you very much for making complex things so simple.

[@backpropalgo]

amazing content. I miss real blackboards like this. I have to admit that the prof looked to be struggling a bit. I heard he passed away, so I would just say thank you for a really great session that I have shared with everyone in my own circle that had questions about how the foundation/basics of modern AI work

[@jvwdigital]

2 years later and this is still a great lecture. Amazing instructor. I actually watched the whole thing. Simple ideas only take a quarter century to find. We humans need to make more observations, put them together, and see what shakes out.

[@ryanalopez]

Good in depth mathematical explanation of neural net components. If new to learning about neural nets, I'd recommend watching a few other videos first which cover the overall design goals of neural nets, how they work at a high level, and the outputs they are trying to achieve before jumping into the mathematical models used to describe errors and performance.

[@TheZudork]

Thank you for this amazing class!

[@bidhanmajhi]

He explained it very well. Sadly he's no more RIP

[@xXxBladeStormxXx]

To think that just as back as 2010, they thought Neural Nets weren't worth spending much time on and now the instructor, I'm guessing, felt compelled to update even the ocw playlist to include these videos, should give everyone an idea of how good a time it is to be studying these topics. In the course of just a few years, deep neural nets have become extremely relevant again. It's indeed a great time to be studying Artificial Neural Networks.

[@limitless1692]

we are at the start of AI age being first here is a edge

[@user-ol2gx6of4g]

we were at the start of AI age since 1950s.

[@MrAlipatik]

wake me up when they create tiny computers in a chip, that can be able to calculate simultaneously, and all hell break loose.

[@ragy1986]

It's the best video on NN on youtube, bar none!

[@bohusb.6879]

This professor is amazing. His lectures are so clear and the same time he goes really deep. Very well structured lectures.

[@maffixwilliam5471]

Thanks MIT for making this lecture public. The Lecturer explained the concepts, which makes it very crystal clear. Thanks. btw rip to the lecturer. done an honorable thing to the world. am benefiting from his work. thnks again to him and MIT. Keep up the great works please.

[@sharifk9860]

What an amazing lecture! I have seen many neural network lectures. This one is by far the most comprehensive and easy to understand. I instantly fell in love with prof. Winston. I hope he is now teaching God and his angels.

[@benjaminhardisty66]

Sweet lecture! This stuff finally makes some good intuitive sense ;)

[@willroman3595]

We live in such an awesome time that this information is available to everyone, free of charge.

[@jonelya]

29:26 the best ever explanation of chain rule..thank you so much

[@dostoguven]

amazing teacher.

[@keskinaytac]

Thank you for the subtitles.

[@OnionKnight541]

that was fantastic. at the end, he says, this miracle was a consequence of two tricks plus an observation. and, all great ideas are simple and easy to overlook.

[@nikre]

what a privilege to be a student in this class.

[@AlwaniAkber]

Though I am not good in math but few of the explanation really make sense ..great professor and video

[@mikeschmit6474]

Just a minor correction at 4 minutes. That is a ring-tailed Lemur, not a Madagascar cat

[@zekeanthony]

superb prof Winston

[@yusuferoglu9287]

RIP Sir!

[@qzorn4440]

a very relaxing lecture, this makes me think of deep learning programs. thanks.

[@NisseOhlsen]

q zorn or maybe deep sleep?

[@thevirginmarty9738]

Awesome course. Someday I will use this to build a robot girlfriend. Thank you!

[@robl4836]

You need a Robot first before you can build it a Girlfriend ;)

[@23Ather]

You need both the robot and the girlfriend to find the minimum of the cost function. (robot - girlfriend)^2 ;)

[@koushik7604]

:)

[@vijayd8634]

Funny, the cost will be half of it!

[@peterkay7458]

When you get it working please make the CAD files available online. PLEAZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ jk

[@heri_prieto]

This was beautiful.

[@tuha3524]

yes, yes, absolutely agree with Professor. "hardly ever go beyond one trick or one observation."

[@ibadurrahman5954]

Thanks for this lecture it was amazing .

[@Ludiusvox]

Right now I am studying a Lexus ES350 Air Conditioning system, and Neural Networks are part of the A/C controls. Not being able to find any resources on it at the school this lecture is very useful. I might add, MATLAB deep learning toolkit is useful also.

[@tuha3524]

I love this course so so much. Exellent!!

[@radsimu]

this nicely explains some of the mathematical decisions of nn models. really good stuff!

[@AdrianVrabie]

hey Radu! I dunno what you are referring to when you say "mathematical decisions" but I agree with it that it's awesome stuff! Btw! You've also done some nice stuff with NLP in Romanian! :) You should contact me and give me the code in Java maybe I can continue in the free time to do some stuff too! Kudos to you in advance! :) (ce lume mica!)

[@radsimu]

Haha :). Will upload it all on github some day. Need to make it more tidy first. Will keep you posted

[@AdrianVrabie]

Radu Simionescu adauga.ma te rog pe facebook ca nu te gasesc. adrian vrabie

[@SuperMaDBrothers]

amazing lecture good points at the end on simplicity

[@KSayar]

Excellent teacher.

[@daniyalali6016]

learn a lot about neural nets from this video course.

[@kabal127]

Best course ever

[@brambeer5591]

Cool guy, awesome lecture!

[@somaprasadsahoo2446]

How the performance function became -1/2 (d-z)^2 ? 28:08

[@chuvaca189]

Gracias MIT con la colaboracion de finis terrae :D

[@tsvisabo731]

What an awesome teacher

[@fraollemecha]

Awesome course. Someday I will use this to build a program that writes programs.

[@hassananwer3674]

50:02 "All great ideas are simple"

[@Yomama4536]

But not all simple ideas are great...

[@drewlaino]

That P at 16:35 was amazing...

[@maoqiutong]

Between 46:00 and 49:00, dynamic programming also uses similar concept to avoid exponential blowup. Maybe back propagation is also a kind of dynamic programming.

[@anandagrawal2497]

This is art

[@mathhack8647]

@26:05, I ike this philosophy. RIP Dear Winston, Your coursers are stil used by students and perpitual leaners, like me , all over the world/ الله يرحمك ويحسن لايك بقدر ما نفعت طلابك وعموم البشر

[@mdtowhidurrahman8406]

I am not sure if it's me or others who feel the same after the pandemic. I feel disturbed and lose focus as soon as the students start coughing in the background. The pandemic left us with a mental phobia.

[@MICKEYISLOWD]

Go look at climate change if you really want Mental phobias! It's shocking. The acceleration of change is scary as fuck. Just 10 yrs from now and economies will begin falling.

[@5hawnK3lly]

really impressive drawing skills i must say

[@bendev6807]

Great lecture. Enjoyed it a lot. RIP Prof Winston.

[@alv2648]

at 4:10 seems he misspoke about misclassified examples by Geoffrey Hinton's U Toronto NN. Appears the right answers (aka labels) are shaded red (second choice for the first two photos). Labels are set by the researcher for the training set - so they chose cherry instead of dalmatian in picture #3.

[@i890ola]

Thanks from Syria 🇸🇾

[@myroseaccount]

This wasn't overlooked but buried by Marvin Minsky in 1970 by his book Perceptrons

[@perrydeng6960]

Backpropagation starts at 26:25

[@IvandroidYT]

Awesome! he explains really good

[@chetjuall2269]

Great ending beginning at 50:00

[@strings1984]

Seems like in the biological model the hill climbing is done by the physical architecture and the pull on the axiom path by the surrounding associated stimuli's, the added advantage of this pull is it lets us know where to head towards when the solution isn't fitting the question.

[@bobbejaan_music]

Problem i have is that if in = 0 then the weight of that in does not change because its weight change depends on its input. (pd sigmoid in / pd w) = in where in = 0. I think weights should change if there is an error.But if out = 1 and in = 0 then w1 does not change.

[@KaiyuZheng]

I don't quite get the last point: the computation with respect to width is w^2 (width squared).Can someone explain?

[@Dennis4Videos]

1 year late but to whom it may concern: it is because you can cross-link the neurons hence w^2

[@nitinsiwach1989]

at 41:00 .. Starting off with weights being the same would not necessarily mean they remain the same. it would if they were in same layer but here the neurons are not.. am i missing something?

[@dnyaneshwardarade6120]

I only dream of sitting there and watching the professor

[@aoweishen3496]

Can you please build a full playlist of this course? Cuz it's really good but i don't know how to find the rest of the course. Thank you!

[@mitocw]

Here is the complete playlist: kzbin.info/aero/PLUl4u3cNGP63gFHB6xb-kVBiQHYe_4hSi

[@devonallary5251]

@24:30, shouldn't the weight for w0 be 1 instead of -1? Then, as long as the sum of the other inputs is greater than 0, they will always pass the threshold since w0 + SUM(w-0) >= T - -> Sum(w-0) >= T - w0 - -> Sum(w-0) >= 0.

[@andrii5054]

I agree, thought the same thing

[@michaelredenti2054]

The fact that the derivative of the sigmoid function is given exclusively in terms of the input/sigmoid is not that surprising since the sigmoid is a function of the exponential function whose derivative is itself.

[@fulliculli]

Awesome video content. Just make the sound louder please.

[@MinusBrain]

Thanks for uploading such an awesome lecture. One point I did not get, though: Could anyone please explain what i and j are in the function to calculate the delta of the weights at 21:24 ? Did I miss where the professor explains where this comes from?

[@aidenigelson9826]

I assume i and j are for x and y values respectively, 2i + 3j is the coordinate of x equal 2 and y equal 3.

[@kjyu]

@@aidenigelson9826 I am pretty sure i and j are unit vectors, that space had only two dimensions w1 and w2 so the unit vectors looked like i =[1,0] and j=[0,1], so yes you are correct

[@aidenigelson9826]

@@kjyu it's a pretty long text for saying yep, I imagine you thought I was wrong, wanted to say sth then read again, found out it's correct but was too lazy to delete it hehehe

[@Jirayu.Kaewprateep]

From his example, how much initial random value create BETTER results since too wide create time approx because approach algorithms or because time widely scope⁉️

[@prinzrainerbuyo3234]

It's 'Fall 2105' in the description

[@tthtlc]

You mentioned 2010 as year when NN is nearly dumped. I tooked an AI course in 1990, and by end of 1990, have convinced myself enough that the whole idea is too probabilistic, and unlikely to show much intelligence superiority, preferring the algorithmic approach instead, and subsequently gave up the subject totally. Well, I was wrong. :-)!!!

[@rustycherkas8229]

You think you've got problems? I was the SysAdm at the UofT during the late '80s who set up Geoffrey Hinton's terminal in his office, and, not knowing any better, turned and asked if he needed any 'training' on how to send/receive emails... How was I to know that he'd become the "grandfather of AI"??? *sob*

[@avawinters6184]

ok, amazing lesson and all, but where do I get one of these chalkboards?

[@daoxiongteng1591]

Good job with the cardio!

[@michaelsu4253]

22:55 "Sadly in Harvard" in 1974 gave us the answers. This makes my day😂

[@RonMartinJr]

Brilliant

[@Velvels]

Excellent lecture by Prof Winston. Can someone share the link to the tool he uses to demonstrate neural net in action ( what he calls "World's smallest neural net in action" )

[@musictest9999]

that was intense

[@josetzo5599]

gracias yanine :)

[@peterkay7458]

Damn this guy is good! wow

[@cagmz]

Does anyone know where the 1/2 comes from at 28:00?

[@ThomasFauskanger]

I think it's just to make the derivative nicer. He uses the derivate at 33:30 , and is just d-z and not 2(d-z) as it would've been otherwise. I think one of his points in other videos is that it's about mathematical convenience. The performance function is arbitrary and can be adjusted to "be nice".

[@PullingEnterprises]

It's however long you want your approximation step length to be. That is, if the optimization function is -1/2 then every step you'll reduce how far you were off (d-z) by half. If it was 1/3 then our approximation would be dividing the off distance by three and traveling just that far. The (d-z) term is how much you were off from the right result, and the -1/2 is just the step size to adjust (iteratively) until your gradient descent is within a threshold to give you the outputs you want while training your network.

[@WhoForgot2Flush]

Makes taking the derivative easier. You don't need it, you'll get the same result it just makes the math easier.

[@WepixGames]

R.I.P Patrick Winston