16. Learning: Support Vector Machines

MIT 6.034 Artificial Intelligence, Fall 2010
View the complete course: ocw.mit.edu/6-034F10
Instructor: Patrick Winston
In this lecture, we explore support vector machines in some mathematical detail. We use Lagrange multipliers to maximize the width of the street given certain constraints. If needed, we transform vectors into another space, using a kernel function.
License: Creative Commons BY-NC-SA
More information at ocw.mit.edu/terms
More courses at ocw.mit.edu

[@alaaeltayeb5794]

Prof Patrick Winston has sadly passed away on July 19, 2019 rest in peace , the knowledge you’ve passed to thousand of students is your legacy and its forever thank you

[@sandorszabo2470]

Very bad news :-(( RIP.

[@haggar11]

A real loss Prof Patrick Winston has dedicated himself to give knowledge on AI.

[@nikaporozov]

Very sad news RIP prof ;(

[@DeepakKumar-uz4xy]

Oh damn...RIP

[@SuperIronwire]

Rest in peace!

[@sansin-dev]

Tremendous respect for any professor who writes out the entire math on board and does not use notes to do so.

[@pranavtagore]

my fluid mechanics professor wasn't that good. she would use books and notes throughout her lecture to write the equations on the board. however, she had a good way of teaching that made everything understandable.

[@user-ip9bm4mz1v]

31:39 Why do I need to find the maximum value of the L value? I was looking for the minimum value of 1/2 * |W|^2, but I don't understand why you're looking for the maximum value of L as you move on to L.

[@pulipakasrikiran9307]

@@user-ip9bm4mz1v1/2w2 has a constraint so you use a lagrengian multiplier alphai multiplied with the constraint and add it to the intial equation.You treat this L as a new minimizing solution to minimize the original equation with the constraint.

[@ChibuRawk]

​@@pranavtagoredid you really compared fluid mechanics equations to this basic linear sh*t? No, seriously?

[@gregmcmahan7420]

​@@ChibuRawkNot sure why you have to be a jerk.

[@afarehdokht1992]

I’m jealous of every single student in this class. And thank god i am alive and can watch this on youtube.

[@romanemul1]

why jealous ?

[@storytel3000]

@@romanemul1 coz he can't use the bathrooms there.

[@jjpp1993]

@@romanemul1 of a world class education

[@cratermoney6941]

I wouldn’t be jealous at all, you’re getting the same education for FREE

[@asfasdfsd8476]

Don't be jealous of any initial condition. In the end it won't matter. You will get there if you are resourceful person anyway, and if you aren't such a person then no initial condition going to help.

[@realisticlevel2553]

I've been watching a lot of MIT, Stanford, Harvard, Princeton lectures, but this... This was phenomenal, hands down the best lecture I've ever seen. Rest in Peace Prof

[@aneekdas3056]

Feel blessed to have attended his lectures live and work under his supervision. Rest in peace Prof. You will always be an inspiration to me.

[@joshrichardson4527]

I love gun soooo much

[@chrism3790]

I just came from Andrew Ng's ML course in order to understand SVMs better. I found something quite interesting. Andrew gets the optimization criterion at 21:49 from an altogether different place. He arrives at SVMs by modifying the logistic regression's cost function, and the optimization criterion emerges from the regularization portion of the cost function. He then explains why that leads to a maximum margin. In contrast, this professor starts by obtaining the margin width algebraically with the intention of maximizing it, and then explains why that leads to separating data. Pretty cool.

[@astropiu4753]

Same here. Now I'm trying to interrelate the parameters of the two approaches.

[@-long-]

lol same here, cheers. Great course from Prof. Andrew too but I couldn't understand everything so I was looking for alternative lecture

[@rogerioluizsi]

Same here!

[@ritayanganguly7786]

I'd argue this approach is working better for me.

[@fruitbaskets7984]

Which course are you taking? Thanks in advance!

[@homataha5626]

RIP ! Prof Winston!!! you inspired lots of ppl!

[@tobiasksr23]

:(

[@TomOtero1984]

:(

[@teenspirit1]

He dead? That's very sad. He seemed like a nice fellow.

[@Mutual_Information]

MIT offering free courses on KZbin in the early moves is the ultimate education move. I respect it.

[@mpm3363]

The historical part of Vapnik’s story is very inspiring.

[@mayurkulkarni755]

Best explanation of SVM on internet !

[@TheMix2ra]

It is good!!! , I found a better one: kzbin.info/www/bejne/iXmld55nirGBoqc

[@bunt7243]

This is why MIT is MIT. Good work Prof and Thank you to the team. We hope to see more lectures related to Machine learning and Data science from MIT.

[@immcguyver07]

pushpender pareek, yes. for about $50,000, they will give you access to a year's worth of additional lectures.

[@lobtyu]

+immcguyver07 Don't forget the ability to work with or for top researchers, the exposure you get to other amazing students with a wide variety of programming backgrounds, the clubs you can join to collaborate with these other students, being part of a pipeline that regularly sends people to silicon valley which allows them to pay off student debt within a few years, or being part of a pipeline that regularly sends people to top grad schools which is the only way to get a job in academia.

[@valken666]

He's always out of breath, as if about to die. That's very annoying to me. lobtyu - Just read the papers, you'll be a much better researcher and for free.

[@scikick]

Machine learning is one of the worst taught classes in schools today - lecturers who are too into implementations and don't understand the basics well enough themselves, don't have motivation to teach well, and overcrowded classes because everyone wants to be a data scientist.. Thank you MIT for releasing this gem into the public domain for millions to watch.. This was easily one of the best SVM lectures ever!!

[@mitocw]

Ackchyually... this is under a Creative Commons License (NC-BY-SA) and not in the Public Domain ...but we are glad you enjoyed it! =D

[@omgcoin]

Professor if you ever read this, THANK YOU. I was actually sad the lecture ended eventually. The world needs more teaching like yours.

[@mikejohnstonbob935]

damn! this instructor's lines are so damn crisp!

[@mcgil8891]

mike johnston Bob ikr

[@ritikjain4811]

Simply loved it! Don't have any words for the professor who taught the sophisticated concepts with such simplicity...

[@shauryasharma2865]

BRILLIANT. Massive respect for the knowledge and simplicity of the professor here.

[@axscs1178]

This is how things should always be taught. Patience, deep understanding and passion to teach. I wish I would've had a professor like him as a graduate student.

[@gutlesswarrior]

When more than half the comments are along the lines of "best SVM explanation I've seen", you know you've stumbled upon a gem of a lecture. Great work, I'll be checking out as many of your other lectures as I can because of this.

[@asinegaasinega]

what do you expect? this is the difference between going to MIT and going everywhere else. It's not the knowledge that's the difference. It's how the teachers are able to relate the material in a very palatable fashion, that's the difference. I sometimes rue in my old age what I missed (because I didn't go to a good school) because i see the difference in my own understanding of things compared to those that went to good schools. It's not that I am not capable of understanding. It's that they have an in-depth grasp of the material because they sat under the tutilage of people like this professor. That's the difference between MIT and coursera or any other school folks.

[@Iamfafafel]

it's annoying

[@akshatchhaya6121]

What a gem of an lecture, Trying to understand equations directly just makes you mug up somethings and like you never understand it fully. What I found is that when you actually run yourself through a simulation of what the inventor of the equation did and follow the footsteps than things start making sense eventually and you arrive at solution and you think huh that wasn't to hard. Rest in Peace Prof Patrick Winston world needs more professors like you really man. This is first video lecture of his I am watching and I can feel what a great man we lost!!

[@mohamedgamal-gi5ws]

RIP Prof Patrick this lecture is gold , Never saw anyone explain all the tiny details this smooth in less than an hour

[@bobeatschocolate]

This professor is very, very well spoken when it comes to explaining SVM's. Clear, concise, focused on one instance of one issue at a time... Many professors try to show you the entirety of the math while walking you through the conceptual ideas and it makes SVM's very difficult to learn! This man is quite the opposite! Great work! (And by math I didn't mean just showing the margins, graphs etc.. meant proof of the equations which two of my professors did in two separate machine learning/data science classes.)

[@OttoFazzl]

This lecture by Patrick Winston is simply amazing! His way of teaching is one of the most insightful approaches to highly technical subjects that I have ever encountered. I am so grateful to MIT for letting students all over the world to learn from such people as him. The lecture on boosting is also very good.

[@headeeg]

I am grateful that I was given the opportunity to participate in this lesson. I really put a lot of thought into getting the double problem of SVM really into my head. Prof. Patrick Winston was the one who made it click for me. It is sad to read in the comments that we lost a great teacher who helped to make the world a smarter place.

[@ahmedwesam7286]

You are lucky my friend, i wish if i could.

[@MrPrince750]

RIP Patrick!!!It is sad you are no longer with us. You are a great teacher..

[@juliogodel]

For the record..Ive started watching one lecture and now I am watching the whole course...Patrick Winston is a marvellous teacher and I wish to watch *everything* this guy has to teach. Are there any other courses he teaches? If so, please record and put them online!

[@nagamallika1982]

One of the best lectures I ever heard-methodical & extremely helpful!Thank you.I will definitely come back for more - appreciate this.

[@shashankaich7632]

One of the best videos on SVM, which also explains the Kernalization so well.

[@jacobgonzalez731]

Best SVM lecture I have seen. This professor does a great job of teaching the concept of SVM and the thought process behind it.

[@faizanbeg7356]

RIP Professor, the world needs more people like you.

[@BharathRamachandraiah]

10:23 the way he interacts with the student. so nice.

[@jeswinaugustine5384]

After going through many articles and online courses, i still didn't understood the idea of SVM clearly. This one is surely the best video on SVM available online. Thanks a lot professor.

[@superteam1]

I didn't really want to watch this video from how long it was and I just wanted to get a quick rundown on the topic of SVMs, but I saw the comments and decided to watch the whole thing and, my god, am I glad I did. What an incredible lecturer and he made the topic crystal clear. Anyone struggling with SVMs should 100% find 50 minutes to just sit down and watch this and you'll be so glad you did.

[@Saravananmicrosoft]

Amazing lecture, i watched almost 3 times back to back, its always gives you always refreshing thoughts. My honour to see this lecture thanks prof, still teaching so many students like me. You are simply great.

[@slavrine]

I wish my math prof had his sense of humor and conciseness! Maby I would be doing my math PhD now instead of coding

[@antonylawler3423]

Thought I would add to the voices of approval. I've just completed an elementary Machine Learning course (SVM wasn't on it), and have watched quite a few youtube videos, including those from Andrew Ng. The clarity of language, display, sequence of demonstration and speed of this lesson are absolutely spot on. Thanks !

[@jamespatrick5348]

Excellent presentation that catches the subtle nuances of SVM and the thought processes that went into its creation. Fabulous!

[@nuraddeenb]

MAN, that was good!! probably the best introduction to SVM available online.

[@balachandar2012]

Helped me with my Statistical Machine Learning class. Thank you Professor. RIP

[@georgedikos1424]

one of the most inspirational lectures ever. Gave me the same energy and motivation like my first courses at Engineering school trying to bring together Finite Element Methods, approximation theory and functional analysis and the code in machine language or fortran.

[@amarparajuli692]

What a beautiful lecture. Thank you, Prof. RIP Prof Patrick Winston.

[@algebra5766]

This is a brilliant lecture!

[@DivineAbhi]

Perfect explanation. so much better than anything else online

[@mohammadrezabateni3626]

He always teachs in a regular way, with a good sense of humor...you never get tired...you dont just learn algorithms, you learn how to think, how to innovate....Last year I watched 6.034 and right after the day that I finished the course I found that he passed away exactly one year before....god bless you professor , I never had an opportunity to meet you In real life but you were my best teacher and inspirator and I never forget you...rest in peace prof winson

[@pedroveloso62]

Really nice class. This professor managed to go through some tricky topics maintaining the simplicity and coherence of his argument.

[@Sumit-dn6ls]

Excellent! Simple explanation right down to the basics.

[@sarkercuisine8693]

Wow. Best lecture for SVM I ever watched. Thanks a lot, MIT OpenCourseWare and Patrick Winston.

[@spvimal]

He did the miracle of teaching things which others struggle with for 3+ hours. Wow, what a class? Miss you Professor. We needed lot more classes from you and all of them are in youtube ;)

[@mohammadashrafulislam7521]

Best lecture on SVM I have seen so far...Just loved the way he explained the concept and the functions! Gosh if I could just attend his lectures face to face

[@CKPSchoolOfPhysics]

This single video is much more powerful than all videos available on youtube about SVM. so, lucky to found his lecture. Simplicity of teaching at its best. Love you prof. RIP.

[@robertchen6104]

Like all students everywhere, I was watching this lecture and thinking, "if only I had had a teacher like Prof. Winston when I started in physics, I would have ...." Or at least, I would have had an easier time in all my other courses, and later in doing research, or just learning new things, like modern machine learning.

[@allisswellable]

This is one of the best explanation of SVM i have ever seen. This professor made this complex concept so easy to understand. KUDOS to him!!

[@sankopanza]

The best explanation of SVM I have come across. Hats off to Prof Patrick.

[@Cyphlix]

One of the best lectures I've seen, so concise and easy to follow :D

[@judedavis92]

"If you can't explain it simply, you don't understand it well enough." ~ Einstein Professor Winston clearly understands the topics he teaches.

[@BrandonRohrer]

Excellent explanation Professor Winston. You have the rare skill of explaining both the math and its motivation clearly to a novice audience.

[@alexanderkurz2409]

One of my favourite math lectures on the internet. I probably wrote the same comment some years ago, but here I go again. Thanks to Professor Winston and everybody else who made this available. I do teach math myself and I deeply admire how he boils it down to the essentials without leaving anything important out. Just looking at how little there is on each board and how clearly the beauty of the subject shines through ... a true master class. And I always thought we should teach more math history, so it is great to hear from him how the ideas actually developed.

[@solomonleo3025]

12:40 , that guy just saved me from suicide, I was like, "wtf, where did that w vector disappear!!" 😂😂😅😅

[@amitgp2007]

Me too......

[@phuthangmaponopono1015]

Same here lol

[@nkundukozerajanvier162]

Amazing lecture. Thank you and MIT in general. we love your priceless support to global education

[@henokgebeyehu1507]

The power of the rewind button in learning is actually phenomenal!

[@saitejabobby9990]

That story about how SVM evolved gave me loadz of motivation. Great ideas always takes some gap and then immortalizes.

[@alalize]

The Caltech prof' Yaser Mofasa explained it another way (more mathy), but this is clearer.

[@eVul6]

I was watching the video and thinking "Wow, the pace of the lectures at MIT is pretty fast. These students must be really bright to follow the professor. No wonder that I'm not studying there". At the end, I found that, unbeknownst to me, I was watching it all along at the 1.25 speed.

[@mcgil8891]

eVul6 OMG... Thank you!! I just realized I was watching it at 1.5x

[@Ash-cc1uj]

i did the exact same thing

[@da_lime]

I am watching it with normal speed, I guess I need to set it on 0.5

[@michaelmarcic9636]

Thanx for the commend. I watched it on normal speed, but thought that this guy is very slow, so I put it on 1.25. now it's fine.

[@veerpal5913]

eVul6 donkey

[@matthewrussell7802]

Someone give this man a medal. Pure brilliance. Thanks for sharing.

[@mohankrishna7979]

Excellent explanation of SVM. Awesome job by the professor. clear, concise and has a story flow

[@billwindsor4224]

Wow, that is an excellent lecture on SVM, thank you! Hang in there until the "miracle" part that starts at 43:30; then he shows the transformations that make SVM amazing.

[@yuriaurelio810]

Wow. This guys is the best teaching SVM.

[@MsVanessasimoes]

I am very thankful for all people that worked to bring this amazing lecture from Prof. Patrick Winston to people around the world.

[@putuaditya3741]

Fortunately, I found this video before my thesis defense tomorrow. Thank you so much, Prof. Patrick Winston. Keep up!

[@mmattb]

That is the best chalk I've ever seen.

[@krakenmetzger]

Note for myself and others: the reason in English why (dL / dw)[(1/2) |w|^2] = w is because dL/dw is a directional derivative. Equivalently, we are rotating the coordinate system such that the w direction is an axis, and taking the partial derivative with respect to w. We can now treat |w|^2 just like x^2 if we're doing normal calculus, particularly because |x|^2 = x^2 for all x.

[@jagannathan1014]

Thanks a lot man i was confused but went on with the lecture since i didnt want to get distracted, i went in the comments anyway and saw this within a single scroll , You dropped this: 👑

[@anushka.narsima]

omg thank you so much, I've been looking around for the past few days to get past that step

[@Iamfafafel]

i guess dL/dw means consider the vector of partials (dL/dw^1,...,dL/dw^n) where w=(w^1,...,w^n). i can't really make sense of your comment

[@nkdms.2031]

The "widest street approach" ... oh man! Perhaps the only lecturer that can throw "gems" like that + the story at the end... In three words he explained everything!

[@deltax7159]

what an amazing lecture and professor. This is what its all about. Some people are just blessed lecturers and he is for sure one of those people.

[@tgowda]

great lecture! Thanks MIT OCW

[@sridharaddagatla]

amazing explanation !!! Most other tutors skip the algebra part which makes learning SVM a black box but this delineated explanation of prof patrick is amazingly simple and thorough. Thanks Prof patrick and MIT opencourseware.

[@chitmingyip1311]

This is my first comment on KZbin ever and I want it to be dedicated to this video. Very informative and clear. I missed some points though and I am going to resit this course. I salute to you, professor.

[@andreidumitrache2077]

Perfectly presented and explained. The best lecture on SVMs I've seen.

[@junecnol79]

i watched several times back and forth, finally, i THINK i understand

[@kellyli1920]

best explaination! I saw many materials that is very hard to understand!

[@sleyking514]

hello

[@vijayakumark5190]

I pay my sincere thanks to the professor for an extraordinary lecture. Amazing. Good Teachers are the Gods.

[@yuwang6841]

professor is good ,the formula goes step by step ,very clear ,it's wonderful to watch this lecture with the paper:A tutorial on support vector machine for pattern recognition!

[@yassineouali1888]

What a professor, may he rest in peace

[@cheeloongsoon9090]

For everyone watching, note that there is a mistake on the board. At 19:20 , a student asked a correct clarification. w dot xPlus should be (1-b) , whereas w dot xMinus should be (-1-b), then you can get the 2/norm(w) equation.

[@sajay96]

It's written as 1+b because negative value of w dot xMinus is considered, so not a mistake.

[@cheeloongsoon9090]

@@sajay96 I guess he corrected it at 19:53

[@mathhack8647]

I came here for few minutes just to consolidate my SVM's Math foundation , and I found myself Stuck to the lecture till the end with a pencil and sheet of paper to take notes. > Amazing professor . I just learned he passed away . His contribution for those who are thirsty for learning is tremendous . الله يرحمه ويحسن إليه.

[@mrodek]

This is an absolutely amazing lecture. Soo much goodness and wisdom. RIP Prof. and thank you.

[@meghnanatraj3360]

The best SVM lecture ever! Thank you soooooo much!!!!

[@amineech-cherif2386]

Do you understand all of it ??!

[@hunir1]

I understand all of it in-fact this is really a basic intro, you shouldn't have a problem with this. If you do I suggest pausing the video at each stage and clear up on points that you feel you have understood. It has only been algebra and calculus.

[@amineech-cherif2386]

I agree that this intro is very easy to follow, but it is too abstract I think. Like for instance the mathematical conveniences, for example when we divide the W by 1/2 is not clear. Also, a thorough understanding of quadratic programming is needed to fathom the optimization part of the SVM. Simply put, this lecture does not cover the entirety of SVM.

[@meghnanatraj3360]

I guess it depends on how much you know initially. (beginner to advanced). This caters to the middle. Who have know some basic ML math and yet are new to ML concepts! Like me! :) So i guess its just perspective and he cannot cater to everyone in the audience! And no, I didn't understand every bit of it. ^_^

[@amineech-cherif2386]

Exactly. In my case, I had to study some of the basics of Calculus 2, like the Lagrangian, as in my computer science dept we don't study it.

[@junzhemiao7118]

Best explanation I have seen so far. Much better than Andrew Ng in my opinion.

[@modusponensthethird]

i have an exam tommorow in india and Prof Patrick teached me what my indian प्रोफ़ेसर couldnt teach me in a whole semester. You sir saved my life...

[@jorgejgleandro]

Cristal clear explanation! A good professor is like this: a bridge (instead of a wall) between the student and the matter. Congratz!

[@ChadieRahimian]

A combination of this lecture with a 10min lecture on SVMs by Victor Lavrenko worked amazing for me!

[@OttoFazzl]

I went to that video and found it really useful, thanks for sharing.

[@venkateshsv7434]

Shadi Rahimian .. I really don't know what is this.. :-( I. very basic

[@alifawzi4566]

thank you shadi itis god advise for victor video

[@abdullahalsaidi6009]

Thanks alot , his video was very useful

[@saliheenafridi9116]

Thanks for telling us

[@sreeganeshvr7561]

12:42 Thanks, Brett, whoever you are. Panicked for a few minutes until you chimed in 😂🙌🏾

[@bitbyte8177]

lol same.

[@VikasKM]

This is by far the best lecture on support vector machine. just amazing lecture- a must watch

[@gmarciani]

The best lesson on SVM that I've ever heard! Thanks for sharing!

[@hussamcheema]

Brilliant Lecture ... Thank You :)

[@daripadaiseng]

Thanks a lot for sharing this. Now I understand the equation. Maybe tomorrow I will forget about this though, lol.

[@WarnerBrosWannaB]

lol watch it again!

[@jianingsun8048]

Best explanation ever! Can't believe he explained all these complicated things (SVM and kernel) in only 50 mins! (I took 90 min in this part of ML lecture in our university but that made me very confused). Thank you so much! prof.

[@hahablahblaah]

This is an incredibly interesting lecture! Had to watch it twice and do some back and forth to fully understand, but really well explained!

[@MelvinKoopmans]

Very good lecture, clear explanation and good pace :) One correction: 44:30 (u*v+1)^n is a polynomial kernel, not a linear kernel.

[@girrajjangid4681]

put n=1 generally, this equation is linear and the value of 'n' denotes the dimension

[@jackdaw205]

"This needs to be in a tool bag of every civilized person" Oh wow, at MIT they have a very specific idea of what 'civilized' means

[@Islam101_Uganda]

😂😂😂😂

[@shapedsilver3689]

For a second I was going to try to defend them but honestly, I think they kinda do

[@teenspirit1]

Let's imagine that everyone knows how to separate pluses from minuses optimally. The world would be a... I guess it would be the same.

[@i486DX66]

He stated necessary conditions. Not sufficient.

[@thedarkknight579]

These lectures on KZbin make world a better place to live in 🥰🥰🥰🥰 Thank you for providing us the content MIT-OCW

[@flxblyyk]

This is the best SVM lecture I have ever heard! Everything is so well explained, so that it helped my ambiguous understanding to be clear. Thanks a lot for sharing your knowledge. RIP