Second Order Optimization - The Math of Intelligence #2

Gradient Descent and its variants are very useful, but there exists an entire other class of optimization techniques that aren't as widely understood. We'll learn about second order method variants, how they compare to first order methods, and implement our own in Python.
Code for this video (with challenge):
github.com/llS...
Alberto's Winning Code:
github.com/alb...
Ivan's Runner up Code:
github.com/Pia...
Please Subscribe! And like. And comment. That's what keeps me going.
Course Syllabus:
github.com/llS...
More learning resources:
web.stanford.e...
www.cs.toronto...
www.quora.com/...
en.wikipedia.o...
• (ML 15.1) Newton's met...
• (ML 15.2) Newton's met...
Join us in the Wizards Slack channel:
wizards.herokua...
And please support me on Patreon:
www.patreon.co...
Follow me:
Twitter: / sirajraval
Facebook: / sirajology Instagram: / sirajraval Instagram: / sirajraval
Signup for my newsletter for exciting updates in the field of AI:
goo.gl/FZzJ5w
Hit the Join button above to sign up to become a member of my channel for access to exclusive content! Join my AI community: chatgptschool.io/ Sign up for my AI Sports betting Bot, WagerGPT! (500 spots available):
www.wagergpt.co

[@abhiwins123]

epic intro :D

[@SirajRaval]

Thanks abhijith!

[@FilipeSilva1]

Bro, new mic, new rythm of teaching... This is quality!

[@SirajRaval]

thanks Filipe!

[@BizTechLife]

who need college when you have Siraj

[@SirajRaval]

I always got bored after ~15 minutes of almost every lecture i attended in college. I sometimes thought something was wrong with me. But the real problem was the system. boring teachers. the content wasn't dense enough. education should be as fun as cinematic movies. we are all apart of a new wave, education is getting disrupted hard af

[@st101k]

Siraj Raval I completely agree.

[@akmonra]

For grammar lessons. Oooh!

[@JustFun-li2dk]

That's Exactly how i feel!! Learn with Fun!

[@Apreston14]

@Siraj Raval couldn't agree more! Starting my industrial placement for an ai company next week, cannot tell enough how much your tutorials have helped me prepare for it.

[@nisargbhatt4512]

your videos are really awesome.. I actually get bored while reading research papers.. This is cool..keep it up

[@rajumondal4283]

I find your low speed great plz keep this speed

[@SirajRaval]

thanks k

[@UToob157]

Had to crank the speed up to 1.25x to get the normal Siraj speed.

[@wh33lers]

Cloak111 did this too

[@SirajRaval]

Cloak111 hm ok. perhaps this was the first video I've made where I actually went too slow. well now I know the lower bound for speed. thanks!

[@kmrfrom]

I watch your video at 0.75 speed .Now watched this and stunned myself that speed is Normal. Thanks maintain this speed.Can I expect a book on machine learning from you with "write what you speak/teach " method Not Like " I want to show my English literature skills and through on 'ML ocean' style" . I still love Rich F Raposa Java learning book style.(Of course i found it after my graduation:( )

[@avijitmitra]

Cloak111 I was gonna comment the same 😂

[@konstaConstant]

Too slow > Too fast You can always change the speed of the video and speeding up the video reduces audio quality less than slowing it down, because no interpolation is needed.

[@PabloMartinez-ut8on]

You are the best! Thanks for all ! From Montevideo Uruguay : )

[@BizTechLife]

outstanding!

[@SirajRaval]

thank you!

[@karanpatel2866]

Awesome introduction on maths of ML !!

[@marcusstoica]

In the 2nd dimensional case: I believe it should be Hessian(X,Y,W) instead of Hessian(X,N,W). Thanks for the video. Using it to implement an iterated GMM estimator.

[@amartyasingh09]

What's the classical music remix played at the intro?

[@SirajRaval]

Black Swan Hip Hop Beat/Instrumental

[@Fightingbadgers]

I like this "chill" Siraj :) I know it's not totally related to the video, but what are your thoughts on spiking neural networks? Would love to see those covered at one point, even though they are not so mature yet.

[@SirajRaval]

thanks will do one in future

[@avishcheetaram1807]

Just awesome as always bro. Cheers!

[@pinkiethesmilingcat2862]

i want todo ask a favor, please add you script to the subtitles in English, so that for me and people of other idioms it is easier to traslate. For my part i want to traslate it into Spanish so that people in Latín América and Spain can Access you good content. Thanks

[@pinkiethesmilingcat2862]

this should be for all!!!!

[@SirajRaval]

copies of both scripts so far docs.google.com/document/d/1oacFxbgoROXCj3Eofyalhlnnzc_kE3vs9WYs80Rk9is/edit?usp=sharing

[@pinkiethesmilingcat2862]

Siraj Raval Thanks!! i'll stay work on this

[@hiral.shah.]

is this script still available? please share. Thanks

[@diptinemade9259]

Taylor Swift lolz... This is the first time I understand what Taylor series actually is! And I have studied engineering mathematics 😯.. I wish your videos existed back then! Life would be so much easier. THANK YOU!!

[@vinayhegde714]

really loved it..keep it up Siraj.

[@BharathLaxman]

Did you decide to switch to regular speed? Now I have to manually dial up to 1.5x 😑

[@SirajRaval]

yes it is the lower bound now i know

[@nikhilchigali]

What's the relation between running anomaly detection on 500 TB data and using second order optimization technique? Didn't quite understand the point there..

[@CristiVladZ]

I dont know why I stopped following your videos a while ago. Silly me. Keep doing what you're doing. I'm glad I clicked this on fb.

[@SirajRaval]

glad to see u back cristi

[@christianpeters3743]

Did you ever try a second order optimization algorithm with a neural network? Sure, it would probably require more computational effort, but I think there are some variations of newton for example, where you do not have to evaluate the whole hessian matrix. I think it would be interesting to see a comparision between first order and second order optimization methods in machine learning.

[@Promixro]

10/10 content

[@McMurchie]

FORTRAN!!!!!! arrrrrghhhhhh!!!!!!! * antiquity intensifies*

[@SirajRaval]

hahah

[@luck3949]

Hold on, Jacobian looks like a matrix only for R^n -> R^m functions, and Hessian only exists for R^n -> R functions, so saying that Hessian is a second order derivative analog of Jacobian is incorrect. Seems that Hessian is a second order analog of a gradient, not a Jacobian. Or am I wrong?

[@avatar098]

This is like Bill Nye The Science Guy but for Machine Learning :D

[@SirajRaval]

awesome to hear!

[@khandarwilliam5439]

why didn't you my lecturer?

[@SirajRaval]

i am now

[@ShankhaShubhra]

at first i thought i was watching at 0.5x speed. did someone pointed out that you explain at a very fast pace?? 😝 anyways great video as always. keep rocking! :)

[@pavanlulla]

Siraj I understand you have to entertain us.. but do you really have to gesticulate so.. really? it’s so frustrating, because you are talking complex stuff but we have to keep getting distracted with what your hands are doing. Almost like a sign language. Why do you torture us so.

[@complovr2971]

What's the abs(0-f(x)) for in the newton method? Why can't it just be abs(f(x))? Does it matter? Can someone answer this?

[@k2s3p]

simple explanation+clear illustrations=intelligence :-) Take a bow (Sir)aj.

[@jasneetsingh4018]

Bible for Machine learning??

[@hammadshaikhha]

I thought the pace in this video was perfect, the content was very clear as well, great work! On another note, I was looking your channel to see if you had any video Github and Jupyter notebook, I didn't seem to find anything. I will try look else where for now so I can start submitting the coding challenges.

[@SirajRaval]

thanks! Hmm good point. i will make github vid soon

[@marceloguaycurus]

If you don't want to wait, this udacity course is very complete (and free): br.udacity.com/course/how-to-use-git-and-github--ud775/

[@maryguty1705]

the second order optimization, how is it used in machine learning and neural network?

[@RELU254]

this makes me feel i dont know anything in machine learning

[@karimabdultankian28]

2 Years of boring Maths explained in 10 minutes, well done.

[@amr3162]

when you try to run "newtons_method_optimization.py" you get an error " 'Float' has no attribute sqrt " which is due to numpy (np.linalg.norm) being unable to deal with with sympy ('sympy.matrices.dense.MutableDenseMatrix') that gets changed(from np.ndarray to sympy.matrices.dense.MutableDenseMatrix) on line 71 tried with both python3.6 numpy 1.12.1 and python 2.7 numpy 1.10.4 tried to solve it but it was apparently way too hard

[@parambhavsar1573]

i also have same problem.If you find solution please let me know.....

[@damedane913]

Taylor swift.... I died.

[@SirajRaval]

lol thanks

[@harshvardhan7546]

root finding method is newton ralphson's method

[@hanman5195]

Your awesome and searching for best words to praise you

[@devangsanghani6765]

That's a cool white shirt!

[@isaacnewton1545]

None of these methods find the Global optimal solution

[@larryteslaspacexboringlawr739]

thank for math and intelligence video

[@frenchpet]

wtf did I just watch 😥

[@ricardodiaz3999]

Siraj please can you make a video or Live session about Faster R-CNN please I really appreciate it there's no information about it online

[@SirajRaval]

hmm considering thx

[@RandhirRawatlal]

Fantastic enthusiasm and dedication to your field - an inspiration to all teachers..! I really liked how you simplified the explanation of the Hessian

[@novovires5625]

She's a gorgeous lady :) lol

[@hanyuliangchina]

Please find the boring theory of interesting applications in reality. I do not like the only boring theory. that is not your style.

[@SirajRaval]

+libai tony hm interesting. Alright thanks.

[@sam_joshua_s]

Understandable video

[@rajantechie]

why is the video quality bad Siraj ??

[@SirajRaval]

will improve

[@syn3rman65]

You need to do more math videos

[@kmeeraj]

Thank you

[@osinachi6887]

Wow wow wow. Am walking on arc welding robot and I have to do the inverse kinematics of 6 dof. I never knew anything about optimisation and I downloaded some articles and studied it. I started with Dampes Least Square for the inverse kinematics. I wanted to study quasi-newton method, then I encountered this guy. Your short lecture connected all the dots I have been studying for a week. Thanks🙏.

[@RudolfEremyan]

Thanks!))

[@ilyajob0524]

Grace 😀

[@griesrt]

dang you the best.

[@onefulltimeequivalent1230]

Do you mind saying "minimum"(sg.) instead of "minima"(pl.) next time in order to optimize the professionalism function? :)

[@MONUKUMAR-jk8xl]

which country are you from Siraj? Guess you are Indian! just asking out of curiousity. Wanted to know about you.

[@absharkhan9954]

Wow!

[@ryanbach937]

Do you have a BTC address for donations?

[@SirajRaval]

yes see about me section, patreon preferred

[@pickarpit]

I came here just to see Siraj Dance ! !

[@SirajRaval]

haha thanks

[@vinodp8577]

Ripping off content from mathematicalmonk but not giving credit? Wew Siraj, Shame!

[@mrvzhao]

First viewing: WTF?? Second viewing: wait, I think I'm onto something Third viewing: the epiphany moment guy @ 9:00

[@suraj2085]

I was kicked outta Polyglot group for saying mathematics a language

[@srivishnuk3451]

Hi Siraj, This is my entry to this week's contest. github.com/Sri-Vishnu-Kumar-K/MathOfIntelligence/tree/master/second_order_optimization_newtons_method I have used the same dataset I used for week#1 and have also compared the performance of the 2nd order optimizer to gradient descent, there is a decrease in error. The dataset, however, was too small to identify changes in execution time. I found function derivatives using the first principle. I hope you like it. Thanks for the fun tutorials. This is serving as a great learning medium for me.

[@SirajRaval]

great work

[@srivishnuk3451]

Siraj Raval thanks a lot Siraj. This served as a great motivator :)

[@gijsvermarien]

The amount of bicycles in the background of shots confirms that this movie was taken in the Netherlands. I suppose you will not speak in the Netherlands anytime soon?

[@SirajRaval]

+Gies yup in Amsterdam, where I now live. Id be down to speak possibly. Email me the venue :)

[@gijsvermarien]

Awesome, I am planning on starting my master applied data science at UvA in 2018:) About the speaking part I was just wondering whether you'd be speaking in the Netherlands soon. Really liked your talk at TNW Amsterdam. Unfortunately I'm not hosting anything sorry!

[@kushajveersingh9973]

We are using gradient descent and the line we will use to fit the data is y=mx+c. I just wanted to know how will you compute the second order derivatice of it knowing that it is always zero. Or please tell me whether i am confused and please clarify that it is the activation function that we actually double derivate. If it is the activation function than double derivative will always be zero for ReLU. Thanks

[@hiral.shah.]

Mother of 2, wants to resume my career back after a long gap. and i landed up here. Wow!, you have given me confidence that a mother can learn math too :) After watching you/your videos I am feeling that I took a wise decision of not joining college instead self learning. :) God Bless you Siraj.

[@sushantahuja9067]

cool!

[@brooklynground7090]

You just killed 98 percent of professor at my college. Amazing.

[@williamchamberlain2263]

... one of the three dudes who invented calculus, I think ...

[@tbass134]

silly question, but what version of python should we be using? 2,3?

[@rishitshah3908]

Hey Siraj, I know it's been a while since you uploaded this video but I have a quick question. Can it happen that the minimum of the estimated function using Taylor series lies on the objective function itself? In that case our program would be stuck in infinite loop. Or, from a second perspective, is it could be the case that the minimum of the estimated and objective function coincide only when we reach minimum of objective function(Not claiming this is true but it seemed intuitive in above example so assumed a generalization for it.). If this is true, what is the mathematical explanation for it? Thanks a lot. Your videos rock by they way. Great work

[@victorarias9688]

ohhh my god, is this seriously?. you are the best professor in AI, greetings from Colombia.

[@soumyadeepdebnath4063]

Hey Siraj. I really like the way you teach. You are doing awesome . I wanted to say that conjugate gradient descent is more powerful than the traditional steepest descent, so why not we use it. Could you please make a video based on solving Supervised learning problems based on conjugate gradient method!!

[@izzyellisonxx1459]

We're u flying Ryan air yesterday from morocco

[@SirajRaval]

no lol

[@chasegraham246]

Siraj, why are you talking sooooooooo sloooooooooooooooow? Also, you forgot to delete your red circle slash thing from your Brook Taylor Swift joke. Siraj, are you on drugs now? :/

[@SirajRaval]

+Chase Graham it was on purpose. Thx. No. I will go a bit faster nxt time

[@chasegraham246]

Sorry, joke went over my head then. Guess I'll be putting this pipe down now.

[@jdjdjdjjdjdjdj7863]

Hi a newbie here! Where is the code to draw that cool function!? And a second question that I have: Where is the code of the moments you show in the video, for instance 6:21 ? Thank you!

[@angellove8641]

kissing the edges of the optimization curve :*

[@cryptverse]

Loving your energy, brother!

[@SirajRaval]

thx brosicle

[@guitarheroprince123]

correction: It's Newton-Rapson method.

[@AkashMishra23]

You Have Great Pick Up skills lol

[@SirajRaval]

thx anything is possible

[@ShahidulAbir]

Siraj Raval did you get the silver play button for 100k subs?

[@suryasikharej8116]

sir where in the code we are finding the tailor series? and what is the output of that hessian matrix?

[@ruxiz2007]

So clearly explained, no single word is with less value! Thanks!

[@sau002]

wow! If only somebody had spoken to me like this 25 years ago!

[@ELStalky]

"Minima" has a singular, it's "minimum". #GrammarNazi

[@정성엽-f1y]

It is really good but difficult. :( I have to repeat it!!

[@jsandmann]

The sound recording sucks

[@SirajRaval]

Jonathan Sandmann will fix thx

[@interstellarsurfer]

You're a lucky man, Siraj. Many men would gladly discus second order regression with your lady friend... if they even had a clue. :-)

[@SirajRaval]

haha thanks

[@thecodingproject7882]

wow, g8 editing and g8 way to give Iformation ..

[@lisetteelgueta5473]

lol who’s the guy on minute 9:04

[@CausticCatastrophe]

I appreciate everything you have done here.

[@guavacupcake]

I saw Taylor Swift coming from a mile away. I was like, "He's gonna say Taylor Swift now, isn't he"

[@SirajRaval]

u know me too well lmao

[@shivajidutta8472]

Really enjoyed the Video!! Really liked the math perspective to things.

[@SirajRaval]

thanks!

[@luvsuneja]

Lol... was that a Leibniz joke at 4:50 ?

[@SirajRaval]

yes

[@soumyadrip]

What a intro...XD

[@franciscomelojunior2535]

Normal time: 1.25x

[@Aryeh-o]

siraj is a newtype

[@kamalnayan9157]

Newton-Raphson

[@thebluebloo]

That was great! Loved it!!

[@ego_sum_liberi]

Great Video!