The Least Squares Formula: A Derivation

[@MathTheBeautiful]

Go to LEM.MA/LA for videos, exercises, and to ask us questions directly.

[@cern1999sb]

It would be great to have the second part of this with an audio recording of the explanation. Everything was crystal clear until the mic died

[@ForcesOfOdin]

There's a timeless beauty to the cadence , passion, and clarity of your lecture series. Glad I could find it again. Different goals now, but same appreciation.

[@evieblue959]

An energetic math teacher. I've never met one in the wild. Thanks!

[@MathTheBeautiful]

And damn handsome, too!

[@muratcan__22]

the most important part is missing

[@vangrails]

What is that part?

[@dcuzhang]

@@vangrails battery😂

[@jonathanpepin6326]

Thanks à lot for your videos. I am à french Student in mathematics and your way of explaining things is so different from here but it all makes more sense. Look forward to the next videos.

[@damian.gamlath]

Sir, I love the intuition you inject into these topics, something most other teachers (and KZbinrs for that matter) fail to provide.

[@javierapuebla2451]

this video is marvelous from start to finish, and the mute subtitles did really help

[@MathTheBeautiful]

Glad to hear that!

[@orangeflow9809]

Even with the mic batteries dead, this was a great lecture.

[@MathTheBeautiful]

..................................................

[@yamansanghavi]

I wish there were more professors like you sir. You are truly genius. Thank you so much.

[@mpja]

The formula I haven see uses the complex conjugate transpose instead of the plain transpose. For real values the two are equivalent but I would make a note of this.

[@via6400]

Tank you very much, You solved my least squares problem.

[@MathTheBeautiful]

It's the least I can do

[@darrenpeck156]

Please record the next video with sound.

[@matthewjames7513]

at 8:41 can somone help me undersatnd why xTATb = bTAx ? I can understand why xTATb = (bTAx)T, but where did the big transpose go?

[@matthewjames7513]

Oh I think I get it now. Since xTATb = is a 1x1 matrix, and since the transpose of a 1x1 must be the same, then you can have the big transpose or ignore it, it doesn't matter in this case!

[@matthewjames7513]

But why at 9:43 is ATA guaranteed to be invertible ? :O

[@HelloWorld-dq5pn]

@@matthewjames7513 AtA is invertible since it is positive definite, therefore, all of its eigenvalues are greater than zero(and real since its symmetric). Note that AtA is symmetric, then consider the magnitude square of Ax, expand it as a dot product and take x to be an eigenvector of A.

[@Acampandoconfrikis]

What an epic way do deal with a microphone problem XXDDXXDXD (also good explaining, thanks!)

[@michaelroditis1952]

yeah i loved the idea!

[@hushengli6697]

wonderful!

[@jozeflodeweyckx4009]

Interesting way of explaining the subject.Could I get a pdf file of the story?Jozef

[@Positive_Vibes_26]

Sir As we express a length as xTx How can express a volume in matrix entry

[@MathTheBeautiful]

Determinant

[@visbellum]

What about the error?

[@pipertripp]

You should have done the second half of the video in sepia. A moustache would have been a nice touch too.

[@BraddockGaskill]

Yes please re-record the audio, it was a brilliant explanation up until then

[@kyang1305]

I was lost from the 'A', 'b' part... like we discovered a similar structure but how did we directly got the value of x...?

[@MichaelStangeland]

Yeah, me too at first. On the left of the board there's something previously covered about "Quadratic Form Minimization", and that operation parallels differentiation in calculus. Note that to find the x that minimizes rᵀr, you take the derivative WRT x and solve for the x that makes that derivative equal to zero. It looks like he compares the xᵀAx to ax² and illustrates how they become 2Ax and 2ax respectively. So he goes from rᵀr = 2(½xᵀAᵀAx - xᵀAᵀb) + bᵀb to 0=2(AᵀAx - Aᵀb) to AᵀAx = Aᵀb to x = (AᵀA)⁻¹ Aᵀb

[@MichaelStangeland]

kzbin.info/www/bejne/pZLMmqx_qMeBmqs

[@muhammadhafiezd2514]

@@MichaelStangeland Thanks, this is helpful

[@nahblue]

​@@MichaelStangeland Thanks, this was the missing step in the video. Would you agree to say that we differentiate w.r.t xᵀ? Maybe it truly doesn't matter, one can always rearrange. And we use the fact that AᵀA is symmetric when we work with differentiation this way.

[@gautamrajeev3345]

Why are they equal because they are transposes of each other?

[@grjesus9979]

4 years later he is still busy hahahahaha. Good class except fot the last 2:30 where magic happens

[@ahmedhemani1259]

Can someone point to me where are the rest of the videos of this course available ?

[@abdelrahman458]

great work very good lecture , but i wished there was an order or number of the lecture so i can visit the previous ones if i dont understand an certain concept

[@jimturner4937]

where is the re-recorded version

[@MathTheBeautiful]

Sorry, never happened!

[@oscarhernandezarzeta6222]

Thank you very much, yur videos are great, they give me a pretty good insight of the subject, please upload this video again :D

[@97Alfinsyah]

as geodesy and geomatics engineering student i always using the least square adjustment to solve problems :D thank your sir for the lecture he he he

[@kallzvx858]

Great video, but I cant understand the end without knowing what the lecturer is talking...

[@chensun2427]

Ah Noice! A Math lesson in Bioshock style xD

[@TuNguyen-ox5lt]

what if ATA is not positive definite (that leads to the rTr does not have minimum value because we can choose x to make rTr as small as possible ) and if ATA is not invertible ? . Please help me .

[@MathTheBeautiful]

Please see this video: kzbin.info/www/bejne/mKGWaXWBhZKrnpY

[@TuNguyen-ox5lt]

So ATA is always positive semi-definite . if A is invertibe so that ATA is positive definite and we can find a global minimum of rTr . If A is singular , ATA will not invertible and the equation resulting from derivative ATAx = ATb can have multiple x to satisfy . As a result rTr will have multiple local optima . That 's what my thoughts . Am I correct ? . Thank you very much for quick response .

[@MathTheBeautiful]

You are correct. More specifically, it will be one minimum value, but it will attained at a whole subspaces of locations. Sort of like the function f(x,y)=x^2.

[@TuNguyen-ox5lt]

I understand your example of f(x,y) = x^2 . The minimum subspaces in this example is a line x = 0(y whatever ) . If we randomly choose location of x and apply gradient descent , do we always certainly get to the minimum subspaces ? Does all optimization problem that can be expressed in quadratic form always have just one minimum value ? (I think yes but I still want a verification ) For other cases that are not a quadratic form , the function will have multiple local optima and the value of function of those local optima are different . If I use gradient descent with randomly choosing the starting point of x , it will lead to different local optima ( in case of luck , we get to the global optima ). So the position of starting x is important . Is there any way that I surely can come to global optima instead of getting stuck of local optima . My idea is choosing multiple starting position x and the get the x that have f(x) minimum . But this way also depends on luck and not solve the situation completely . Can you come up with a good solution for this problem ?

[@shimaalcarrim7949]

Thanks

[@vikraal6974]

This feels like a Charlie Chaplin movie after mic died.

[@johnfykhikc]

didn't get the final step o f the proof. anyone found the video?

[@unrmaestro]

ty sire

[@daniel.fajtai]

modern problems require modern solutions

[@andrerossa8553]

very funny! Tks a lot for such a great lecture

[@FsimulatorX]

What is everyone here studying? I just came here because of an equation I saw for Linear Regression XD

[@israelmcintosh9745]

@Max Kirkman linear algebra

[@flightrisk7566]

I’m actually here for NLP

[@TheTacticalDood]

Computer Vision

@@flightrisk7566 NLP x2

[@edilgin]

regression

[@MyJuicehole]

I think the word you were looking for was "stamp"

[@MathTheBeautiful]

That's right!

[@edsonsabino]

very nice!

[@devoglumehmet4941]

Great !

[@roflchopter11]

Slow until the audio cut out, after which the interesting part takes place. This could have been really good, but the audio problem was not handled well. "What a beautiful equation" for 20 seconds the end immediately after the solution was some frustrating editing. Keep it up, but that's some constructive criticism

[@channelpanel8259]

NOOOO !!! This is horrible!!! This was shaping up to answer all my nagging questions about least squares :( :(

[@MathTheBeautiful]

Give us one of those nagging questions.

[@JohannesSchmitz]

MathTheBeautiful in the last step did you take the first derivative and set it to zero to find the minimum of r^Tr?

[@pavan1000]

feels like the battery understood what he said just earlier, hearing wont help, read out it yourself to better understand , LOL

[@MathTheBeautiful]

yes the battery is in charge

[@jimmyjonestodd2556]

Damn the mic dying!!!!

[@droully]

stump it out

[@demidrol5660]

like it

[@darrenpeck156]

Please rerecord or talk over!!!! Please!!!

[@FB-tr2kf]

Gave you a dislike because of the sound towards the end.

[@rovshanabdurrahimov5892]

but it is not his mistake. u should be respectful. if u dont like go other channels. this man explains in best way.

[@aboodfarhood9316]

Math is super fun, but when we listen to about one xillion detail at every corner? It becomes so boring.

[@MathTheBeautiful]

I think it's the one zillion first detail that one too many!

[@difjaoisdjfoaisjdfoaiogeis]

i was enjoying it until the CHALK SCREECHED AGAINST THE CHALKBOARD UNGHGHHGH IM SO UNCOMFORTABLE NOW

[@ozzyfromspace]

Geez my guy, for every 5 seconds of math, there's like 2 minutes of you talking about "stamping it out". It's painful to watch, sorry.

[@MLDawn]

Is this a joke?

[@_random_dude]

The guy talks too much.

[@MathTheBeautiful]

Agreed. His wife must be sick of him .