17. Orthogonal Matrices and Gram-Schmidt

[@mitocw]

Removed long video delay after the title.

[@shubhamtalks9718]

Where can I download these videos in 480p? I checked the ocw website, the download links are for 360p or less quality. And, thank you for this awesome course.

[@qinglu6456]

That just also removed many comments under the original video.

[@biao9957]

Sounds like a git commit message

[@TMAC02010]

@@biao9957 😂

[@andreworals7677]

@@shubhamtalks9718 youtube-dl

[@ozzyfromspace]

The fall from 2M views lecture one to 18k views lecture 17 (half way through) is crazy. Congratulations if you're still here. I've followed every lecture so far (notes and all) and I've gotta say, it is 100% worth my time. I might not have a degree from MIT, but I can say I've experienced how superb their education is firsthand. Thank you to Professor G. Strang and MIT OCW for making these lectures available online. You have made learning for me interesting again.

[@LogviNata]

Lecture 18 has 200k views. The fall is still big, but I believe this lecture has 20k for the last 6 months after the video was replaced.

[@Labroidas]

Hello, this video replaces one from 2009 because the original video had a delay problem, if you look you can see that it was posted only recently in September of 2019. So unlike all the other videos in the series, it didn't have 10 years to accumulate views, but to be honest, seeing that 20 000 people have come this far in the course in only 6 months is quite astonishing!

[@bigbangind]

chill dude why does it matter

[@Abhi-qi6wm]

calm down bruh, we get it. You're studying lmao.

[@cherma11]

Well as long as students have to deal with linear Algebra, they will meet Gilbert Strang - lol.

[@CL-dy7ni]

Gram-Schmidt portion of the lecture begins at 26:07 for anyone only interested in that

[@diegommleonardo4776]

This should have more likes

[@alierenozylmaz3488]

Isn't it crazy we still follow 15 years old lecture because its still best. What a legend G.Strang

[@theblinkingbrownie4654]

@glyn hodges no you wouldn't wanna watch pythagoras's videos

[@ΑλέξανδροςΣαββίδης-ξ5κ]

pulling an all nighter studying linear algebra and I would have never done that if I hadn't stumbled across his lessons, hands down best teacher I had ever witnessed. I might fail in the exam but these lessons have helped me follow and understand much more than ever.

[@theodorei.4278]

Αλέξανδρε το πέρασες τελικά?

[@azrael6882]

@@theodorei.4278 Δυστυχώς οχι,το ξαναδεινω φέτος ομως σε συγκριση με τον καθηγητή του πανεπιστημίου έμαθα πολυ περισσότερα απο αυτά τα βίντεο. Edit: Για λιγο δεν περασα,0.25 που ηταν να γραψω 5 μοναδες παραπανω στο τελικο, ανεξάρτητα ομως με βοηθησε πολυ

[@kkounal974]

​@@azrael6882 Σπουδάζεις μαθηματικά, φυσική, μηχανική ή επιστήμη υπολογιστών;

[@azrael6882]

@@kkounal974 επιστήμη υπολογιστών

[@unlockingthemind]

its been a while... did you end up passing your final??

[@thomaswijgerse723]

It amazes me how much a good teacher helps in grasping the subject. Thank you professor Strang

[@NIKHILSAI-y6j]

00:00 Orthonormal basis and matrices have many advantages in numerical linear algebra 06:42 Orthogonal matrices have orthonormal columns and their transpose is their inverse. 13:10 Orthogonal matrices have orthonormal columns. 19:21 Projection matrix onto orthonormal basis is Q Q transpose 25:25 Graham-Schmidt algorithm produces orthonormal vectors. 31:12 Finding the perpendicular vector using Graham's formula 37:07 Graham-Schmidt process for orthonormal basis 42:44 Graham-Schmidt produces orthonormal columns and a triangular connection matrix.

[@FirstNameLastName-gf3dy]

In order to understand the A= QR part, my suggestion is to take advantage of the fact that Q^T is equal to the inverse of Q. So, multiply the both sides of the primary equation with the Q^t from the left. You will get Q^t A = R. In this form, you will see more easily that why the elements of R matrix should be what they are and why R is an upper triangular matrix.

[@aashnajain6519]

Thank you, i was really thinking how he arrived at R.

[@hsmelodic7779]

Thanks a lot!!!

[@amol5146]

Won't we get the transpose of R then? The elements will be q1ta1, q1ta2 and so on, right?

[@marcuslu4214]

you are the savior of all mankind

[@sharafmakk2936]

Q^T isn't quite the inverse of Q. Q may not have an inverse (if it is not square) instead think: A = QR (multiply both sides by Q^T from the left) -> Q^T * A = Q^T * Q * R -> Q^t A = R ( because Q^t Q = I by definition of Q) From here you can see that R is just the dot product between q1, q2,q3 ... and a, b, c... So, q2 dot a = 0 (because q2 is just b - the component of b in a's direction)

[@jeyanthr6284]

19 years and still the best lecture on gram-schmidt process on youtube ! hats off sir

[@javiergallegos8515]

For the QR decomposition ( 47:00 ) you can use Q^t * Q = I. With this you get: R = q1^t * a1 q1^t * a2 q2^t * a1 q2^t * a2 but since we work in real numbers the internal product A^t. B = B^t. A Using it, we are left with: R = a1^t * q1 a2^t * q1 a1^t * q2 a2^t * q2

[@joseguerra225]

" I don't know what Schmidt did" ahhh priceless.

[@Mac_n_CheeseSauce]

I was reading Introduction to Linear Algebra and decided I needed more direction. I was so exicited when I released I get to see the author teach the material.

[@rishabhchopde6709]

I had learnt the Gram-Schmidt process during my signal processing class, and now I'm learning about it again, this time due to no compulsion from course work. Instead I'm learning it for QR decomposition, so I can solve the problem of least squares for a project which requires procedural geometry generation. I never thought I would be back learning about this, but here I am. This video really helped out a lot more than the material I used when I was learning out of necessity 😅

[@georgesadler7830]

This is another fantastic lecture by the ring master of MIT mathematics DR. Gilbert Strang. GRAM-Schmidt is a classic topic in linear algebra. An individual student cannot have too much linear algebra.

[@seanpitcher8957]

I got the textbook to go with this series and Im doing the HW.... Its the most readable math textbook I ever had, follows and expands on the lectures and the exercises carefully are crafted to expand your knowledge of the subject. They also drive home by practice the concepts.

[@HarshSingh-nu9vy]

Completeing all lectures before 1 day of exam... And it worth it..💯💯💯

[@cianeastwood1354]

Poor Schmidt. Legacy ruined.

[@jamesxu2224]

His legacy is already tainted from being a Nazi unfortunately

[@pseudolullus]

Yeah ehm, he ruined it all by himself.

[@yanshudu9370]

Conclusion: A set of orthonormal vectors combine into a matrix called Q. Q'Q always get an identity matrix. If Q is square, it's an orthogonal matrix, and Q'=Q^(-1). If we use Qx=b instead of Ax=b to solve the projection problem, it will be much easier. Because the projection matrix P=Q(Q'Q)^(-1)Q'=QQ'. For the A'Ax=A'b problem, if we use Q instead of 'A' we can simplify it as x hat equal to Q'b. Gram-Schmidt method: A way to transform independent column vectors into orthonormal basis. A=QR => Q^(-1)A=Q(-1)QR => Q'A=R

[@CKPSchoolOfPhysics]

Professor Gil Strang has spoilt our learning process, so much intuition is mesmerizing. I started LA for the first time to learn on my own, and if I know this subject then all credits to him and MIT. ❤️

[@pranavhegde6470]

I don't have much time left for my final exams, but here I am watching all the lectures of the playlist. Dr. Gil strang is a legend.

[@kate-turn]

his chalk on the blackboard sounds are just so satisfying 😍 why doesn't all chalk sound like this??

[@lugia8888]

calm down girl hes not interested 😂

[@nguyentranconghuy6965]

wow, this professor just get rid of all my confusions about the formula of Gram-Schmidt

[@daniel_liu_it]

2020 still there, following every tape and note so far,

[@flewolfswonderland3063]

2023 still here😆

[@hritikdubey4501]

Now I get to know that why the children's of MIT are genious. This is just because of such a brilliant professors knowledge delivery. Best lecture ever had in maths.

[@lugia8888]

got my degree from there but i disagree. you have plenty of great mathematicians in other universities too

[@robertof.8174]

what an amazing lecture, I already finished college, but I still need some of these classes. Thanks for put this on youtube!

[@avi123]

11:20 the matrix is not a rotation of any angle, it is actually a reflection. (You can see that from the fact that det(Q)=-1)

[@tarlanahad]

you are supposed not to know the determinants

[@matiascosarinsky]

@Avi Aaron if you are refering to the Q from the top it is indeed a rotation

[@ryanjackson0x]

It's both. The cool thing about math is that you can think about things in different ways.

[@cache-re8if]

When I have a hard time in my life, I return to this set of videos.

[@ozzyfromspace]

27:00 *"Gram had, like, one idea. I, I, ...I don't know WHAT Schmidt did"* 😂 Two seconds later: *"Meh, we don't need either of them actually."* 😂😂😂 I read the comments in advance, then waited patiently for that roast 🦃🍗. Not disappointed 😂👽

[@jwmeyling]

Only to finish him of "maybe thats what Schmidt did - he, brilliant Schmidt - thought okay, divide by the length. Okay, that - is Smiths contribution :') Absolute favorite part.

[@agarwaengrc]

@@jwmeyling Schmidt was a Nazi sympathizer.

[@shershahdrimighdelih]

All these videos previously had a delay between the audio and video. These are reuploads, and that's why there's so few views

[@MrMechyserra]

Thank you professor Strang for all these lectures! I've remembered why I used to love math!

[@ceejay1684]

fav lecture so far in the series

[@AlgebraParaTodos]

I like your style man! I wish I had those blackboard oh man.. Thanks for the passion you put in

[@sanatanmeaning]

thanks OCW :) for this wonderful series !!

[@Maxi-ym8du]

Selfeducation with such good teachers is the future! Thank you Professor Strang.

[@schmetterling4477]

University was always 95% self-education. That's not really what the professors are there for.

[@Mark-nm9sm]

great as always , half way there motivated to learn more, that is why i love your teaching

[@chinmayrao9596]

Instead of writing "Eg" he writes "Example" . DEDICATION LEVEL

[@quirkyquester]

amazing lecture. Thank you Professor Strang, Thank You MIT!

[@GainsGoblin]

"hm I don't know if I've done this too brilliantly" I also say that after my algebra exams !

[@akistsili8574]

I am grateful for your lectures and your teaching.. "oh, it's no big deal, maybe that's what' Schmidt did". :) :)

[@santyias87]

Watching this in 2021 and gasping when Prof Strang coughs "oh no, he better not have Covid"

[@michaelguerrero7959]

The classes are wonderful, excuse me, what linear algebra book do you work with?

[@mitocw]

The readings are assigned in: Strang, Gilbert. Introduction to Linear Algebra. 4th ed. Wellesley-Cambridge Press, 2009. ISBN: 9780980232714. See the course on MIT OpenCourseWare for more info at: ocw.mit.edu/18-06S05. Best wishes on your studies!

[@SphereofTime]

30:47 B is an error vector here

[@priyanshubansal6776]

he ask ques to audience and answer himself before they gave that and it is interesting point in his lecture . it seem like he teaches himself not the audience .

[@ruixli3220]

breathtaking

[@rakuuun4582]

Gram-Schmidt Algo starts from 25:30

[@pallavsharma9110]

i am like schmdit in my group projects.

[@liyuanzhang52]

what's the intuition behind 'Ax = b has no solution, however Transpose(A) Ax = Transpose(A) b has a solution'. I am having a hard time

[@byungwookang5833]

Multiplying by transpose A is the idea that was used in projection when vector b is not in the column space of A. Also, Transpose A * A makes it a square matrix which has unique solution or infinitely many solutions.

[@angfeng9601]

A further elaboration of 강병우 's comment. Ax = b has no solution when b is not in the column space of A, meaning no non-zero combination of columns of A gives b. The best solution \hat{x} expresses the projection of b onto the column space, p = Pb in the column space, so that p = A \hat{x}. This means b is actually decomposed into p and the error vector e in the subspace that is perpendicular to the column space of A (left nullspace), i.e., b = e +p. This error vector has been dropped, but its length is minimized. 'Best' is in this sense. As the error vector is perpendicular to the column space, A'e=0-->A'(b-A\hat{x})=0, thus we have A'Ax=A'b. Recall the projection matrix P = A (A'A)^-1A', which has an inverse when A has independent columns. Alternatively, multiplying A' to both sides of A\hat{x} = Pb, we also get A'Ax=A'b. To solve this new equation, just use the inverse of ( A'A), so that \hat{x} = (A'A)^-1A'b.

[@sergiohuaman6084]

@@angfeng9601 that looks like some serious LaTeX lover!

[@zorroesclito5286]

When A has independent columns AtA is invertible , actually the nullspace of A is identically the same as of AtA

[@user-qj6hl5xb8q]

Around 33:00, for the right-hand side of 'B', how did we obtain A'b/A'A * A? The previous lesson, we obtained 'p' as aa'/a'a * b. How did the aa' *b in the numerator get switched to A'b * A?

[@AftabAlam-xh2og]

I also have the same query??

[@tilki3114]

He place it wrong

[@rohakdebnath8985]

actually, since the A here is treated like a n dimentional vector, i.e. a nX1 matrix, where the properties of vectors and matrix algebra is the same.

[@hithe7217]

Since A' just a row, A'b a scalar, the switch is fine

[@barbaraiisager3910]

At 15:34. Did he make a mistake here? The two normalized vectors 1/3(1, 2, 2) and 1/3(-2, -1, 2) aren't orthonormal because their inner product is not zero. It is 1/3(1*-2 + 2*-1 + 2*2) = 4/3.

[@sammatthew7]

1/3(1*-2 + 2*-1 + 2*2) = (-2/3 -2/3 +4/3) = 0 , it is correct only

[@ianstorey1521]

Maybe we couldn’t hear the audience. I hope his joke that Schmidt didn’t have much to do got a laugh. Really nice lecture!

[@SphereofTime]

26:09 make the column orthogonal

[@gangchen2891]

Can someone explain the difference between two approaches of solving the least square problem 1) using Transpose(A)*A*x = Transpose(A) * b, or 2) Using QR decomposition, i.e. R * A = Transpose(Q) * b

[@KooKim-n8o]

Dislikes(if they exist) came from Schmidt fans

[@jaihind6472]

Thank you for saving 10 points

[@CRAZYSARAN]

35:00 This is like a typical home work quiz problem.

[@MuhammadAhmed-rt1mr]

guys at the end of the video, why isn't b(transpose) * q1 = 0, like a(transpose)*q2 is 0 ?

[@hurbig]

Five minutes in and my mind is blown

[@DuoGamesVideos]

2nd half: 25:06

[@moranreznik]

Let's make Matrix Orthonormal Again!

[@Fan-vk8tl]

somethings wrong in the last A=QR? should the transpose on the q_1 q_2 not on the a_1 a_2??????

[@suvigyajain9360]

Poor Schmidt. He didn't deserve the shade

[@agarwaengrc]

Oh, he did.

[@sviswesh3555]

@33:10 , is he dividing by A^TA , because A is currently having just one column, and hence A^TA will be a scalar value? Otherwise, it should be (A^TA) inverse, right?

[@shubhamtalks9718]

My neurons are enjoying...

[@DeadPool-jt1ci]

why is Q times Q transpose not equal to the identity if the matrix is not square ?

[@helikthacker413]

As he had showed initially Q transpose times Q = Identity and if Q is square it will have inverse (Since all columns are independent). So we can show Q inverse = Q transpose. Now A inverse times A = A times A inverse = Identity If A is an invertible matrix. Using this we can show Q times Q transpose = Q tranpose times Q = Identity. When Q is not square then we don't have an inverse. So this is not possible and we are left with Q times Q transpose. Correct me if I am wrong!

[@dianel.9238]

@@helikthacker413 It seems right in this formula way, but when I wanna explain it in multiplication way, sth is wrong there... When Q times Q^T we got the (1,1) item in the result from the row1 in Q times col 1 in Q^T, it desnt get 1.... Hope you could help with this puzzle. Thx

[@helikthacker413]

@@dianel.9238 What Q are you talking about? A square one or a non-square one. In square taking a few examples you will get Identity matrix. But in non-square it is expected that it will not be an identity matrix and so you wont get a 1 in diagonal. This is my understanding, please correct me if I am wrong.

[@dianel.9238]

@@helikthacker413 It is the columns in Q are orthogonal ,right? not the rows. In this case, the (1,1) item in Q*Q^T is the result of row1 in Q times col1 in Q^T ,ie row1 in Q... how come that gives 1 (in this case square or not doent seem matter)?

[@helikthacker413]

@@dianel.9238 kzbin.info/www/bejne/Zn7XqKSeftyXqKs This shows if Q is square, Q^T = Q^-1 since Q*Q^T = Identity. So after that it is clear that Q^-1 can be left multiplied or right multiplied to Q to get Identity. I don't know the proof but taking any Q and right multiplying by Q^T gives Identity.

[@kewtomrao]

What my lecturer took 2 hours,you sir took 50 mins.

[@Abhi-qi6wm]

I guess we could also take the 3rd vector by the cross product of the 2 earlier ones.

[@tayoukachukwu3105]

Good point. Thanks

[@gowtham6071]

But in higher diminsions, how do you gaurantee that the cross product gives the vector that is in the space spanned by the 3 vectors that we started with (a,b,c).

[@PretamRay]

what is the reason behind R being upper triangular?

[@sreeganeshvr7561]

We know that we choose 'A' (the first ORTHOGONAL vector) as 'a1'. Later, we see that 'q1' is just 'a1/||a1||'. This means that 'q1' is the unit vector of a1. Now, in the equation, the first column of the LHS is 'a1'. Similarly in the RHS, the first column in the first matrix is 'q1'. Since q1 and a1 are the just same vectors with different directions, all you need is q1 to be scaled by a constant to become a1. This means that the q2 vector is not required. Not required naturally means the coefficient is 0 (remember that a1 and q2 are orthogonal vectors and their dot product gives zero ). Since the second matrix in the RHS is a 2x2 matrix, the cell at row =2 and column = 1 is 0, the matrix becomes upper triangular. That's all.

[@ryanjackson0x]

I think the secret to this is that we're finding parts of triangles.

[@dianel.9238]

19:28 since the fact that Q transpose times Q equals identity matrix is true for all matrix with orthonomal columns, doesnt Q times Q transpose also equals I ? the P= I always holds true for matrix with orthonomal columns?

[@electroe2143]

Why the position of A is changed in the projection (B = b - A(A^T.b)/A^TA ?) ?

[@hilmar5385]

It doesn't matter because A is a vector so (A^T b)/(A^T A) is a constant

[@ericksantiago9696]

Placement of A does not matter in case where A is a vector, but does matter in case where A is an nxn matrix. In latter case, projection matrix is P = A(A^T A)^-1 A^T, where projection matrix P acts on some input (vector b, for example). Full illustration of P acting on vector b is: A(A^T A)^-1 A^T b

[@TUMENG-TSUNGF]

Thanks!

[@sarvasvarora]

DID HE JUST DIS MR. SCHMIDT👀😂😂

[@dougiehwang9192]

43:30 How can column space of Q be same as column space of A?? I know that their dimensions are same, but Q is composed with orthogonal vectos and A is not. I need some helps guys {{{(>_

[@dougiehwang9192]

@CoeusQuantitative SO If I suppose a,b are basis of matrix A, then the error part, e, is a linear combination of a and b. that's why column space of Q is same as column space of A. Is this what are you trying to say?

[@dougiehwang9192]

@CoeusQuantitative ok, wait, I am figuring it out..... anyway I am really appreciate your replies. Live long and Prosper.😄👍🙏

[@dougiehwang9192]

@CoeusQuantitative what a great picture!! I finally fully understand the picture. The problem was I was confused vector 'b' becasue of merely overused terminology........ In the previous projection and least sqare chapter, vector 'b' was out of C(A) and we tried to project vector 'b' onto C(A). That is Ax=b, the right side. However, in orthonomal and Gram process(pf.Gilbert doesn't like SChmidt so...😅) chapter, vector 'b' is just one of the basis of matrix A. That's why I was so confused 😥. My thought was the error vector 'e' should be in the left null space, and how can vector 'e' be in C(A)?? Just using a1, a2, a3 would be much easier to understand....I figured this out by solving some questions in the text book. And I watched your video, that was awesome. you made it for me😉. I want to say thx again.👍

[@dougiehwang9192]

the vector 'b' in orthonormal bases chapter is the one which is in the right side of eqation. It's not the basis. Be careful guys.

[@losboston]

In finding the perpendicular vector why the subtraction why not project (dot) then cross?

[@cooking60210]

How would that work? Why project first?

[@akaci-w3k]

Thank you Professor Strang and MIT.

[@JuliusFrischholz]

Hi, does anybody know the name of the book hes always referring to? :)

[@山田林-f5b]

thank a lot

[@amangurmani4379]

Why did we divide by square root of 2 at 42:30?

[@sherlockdong5591]

Converting to unit vectors(it’s already 4months I hope u get it already lol

[@lugia8888]

To normalize

[@apekshasingh8016]

Sir....could we have any solution book of your book linear algebra and its applications fourth addition.

[@ducthangnguyen0108]

Congratulations if all you guys are still here, the lecture 17

[@ianjoshi6778]

Why does he do 1/root2 at 11:00?

[@shubhamkumarsinha1380]

to make column vector orthonormal.

[@DeadPool-jt1ci]

Because he wants it to be an orthonormal matrix. Which means all columns are PERPENDICULAR to each other and of UNIT LENGTH.He shapes his matrix in such a form that the 2 columns , when dotted with each other ,return 0 , which is what we would expect of perpendicular vectors.However They do not have unit length. The length of each column vector would be sqrt(1²+1²) = sqrt(2). So if we wanna "normalize" the vector , meaning turn it into a unit length vector , then we just divide by the length. Since everything iin the matrix is divided by the same number , the dot product remains 0 (you can just factor the divisor out),and now the column vectors have length 1

[@quirkyquester]

@@DeadPool-jt1ci thank you so much! This is helpful!

[@quirkyquester]

Unit vectors | Matrix transformations | Linear Algebra | Khan Academy kzbin.info/www/bejne/ooLRaJmhqMZ3p5I

[@StewieGriffin]

85 years old not a single white strand of hair in sight.

[@cappy6436]

*65

[@amanjain256]

P = A.x(hat) ,right ? so at 32:10 why is he substituting P as x(hat).A. Please help me with the intuition behind that.

[@TheSalosful]

From the lecture on projection: for projection in two-dimensional space, you use the following formula: A * A^t / (A^t * A) * b, which is the projection matrix times the projected vector b: p = P * b. You can rewatch the video on how he derived it. Basically, you find out what is x(hat), substitute it in P = A * x(hat), and you'll get P = A * A^t / (A^t * A) for the two-dimensional space.

[@lugia8888]

@@TheSalosful not quite

[@tchappyha4034]

47:34 A = Q*(Q^T*A) and obviously Q^T*A is an upper triangular matrix.

[@anilaslihak]

It is said A^T*Q=R, but it should be Q^T*A=R, am I wrong, is that what you are saying?

[@DekhiJagi]

justice for schmidt

[@styloamnald]

25:32

[@rabinadk1]

This man hates Schmidt. XD

[@momke8169]

w lecture

[@bfl9075]

What did Schmidt do actually?

[@harshitagarwal6870]

I think Prof. Strang really hates Mr. Schmidt ;)

[@seanpitcher8957]

Dear God, no wonder those MIT kids are dangerous with teachers like this.

[@3v3pirat37]

I think im just to stupid for this .......I just dont get it .......

[@lugia8888]

what dont you get? a lot of this is computational.

[@rutika17]

👍👍

[@alenjose3903]

kookoo transpose