Linear Regression and Multiple Regression

In this video, I will be talking about a parametric regression method called “Linear Regression” and it's extension for multiple features/ covariates, "Multiple Regression". You will gain an understanding of how to estimate coefficients using the least squares approach (scalar and matrix form) - fundamental for many other statistical learning methods.
⭐ Coursera Plus: $100 off until September 29th, 2022 for access to 7000+ courses: imp.i384100.ne...
MATH COURSES (7 day free trial)
📕 Mathematics for Machine Learning: imp.i384100.ne...
📕 Calculus: imp.i384100.ne...
📕 Statistics for Data Science: imp.i384100.ne...
📕 Bayesian Statistics: imp.i384100.ne...
📕 Linear Algebra: imp.i384100.ne...
📕 Probability: imp.i384100.ne...
OTHER RELATED COURSES (7 day free trial)
📕 ⭐ Deep Learning Specialization: imp.i384100.ne...
📕 Python for Everybody: imp.i384100.ne...
📕 MLOps Course: imp.i384100.ne...
📕 Natural Language Processing (NLP): imp.i384100.ne...
📕 Machine Learning in Production: imp.i384100.ne...
📕 Data Science Specialization: imp.i384100.ne...
📕 Tensorflow: imp.i384100.ne...
INVESTING
[1] Webull (You can get 3 free stocks setting up a webull account today): a.webull.com/8...
More on Matrix Calculus: atmos.washingt...

[@xavierfournat8264]

Fantastic work. Usually all tutorial videos about linear regression or multiple regression are simply giving the formulas out of nowhere, without explaining the rational in the background. Thanks for taking the time for diving through the underlying maths :)

[@speakers159]

Pretty amazing, especially since nobody really covers the mathematics behind ML, really appreciate the math based content.

[@CodeEmporium]

Yesss! Math is underappreciated

[@joelwillis2043]

this is a very well made video but this is always covered in statistics

[@sidharthramanan6780]

This is a great video - I was looking for the math behind calculating the co-efficients in multiple linear regression and this explains it perfectly. Thank you!

[@CodeEmporium]

Thanks Sidharth! Glad it helped! Mind sharing the video to help others like you? :P

[@sidharthramanan6780]

Thank you for the video! And I'd love to share it with others :) Also, you just got a subscriber! Let's see you get to 1K soon !

[@CodeEmporium]

Thank you! Much Appreciated! I'm trying to upload more regularly than I have done in the past. There should be a lot more where that came from very soon.

[@sidharthramanan6780]

Yup! Any platform I can network with you on by the way? Quora for example?

[@CodeEmporium]

Sidharth Ramanan Quora is good. I'm under the name "Ajay Halthor".

[@martinsahmed9107]

This exposition is timely. I have battled over the disappearance of Y transpose Y in the matrix approach of a least squares for months until I came across this video. This is awesome. I am speechless.

[@sanathdas4071]

very well explained. I have been searching such video for many days. Now, the concept is crystal clear.

[@bluejays440]

Excellent video, highly illuminating to finally see a comprehensive explanation of things that are too often left unexplained. I wish far more people, books, and videos explained statistics in similar detail.

[@yuxiongzhu4249]

Wow! This is the best video to quickly understand the derivation of linear regression formulas!

[@arpitbharadwaj8799]

after multiplying and opening the brackets at 9:00 third term of the resultant should have transpose of B hat and not just B hat

[@chachajackson_original]

Correct

[@physicsfaith]

yes just a typo....

[@ledzeppelin1212]

Needed some refresher on a math class from grad school, and this really hit the spot. Thank you!

[@kerolesmonsef4179]

after week of searching . finally i found you . Thank you so much great explanation . keep going on

[@CodeEmporium]

The search is over. Join me in turning this world into -- nah just kidding. Glad you Finally found me. Hope you stick around

[@Anonymous-ho1mt]

I have tried many ways to find a decent derivation for multiple regression, I found the key term is understanding matrix derivation rules which I was missing all those times. this is first time I got the clear understanding of the formula. Thanks a lot.

[@trackmyactivity]

Amazing, thanks to the map you just drew I feel confident to learn the deeper concepts!

[@alphatakes]

this is the best video on multiple linear regression

[@CodeEmporium]

Thanks so much!

[@hyphenpointhyphen]

I am binging the concepts and might forget to like - great channel.

[@donaldngwira]

One of the best explanations on this topic. And the presentation is superb

[@anandachetanelikapati6388]

Excellent explanation with precise terminology!

[@FindMultiBagger]

Hats of for your efforts ! Really Fun way to learn algorithms, Please post more videos of other machine learning algo.

[@mustafizurrahman5699]

Splendid and now words are sufficiently enough for such lucid explanation

[@CodeEmporium]

Thanks for the compliments! :)

[@amogh2101]

Is there something wrong at @8:58? Shouldn't B(hat) be B(hat)(Transpose)?

[@MrStudent1978]

Very nice explanation! Very clear! I was looking for exactly the same.

[@anujlahoty8022]

This video just made my day. Absolutely loved it...

[@kuntsbro4856]

Thanks a lot. This is the most comprehensive regression video on KZbin.

[@CodeEmporium]

Kunt's Bro Thanks! Regression is an important topic, thought I'd take time explaining it

[@satyendrayadav3123]

Hands down my dawg❤️❤️ Very well explained

[@capnic]

Wonderful video! very useful and clear!

[@mike_o7874]

Great video, exactly what i was searching for, how did they get that matrix equation was exactly what i needed! thanks a lot man!

[@ashlee8140]

This is a great video and explained things so clearly! Thanks!

[@CodeEmporium]

Thanks for the compliments! :)

[@leochang3185]

At 11:10, the quadratic form of matrix differentiation should be x^T(A^T + A). Under the condition of A being symmetric could the derivative be 2 x^T A (as being used in the last term of d(RSS)/dx).

[@kastenkarsten]

Incredible video for the derivation!

[@elishayu8002]

Super helpful and very clear! Thank you so so much!

[@CK-vy2qv]

Excellent! Very nice see the scalar and matrix approach :)

[@souravdey1227]

Extremely clear. Bang on!

[@jean-michelgonet9483]

In minute 10:27: X is mx1 and A in mxn. The 3rd differentiation rule is about y = X*A. But, given the sizes of the matrices, how can you multiply X*A?

[@noelcastillo3369]

I have the same question. Were you able to clear it up?

[@xiaodafei4738]

here the differentiation rule should be: let scalar y=x^T A, then dy/dx = A^T It's nice that the video shows some matrix differentiation rules, but I recommend the more serious propositions in: atmos.washington.edu/~dennis/MatrixCalculus.pdf

[@jugglingisgreat]

You explained this 1000000000000000000000000000x better than my professor. Thank you!

[@CodeEmporium]

Ryan Smith Thanks! So glad it was useful!

[@JAmes-BoNDOO7]

Finally a video which makes perfect sense. Thanks a lot bro.

[@CodeEmporium]

Making sense is what I do :)

[@carolyneatieno1927]

This was really helpful. I'm taking a unit on Data mining with no statistics background. Thank for sharing your knowledge 👊

[@cecilia1300]

wow good luck!!!

[@ellenma3523]

beginning from 8:54 the RSS should have the third term as -(β_hat)^T X^T y instead of -(β_hat) X^T y, the transpose sign is missing here

[@nguyenkuro3827]

You're Right. And I think it should be:" y=x^T.A => dy/dx = A" .

[@gcumauma3319]

Excellently explained. Very lucid

[@CodeEmporium]

Glad this is useful. Thank you :)

[@amineelfarssi3902]

Very clear explanation … better than doing it by considering the projection on the model space and using the projection formula (t(X)X)^-1t(X)Y

[@user-gn7op1nq3d]

Thanks! You just saved my life!

[@CodeEmporium]

Raquel Morales Anytime. Saving lives is what I do.

[@MOTIVAO]

Amazing work

[@DM_musik-01]

Thank you very much it was very helpful

[@krishnachaitanyakr1237]

Very well explained. Thank you.

[@andrewzacharakis8583]

great job man !

[@lucaslopes9907]

Thank you! It helped me a lot.

[@MrKaryerist]

Great video! Probably the best in explanation of math behind linear regression. Is there a way to do multiple non-linear regression?

[@swapnilborse3150]

Thanks....put more videos on regression analysis

[@CodeEmporium]

Glad you enjoyed it! Will think of more Regression based videos in the future

[@NEOBRR]

Thanks, this is an amazing video. It was very helpful.

[@CodeEmporium]

Many thanks!

[@zachc5261]

Great video, thanks for your effort 😁 I just have two questions: 1. in the last RSS equation, why is T removed from beta_hat in the third term 2. how is y = xA feasible given x has dimension (m x 1) and A has dim (n x m) Appreciate your help please. Thanks!

[@benisbuff]

Great explanation

[@tomarkhelpalma138]

Way too cool!!! I am enjoying this video!

[@k_anu7]

1 question. The method that you described above is of normal equation as of andrew ng machine learning course. The other way to find coeff. are gradient descent, BFGS, L-BFGS etc. Correct me if I am wrong.

[@baa3124]

thank you so much from korea

[@snehashishpaul2740]

Very nicely explained

[@ahmedmarzouki3991]

thank you very much for this amazing video, it was really helpful do you have any other videos about : polynomial regression and non linear regression ?

[@tsehayenegash8394]

really nice explanation you have deep knowledge. hoa can we minimize the error term?

[@ashokpalivela311]

Very well explained!! Tq❤

[@nilankalord5974]

Great video, thank you!

[@Rigou27]

There's a mistake at minute 9:00, the third term of the expanded version of RSS is -(beta' x' y)

[@mohannadbarakat5885]

why is sum(sqr(e)) = e^T * e

[@juanitogabriel]

Nice video. What software do you use for writing that math expressions? I mean, is it editor equations from ms word? Thank you.

[@DiegoAToala]

Great math , thank you

[@tmrmbx5496]

Good job, thanks

[@rajeshsoma143]

9:40 How does the third formula work? Here, the dimensions of xA do not satisfy the condition for matrix multiplication

[@MrDiggerLP]

My friend, you Saved my Bachelorpresentation.

[@cghale3692]

when u removed the bracket...what happened to B transposition and X transposition while multiplying with Y? B transposition is not there just B is there ...the last line of simplification ?

[@xofjo6239]

why is that, I am very confused

[@raihanahmimi8233]

27.6K Subscriber on 13 July 2020... is that close enough from your prediction?

[@chaosido19]

I come from Psychology and am following data science courses rn. The completely different way of approaching regression was a mystery to me, but this video helped me a lot. I do feel like I should practise stuff like this myself too, do you have any suggestions for places where to find exercises.

[@CodeEmporium]

Thanks for commenting and watching! Maybe a textbook might be good for establishing a foundation. You can check out the “Introduction to Statistical Learning”. Aside from that I have I lol playlist on linear regression, though I admit it hops around some concepts. It still might be worth your watch.

[@rajinfootonchuriquen]

Something i don't understand is this: in simple linear regresión, you take the mean of square of error, but in múltiple regresión, what happend with taking the mean? X and y in the result fórmula have components with the mean?

[@christophereicher3928]

Thank you!

[@surajjoshi3433]

Why y predicted=beta hat X?? Instead Beta not should also be included

[@dellazhang5785]

Super good

[@Dra60oN]

Hey, in 2nd example, you got y = xA, how can you even multiply those two when dimensions don't match? (m x 1) * ( n x m) , thus 1 != n Similar for 4th example where you got y = transpose(x) Ax ... I think A should be square matrix in this case (mxm).

[@CodeEmporium]

2nd example: y = Ax, not xA. 4th example: You're right here. x^T A x has shape (1 x m) *(n * m)*(m*1). This is true if n = m i.e. A is a square matrix. Good catch! Should have mentioned that. In the derivation, we use it with X^T X -- which is square.

[@Dra60oN]

Hey, sorry my typo,I was referring to the 3rd example, y = xA, not the 2nd one. And also are you sure that the last term B^T * X^T * X * B is the case of your 4th example. Because you can rewrite that expression as (X*B)^T * (X * B) and then it's a norm squared of matrix, and you say g(X) = X * B, and then you can apply derivative with respect to beta given by this formula: 2 * g(X)^T * d(X*B) / dX, which in this case would yield the same result, so after all you might be correct as well. All the best.

[@youcefyahiaoui1465]

In your logistic regression, I am not sure how you came up with the two exponents when you formed the two product of the product of p(x) and 1-p(x)

[@yunqianggan2906]

Thank you so much, this video helps a lot :)

[@CodeEmporium]

Yunqiang Gan Thanks! Glad you liked it!

[@Wisam_Saleem]

GREAT!

[@akzork]

Can you upload a pdf of these formulae you shows in this video?

[@fernandoluis53]

this video made me more confused.

[@Dontonethefirst]

It's mostly for ML shit it's actually really helpful

[@norbertramaj3024]

well' it's almost 2020 and you have almost 40k subs. This what you predicted?

[@PetaZire]

thank you, you also saved my life :)

[@sergeyderbeko9954]

This video is so good, it explained several weeks of a course to me in 12 minutes smh

[@CodeEmporium]

Glad this is useful :)

[@eewitht3190]

amaizing. thanks

[@CodeEmporium]

Welcome! :)

[@surajjoshi3433]

9:44 Actually it should be 2AX if A is a symmetric matrix, am i correct ??,help me anyone please

[@rahuldeora5815]

Hey great video. Can you suggest some mathematical test that explains ML with some debt as in this video?

[@peacego624]

Thanks

[@CodeEmporium]

Welcome!

[@vishu8770]

Thanks a lottt

[@deeptigupta518]

Can you give us the reference for the matrix differentiation used here?

[@chd1024]

There's something I didn't understand. At 9mn, why in the third term of RSS, the beta hat has no transpose sign ? Thank you for this vidéo.

[@stevendale8813]

You're correct, that beta transpose should be there. If you substitute the correction and then calculate the derivative you end up with the same final equation. Here's how the derivative of the corrected term works out: delta(- beta^T * X^T * y) / delta(beta) = # derivative of corrected term wrt beta = - X^T * y # this comes from the derivative rule delta(a^T * B) / delta(a) = B = - y^T * X # this comes from the manipulation rule B^T * a = a^T * B (where 'a' is a vector and 'B' is a matrix) And that is equal to the derivative that he calculates despite the error. So you can rest assured that our answer still remains the same. Good catch. And also thank you @codeemporium for creating this awesome video. Helped me out a lot!

[@chd1024]

@Mohsen Kheirabadi Thank's a lot

[@chd1024]

@@stevendale8813 Thank you for your detailed answer

[@Trendtidesind]

At 7:22 the beta vector is Nx1 not Px1

[@BlackJar72]

Good, but I think I need to review some general math and sit down to work it out -- solving its not hard, but its good to know why it works.

[@CodeEmporium]

For sure. One can always use builtin libraries to code it in a single line. But understanding why it works the way it does will help understand when to use it.

[@purvanyatyagi2494]

Pls tell me that RSS is same as mean squared error

[@aigaurav5024]

Thanku so much

[@purvanyatyagi2494]

Even the most simple things are hard to understand in depth

[@runethorsen6230]

Is it possible that there is a little error at 12.54 min. 3'rd term of RSS: beta should be transposed?

[@ananthakomanduri1567]

THANKS

[@xeeeshaunali]

Which book you consulted??

[@techchit6797]

Does the inverse( X*transpose(X)) always exists in the formula? Why?

[@1UniverseGames]

How can we obtain intercept and slope of B0 and B1 after shifting line l to l'

[@deeptigupta518]

How did you get y=xA as A transpose ? As both x A doesnt have the dimensions to get multiplied?

[@BHuman2024]

8.54 min: last line 3rd term, I cannot match, could anybody clear me, please?

[@P3R5I3dark]

At 8 56 last line, 3rd term shouldn t it be BTXTy instead BXTY? It also doesn t make sense bcs matrix sizes dont fit

[@CodeEmporium]

Nevermore You're right. Someone mentioned that in the comments as well. When you take the derivative wrt Beta, you get the same result. So that part of the video is incorrect, but the rest is fine.

[@P3R5I3dark]

Yea the thing is -yTXB=BTXTy so you obtain deriv(RSS)=deriv(-2BTXTy+BTXTXB)=2XTXB-2XTY=0. Wich lead us to b=XTY*(XTX)^(-1) . Ty anyways for video, it helped to understand better this method. Tomorrow i have exam from this method's aplication in System's models.

[@CodeEmporium]

Death Xeris yup. You're right again. Still, glad the video was helpful. Good luck with your exam!

[@anmoltariqbutt8987]

@@P3R5I3dark -yTXB = BTXTy? How? If we use this equality then the whole term will be eliminated

[@juandesalgado]

It's a pity we have not defined what would the inverse of a non-square matrix would be. If we had, (XT . X)^-1 . XT . y would be X^-1 . XT^-1 . XT . y = X^-1 . y , and I'd have more time to play video games.

[@justahardworkingjoe]

Moore-Penrose. Pseudo Inverse.

[@juandesalgado]

@@justahardworkingjoe Thanks. Will read!