Simple Linear Regression: Checking Assumptions with Residual Plots

An investigation of the normality, constant variance, and linearity assumptions of the simple linear regression model through residual plots.
The pain-empathy data is estimated from a figure given in:
Singer et al. (2004). Empathy for pain involves the affective but not sensory components of pain. Science, 303:1157--1162.
The Janka hardness-density data is found in:
Hand, D.J., Daly, F. , Lunn, A.D., McConway, K., and Ostrowski, E., editors (1994). The Handbook of Small Data Sets. Chapman & Hall, London.
Original source: Williams, E.J. (1959). Regression Analysis. John Wiley & Sons, New York. Page 43, Table 3.7.

[@48956l]

I'M GIVIN THIS VIDEO THE BIG CHECK MARK

[@jbstatistics]

Thanks!

[@read89simo]

ME TOO + A BIG SUBSCRIBE

[@messididit]

@@read89simo + A BIG LIKE BUTTON

[@user-pl7zr2jm5h]

this was posted 11 years ago T-T and has the best explanations and videos on statistics I have ever found, thank you so much for all your hard work and legacy, i hope you know you're my savior.

[@jbstatistics]

I'm glad to be of help! 11 years, where'd they go? :)

[@jbstatistics]

"think you guys should get more views..." Thanks! (And I'll take as a compliment that you said "you guys", since this is a one man show.) Getting lots of views isn't very high on my priority list -- I'm just trying to provide the best resources for my students that I can. (I haven't done any promotion, and I don't allow ads on the videos.) There are many students in intro stats in North America and around the world, and I'm glad that some of them find my videos helpful.

[@williamlee0]

I'd upvote you x10 if I could just for the anti-advert policy.

[@nkululekoshabane3373]

One of the best, if not the best, video on regression analysis I've seen. Thank you very much for creating it. Your service is highly appreciated.

[@jbstatistics]

Nkululeko Shabane You are very welcome, and thank you very much for the compliment!

[@GuppyPal]

This is exactly what I have needed. My professor goes over these plots but has been doing statistics at a high level so long that I think it's hard for him to relate to someone who is new to it. I really needed someone to just explain it all from start to finish, and you did that. Thank you so much! Your videos are so, so helpful. Sincerely, a first year statistics graduate student.

[@jbstatistics]

I'm glad to be of help!

[@valeriereid2337]

Thank you for this excellent lecture. It certainly helps.

[@jbstatistics]

You are very welcome Simon!

[@johncasey722]

I'm so fricking glad these videos align well with my UIUC stats class. Much appreciated!

[@doodelay]

"The residual plot removes that increasing trend and then re-scales the y axis, so it's a little bit easier to see these issues.. sometimes in the residual plot." Now that is some serious insight. Thank you so much and this video was superb with really excellent examples!

[@jbstatistics]

Thanks for the kind words!

[@snake1625b]

Excellent methods used to help students learn in this vid. This is the future of education!

[@raseshgupta6276]

I was struggling to understand the assumptions in simple linear regression through other sources. This video has made it clear

[@jbstatistics]

I'm glad you find them useful John. Best of luck in your course!

[@hichamitani6433]

Thank you Need more like these videos on outliers in residuals

[@deniskapliy2642]

Small...and then they're big...and then they're small...and then they're big.. Great video, pretty simplistic, but very useful, thank you!

[@dedraryqui5606]

very clear, easy understandable video

[@Maha_s1999]

Prof Balka knocks it out of the park every time! We miss your videos. Could you do some videos on multiple linear regression? Hope you come back soon with new vids!

[@jbstatistics]

Thanks for the compliment! I'm trying to make time for video production, but probably won't get back to it until the new year. It's been a busy few years, but returning to the videos has always been part of the plan (with multiple regression videos up near the top of the list). Cheers.

[@Maha_s1999]

YAY!! Thanks Prof !! I will look out for them.

[@jbstatistics]

We often simply rely on an appropriate sampling design or experimental design to ensure independence. But if, say, we have recorded the observations in some sort of time order, then plots of the residuals through time can give us some indication of whether the residuals are correlated.

[@Jelly-cy4vh]

This was very useful, thank you for all the information

[@hritwick1221]

you are great man . thanks for your content . I am forever great full to you .

[@rodrigopaolinelli6448]

This is a definitely a great video, thank you! You are awesome!

[@MohamedAbdo-xs7bf]

You are Awesome! Thank you so much for sharing your valuable knowledge.

[@muhammadusama1558]

The more I watch your video, the more I hate my uni. Much love man

[@Pavankumar-zw2fz]

Very good Explanation Sir.Thank You

[@carnationize]

Thanks a lot! All your videos on stats are very clear and have been very helpful!

[@jbstatistics]

You are very welcome!

[@shayd146]

JB thank you so much you have helped me more than you'll ever know! My only suggestion to you would be to create playlists for associated topics. Other than that your teaching methods are incredible! Thanks!

[@jbstatistics]

Thanks very much for the compliment Shaydoyle! I believe I do have playlists ordered by topic. I've also set up a website (www.jbstatistics.com), which keeps the videos in a more organized fashion. (I'm not plugging anything on the site - it's just organized lists of my videos.) Cheers.

[@bharathganeshkumar7071]

Thanks for their video.. Short and sweet...!!!

[@jbstatistics]

You are very welcome!

[@rahkshi96]

Thank you very much jb statistics. This is incredibly helpful and well explained.

[@jbstatistics]

+Peter Song You are very welcome. Thanks for the compliment!

[@syedahmedali7417]

you are such a great teacher...

[@jbstatistics]

Thanks!

[@linneajohansson3796]

This was very helpful! Thank you!

[@angelinelam5862]

Thank you for this useful video !

[@mostafaali8684]

Good video, thank you very much for uploading it.

[@jbstatistics]

+Mostafa Ali You are very welcome. I'm glad you found it useful!

[@bibekanandasahoo3497]

thanks for this great explanation sir .....

[@Jemimakl]

So helpful! Thank you for this :)

[@selinechung1692]

LMAO BRO WHY ARE YOU HERE

[@Jemimakl]

Seline Chung WHY ARE YOU HERE

[@selinechung1692]

@@Jemimakl WHY ARE YOU SO HARDWORKING

[@selinechung1692]

@@Jemimakl BRO YOU STARTED A WEEK AGO

[@Jemimakl]

Seline Chung I WAS DOING HOMEWORK

[@jamiebond8481]

good and simple explanation of residual plots and assumptions.

[@jbstatistics]

Thanks!

[@wenlidi1604]

very clear explanation.

[@TB3hnz]

<a href="#" class="seekto" data-time="252">4:12</a> "I'm giving this the joker variance, because *let's put a SMILE on that FACE!* "

[@dylanburns9381]

great video. such a clear explanation. subbed.

[@pubgvulcanizer7857]

Very nicely explained 👍

[@babc24]

Very nice teaching. Thanks

[@Stephanbitterwolf]

Great video! Thank you!

[@sivanschwartz3813]

thank you for this amazing video!!!!!!

[@jbstatistics]

You're very welcome!

[@ananyapamde4514]

Great video!

[@hanaizdihar4368]

this really helps, thank you

[@CHIRAGPERLA]

This is gold!

[@Riley8185]

These are very good videos

[@yingdili2219]

perfect video

[@savageprincess2796]

im giving this video A BIG CHECK MARK (2)

[@davidli6068]

thanks a lot your a king

[@DHDH_DH]

Still extremely helpful in 2024

[@GlorifiedTruth]

So helpful! Thanks.

[@renshiue]

nice and clear

[@siryohannb3626]

thankyou very much

[@frederikhe707]

Nice! The only improvement I would suggest is that you actually name the violated assumptions. I mean people can draw that conclusion on their own but that would make it even more clear.

[@aayushiagarwal6188]

Perfectly explained ✨️ Could you please let me know, if the white centre line (the one around which all the ebsilon points are there) is itself not straight and showing a pattern ,what do we interpret? Does this mean that the mean of errors is non zero and hence our assumption is contradicted?

[@Bombingp]

Thanks! Helped a lot!

[@jbstatistics]

You are welcome!

[@JoaoVitorBRgomes]

At <a href="#" class="seekto" data-time="116">1:56</a> you can't plot against Y because there is dependence between Y and the residuals? You mean the residuals are the difference between the observed and the estimated, so makes no sense to plot against the observed? But why? Could you clarify this?

[@willtube9]

Prof, Could you do some videos on multiple linear regression? Hope you come back soon with new vids!

[@VivianGameCollections]

safe my day

[@simonschacht1810]

Thank you

[@frederickrosas5248]

Hi Sir. May I know what statistical tests/treatments being used in residuals plots to confirm what is allowed and not? Thank you for your help.

[@aabinamasoodgundroo5971]

my graph is blank, what does that mean?

[@carlosaugusto212]

Shouldn't we analyse the standardized residual plot? I mean, the residuals will be naturally bigger as the y value gets bigger, won't it? If the y range goes from 0,1 to 10 thousand, we expect bigger residual absolute values near the 10 thousand mark. Correct me if I'm wrong, please

[@Patriciacx]

Thank you!

[@jbstatistics]

You are very welcome!

[@sarita-ey5cw]

+jbstatistics ऊघजेऐऊ

[@infoesenn]

Question: Why do you assume normally distributed errors? From my understanding, in large samples iid-errors with from any distribution should be sufficient (Central Limit Theorem).

[@nightwalkers5579]

think you guys should get more views... may be there are not enough stats students in the country

[@purityrima1366]

@jbstatistics, thank you so much for helping me understand these plots! You are the best teacher:) I give you a big check mark for this video too. awesome explanation!

[@JoaoVitorBRgomes]

<a href="#" class="seekto" data-time="203">3:23</a> what kind of graph indicates non normality?

[@pate1495]

I have a question regarding the Normal Q-Q plot. On the y-axes, does it show the quantiles of the residual distribution, or the residuals itself? On the x-axes it shows the quantiles of the residual distribution if it were normal, correct? Thank you, great video!

[@jbstatistics]

There are different ways of formatting these plots, but here I have the ordinary residuals on the y axis. (The y axis value for any point is the ordinary residual of that point.) Any value could be considered a quantile. The x axis represents the corresponding quantile from the standard normal distribution. So if the residuals were normally distributed, we'd expected those values to fall (roughly) in a linear pattern. (There are some technical issues here, as the observed residuals aren't technically iid normal, even if the OLS assumptions are true, but it's a rough approximation.)

[@purityrima1366]

Please can you share me a link with your video on how to correct the unequal change in variance problem shown on the plots. Thanks in advance

[@ujasdiyora2804]

I have one doubt, here we are talking about simple linear regression in all of these videos in playlist. So this assumptions are also true for linear regression, multiple regression and polynomial regression ? , and all of these theory of finding confidence interval and hypothesis testing at the end to find whether coefficients are statistically significant or not , are these methods also applied in any other linear regression ?

[@jbstatistics]

The general idea still holds, yes. The specific formulas for the standard errors, degrees of freedom, etc., will change when there is more than one predictor. And there are many subtleties when it comes to multiple regression, so it's best to learn all about MLR rather than think something like "well, it's just like simple linear regression but with more predictors." That said, yes, the general ideas port over from simple linear regression to multiple linear regression in a natural way. Polynomial regression is a type of multiple regression, so same idea there.

[@harrygroundwater2590]

Ver helpful

[@SaranathenArun11E214]

brilliant

[@jbstatistics]

Thanks!

[@lowerterror7993]

No one people like data analytics

[@maydin34]

Nice video.Thank you. But it is just plotted between random part vs independent variable(x). What if we have multiple independent varibles ( say z,t,w etc.). Do we need to check for all those seperately by expecting very same variance again regardless of independent variable? (random vs z, random vs t , etc.) Or is it ok just plotting predicted-y vs random part?

[@sanjaypandey6586]

is it ok in linear regression if dependent and independent variable are not normally distributed if not what should be the optimum solution for negative skew and neg kurtosis

[@ohhrelingo6271]

If I can't find out if the variance is constant from the plot what should I do?

[@davidchau6874]

how to check the independence in the residual plot?

[@Kingshuk91]

Great video. How is plot of e vs time and plot of e(t) vs e(t-1) different?

[@abhishekbhatia6092]

While interpreting the residual plots, can I first pool the residuals in specific bins of X (say each bin 1 unit long or whatever) , so that it looks more like the previous plot with residuals for a given value of X, enabling me to verify the homoscedasticity (and also normality somewhat) more clearly? Edit: Q) You mentioned that one of the assumptions was that for a given value of X, the error terms are normally distributed with a constant variance sigma-squared (same for each X). Then at <a href="#" class="seekto" data-time="350">5:50</a> you took all the residuals disregarding the value of X, and graphically checked it for normality using a Q-Q plot. Didn't you mention that the normality assumption was for errors for a given value of X? I am confused. pls help.

[@n9537]

answer to edit: If we assume that sampling was completely random, then data from all treatments/groups/sub-populations/values of X were equally likely to be represented in your sample.In that case all the residuals can be clubbed together and checked for normality.It s same as checking for each treatment group.Note this applies only for the residuals, not the variables.

[@n9537]

In regression we usually have predictor variables continuous. so it is impossible to check normality for each value of X. in case of ANOVA , the predictor is usually categorical and you can venture to check residual normality for each treatment group/category.Both ANOVA and Regression come under Generalized Linear Model(GLM), so the assumptions are the same but they play out differently.

[@n9537]

Actually all assumptions are on the error terms.But since Residuals are an estimate on the error, we check for "good behavior" on the residuals. We have to make do with what we have(which is the residuals, the error is unknown)

[@n9537]

this also follows from the assumption that error ~ i.i.d N(0,sigma^2). So all residuals(used in place of error as a good estimate) are identically distributed(same mean and variance) and are independent of X , implies you can't look at a set of residuals and figure out which value of X it came from. For all you know, they all could be from the same value of X or different values of X.Needless to say, they must be sourced from the same population, you can't club residuals from different populations/different predictor(s). So for checking normality of residuals, you can disregard value of X.This is not the case for Y (dependent variable).

[@karrisgiani5137]

Brill video! If residuals appear to show an inverted U, how can I improve the model?

[@jt007rai]

at <a href="#" class="seekto" data-time="261">4:21</a> , can we determine which model will solve this issue based on just looking at this residual plot?

[@omkareshpali8486]

Hi I have a question, let's say I built a model and the R2 value came out 70% How do I make sure that is the maximum variance I can explain by looking at the residuals.

[@Kaa279]

in <a href="#" class="seekto" data-time="280">4:40</a> you said that there is another feature that we didn't included in our model. but it can also conclude that my model is not good, right?

[@KingQuetzal]

So I got the <a href="#" class="seekto" data-time="252">4:12</a> graph how can I find out what kind of data I have?

[@MohitSingh-ub9gc]

but why are we doing this, please explain ?

[@yousifsalam]

@<a href="#" class="seekto" data-time="280">4:40</a> why did you say the residuals are small then big then small.. don't you mean they're negative, positive.. since their magnitude is the same?

[@dariopl8664]

I think it's because "ε" is a random variable (as he mentioned it in previous videos), and should stay so. If they appear at time sections up and then a bunch of them down, that randomness breaks up, since when a whole of them are up you can forsee they'll be down next time (then where's the randomness?). I think that if all are up the same amount they're down (as you see @4:40), then, would they still have a normal distribution? no, it would be just a straight line probability distribution, in which you know the moment you're up, next will be down, and so on. This model assumed ε follows a normal distribution, which is reasonable, since in real life many events occur this way. If they're are jumping up and down in clusters then we're not dealing any longer with this reasonable distribution. But of course, at the end he'd some way deal with this time effect he didn't know beforehand was causing this, maybe so as to normalize them, as they should be to fit the model🤔. I don't know yet how he tackles this problem. If I find about it I'll tell you. Hope this reply was helpful. Best regards.

[@ogedaykhan9909]

E X C E L L E N T Thanks a lot

[@Doh333]

Would it be relevant to make residual plots if i want to check a categorical variable in a lineare regression ?

[@jbstatistics]

Yes, some types of residual plots are still informative for categorical explanatory variables. With a categorical variable, a check for linearity is not required, but residual plots can still help to check the normality and common variance assumptions.

[@alexkay7199]

Please help: Do the residuals have a unit or are they unitless???

[@jbstatistics]

The residuals are the differences between the observed values of Y and the predicted values of Y. The units of both the observed and predicted values of Y are just the units of Y, and thus the units of the residuals are the units of Y.

[@alexkay7199]

@@jbstatistics Thanks a lot! Really helpful!!

[@Nias0404]

I'm not sure how to get the Q-Q plot... can anyone explain?

[@jbstatistics]

It's almost always created using software. My intro to normal QQ plots is found here: kzbin.info/www/bejne/jmrCeoaAZb6jfdk

[@Nias0404]

Thanks a lot!! Much appreciated

[@MrPreston1056]

Is there a way to t test the residual plot?

[@jbstatistics]

What kind of test are you hoping to do? There isn't going to be an overall increasing or decreasing trend in the residuals in simple linear regression. There may be curvature, and we could test to see whether adding higher order terms (e.g. X^2) results in a significantly improved fit. Cheers.

[@MrPreston1056]

Can you use the t statistic to test if H: E(e "hat"sub i)=0 vs. H: E(e"hat" sub i) not equal to zero. E being mean and e "hat" being error

[@jbstatistics]

Preston C No, we can't test that. The (observed) residuals always sum to 0 in simple linear regression. When we say the expectation of epsilon is 0 (at every X), we are in effect saying that E(Y|X) falls on the line beta_0 + beta_1X. Conceptually, we could have a different model where the expectation of epsilon was assumed to be 2 instead of 0. This would change very little, except that beta_0 in this model would be 2 less than beta_0 in our usual model. This would unnecessarily complicate things, so we define epsilon to be a random variable with a mean of 0. Cheers.

[@MrPreston1056]

But if we tested to see if e_i=0 vs not equal to 0 and rejected the null hypothesis that e_i=0 wouldn't that indicate that the residuals did not sum to zero and our previous assumptions were false?

[@jbstatistics]

Preston C The observed residuals sum to 0. That is not an assumption, it is a consequence of the least squares fit. If we attempted to test the null hypothesis that the true mean residual is 0 with a t test, we would end up with a test statistic of 0 and a p-value of 1. So that wouldn't really be a test. If you're wondering about testing the null hypothesis that E(epsilon) = 0 at *any given value of X*, that's a bit of a different story. We do something along those lines when we carry out a lack-of-fit test. (This tests the null hypothesis that the means do indeed fall on a line. We can do this sort of thing when we have multiple observations at at least some of the X's.)

[@ukrainrussiawarvideos2810]

A GOOD GAIED Program

[@unofficiallyofficial2149]

Probably a simple model for college students, not high school.

[@jbstatistics]

The "simple" in simple linear regression refers to there being only one predictor (one x), and not because it's simple or easy. It's just the well-established name of the model. Unlike many others, I don't use any clickbait words like "easy" or "simple".

[@unofficiallyofficial2149]

@@jbstatistics Oh, I understand. Thanks.

[@JoshuaDHarvey]

Nothing he is explaining makes any sense.

[@jingwen8133]

Very useful video ! Thank you