Regression Episode 2: Ordinary Least Squares Explained

In this episode of Office Hours Dan extends his prior introduction of the linear regression model to show how the model is fit to sample data using ordinary least squares estimation. Dan’s presentation is less mathematical and more conceptual, and attempts to provide some insight into what happens behind-the-scenes when estimating a regression model in practice...
Dan begins by reviewing the three parameters that define a one-predictor regression model: the intercept, slope, and residual variance. He then describes the analytic goal of obtaining optimal estimates of these parameters from the sample data. He uses graphical representations to highlight the goal of calculating sample estimates that result in the smallest possible sum of squared residuals. This episode establishes several basic principles that will be revisited in the following two episodes on inference.

[@laideadigun9980]

Thank you, you have really simplified OLS!

[@TennisRulez007]

Thank you so much, you explained this so well

[@raulhou1739]

thank you so much, i had this question of why not using absolute number since the first stat class in my first year of undergraduate. now I have an answer finally

[@arielhaven8488]

Thank you! You made it easy to understand!

[@file_one]

Thank you, great explanation!

[@Kenkoopa44]

Really nice video, thank you!!!

[@justindaniel8363]

Can you use a OLS Regression with a dichotomous measure? So 1= yes, 0= no?

[@centerstat]

Hi Justin, If you are referring to the predictor side (x), yes, this is straightforward. You can code dichotomous predictors in a variety of ways (the simplest being 0,1) and include it in the model. Most regression textbooks will describe this in detail, as well as categorical predictors with more than two levels. If you are referring to the outcome side (y) then the answer is more complicated. Say you have a one-predictor model y = b0 + b1x + e. The expected value of y is then E(y) = b0 + b1x. For a binary outcome, an expected value is also a probability, i.e., E(y) = P(y=1). Thus, when applied to a binary outcome, the linear regression model is referred to as a linear probability model, as P(y=1) is a linear function of x. There are some problems with this model. First, binary variables are inherently heteroscedastic (don't have the constant variance we usually assume in OLS). The variance of a binary variable is actually P(y=1)*P(y=0), so it is a direct function of the expected value. Given this, if you use a linear probability model you need to also use the White estimator to obtain corrected standard errors that are consistent under heteroscedasticity. Otherwise your inferential tests will be off (incorrect p-values and confidence intervals). A more fundamental problem is the linear aspect of the linear probability model. If x is continuous, there's nothing to constrain E(y) to the zero-one interval and you can get impossible implied probabilities that are negative or exceed one. Finally, a third problem is that the residuals won't be normally distributed (mainly a problem for inference if you have low N). For all of these reasons, most analysts prefer to use the logistic regression model when working with dichotomous outcomes. The logistic regression model explicitly takes into account the binary nature of the outcome. Expected values are bounded between zero and one (follow a logistic curve rather than straight line) and the outcome is assumed to have a binomial (or Bernoulli) distribution at any given level of x. Using the binomial distribution captures heteroscedasticity and replaces the assumption that the residuals are normally distributed. So, all around, the logistic regression model is your best bet for a binary outcome (though you will still sometimes see people use the linear probability model). Hope this helps... Dan

[@upekshaish]

@@centerstat thank you it helped me too