Economics 215, 2015 Allin Cottrell
OLS cheat sheet
Here are some basics that you should know about Ordinary Least Squares. Note that several of the points that
are simply asserted here are proved and/or explained more fully in the notes titled “Regression Basics in Matrix
Terms”. Key assumptions are marked as, for example, “[A1]”.
1. The linear multiple regression model can be written compactly in vector–matrix form as
y=Xβ+u(1)
where the dependent variable yis n×1; the regressor matrix Xis n×k; the parameter vector βis k×1; and
the error term uis n×1.
2. The OLS estimator of β, which we write as ˆ
β, is given by
ˆ
β=(X0X)−1X0y(2)
This exists provided that X0Xis non-singular, which requires that the Xmatrix is of full column rank (no exact
collinearity among the columns of X, [A1]).
Assuming ˆ
βexists, two useful additional vectors may be formed: fitted values, ˆy=Xˆ
β, and residuals, ˆu=
y− ˆy=y−Xˆ
β.
3. If the data-generating process conforms to (1) [A2] then the expectation of ˆ
βis given by
E(ˆ
β) =β+Eh(X0X)−1X0ui(3)
On condition [A3] that E(u|X)=0 the second term above disappears and we have E(ˆ
β) =β, or in other
words the OLS estimator is unbiased.
4. The variance of ˆ
β(a k×kmatrix) is, from first principles,
Var(ˆ
β) =Eˆ
β−E(ˆ
β)ˆ
β−E(ˆ
β)0
If the condition for the estimator to be unbiased is met, then
Var(ˆ
β) =(X0X)−1X0E(uu0)X(X0X)−1(4)
If the error term has a constant variance, σ2
u[A4], and the drawings from the error distribution are independent,
such that E(uiuj)=0 for all i6= j[A5], then E(uu0)=σ2
uInand the OLS variance simplifies to the
“classical” formula,
Var(ˆ
β) =σ2
u(X0X)−1(5)
which can be estimated by using
s2
u=Pn
i=1ˆu2
i
n−k
in place of the unknown σ2
u.
5. Note the role of the various assumptions: [A1] is required for ˆ
βto exist; in addition, [A2] and [A3] are
required for OLS to be unbiased; and in addition [A4] and [A5] are needed for “classical” standard errors to be
valid. (The standard errors routinely reported alongside OLS estimates are just the square roots of the diagonal
elements of Var(ˆ
β)).