Linear Regression

Linear Regression, Part 3: Diagnosing Linear Regression -- When Your Standard Errors Lie

Three problems that leave the coefficients themselves untouched and quietly wreck everything built on top of them -- with a real dataset where a naive standard error is off by 3.6x, and a simulation where a nominal 5% test rejects a true null 76% of the time

The third installment of the Linear Regression series, covering the first of two failure buckets the series is organized around: problems that leave OLS's coefficient estimates exactly as unbiased as Part 2 proved, but wreck every standard error, t-test, F-test, and confidence interval built on top of them. Unifies heteroskedasticity and autocorrelation under one derivation -- the general 'sandwich' variance formula (X'X)^{-1}X'ΩX(X'X)^{-1}, which collapses to Part 2's familiar σ^2(X'X)^{-1} only when Ω=σ^2I exactly -- then treats each with a real worked example: Engel's 1857 food-expenditure data, where the naive standard error understates the robust HC0 estimate by 3.6x, and Longley's macroeconomic data plus a Granger-Newbold (1974) spurious-regression simulation, where a nominal 5% test rejects a true null of no relationship 76.5% of the time. Closes with multicollinearity, deliberately treated as a different mechanism entirely -- not a broken formula but a correct formula applied to a design matrix where (X'X)^{-1} is itself large -- with an explicit, honest statement that neither this article's diagnostics nor Part 5's remedies fix it; that fix is the site's already-published Ridge Regression, Lasso Regression, and Elastic Net trilogy.

This article is part of an Analyze subscription.

Subscribe to read Linear Regression, Part 3: Diagnosing Linear Regression -- When Your Standard Errors Lie in full, along with every other article in the library.

See plans