Ridge Regression
A complete, worked treatment of the L2 penalty -- its closed form, its Bayesian reading, and the theorem guaranteeing it can beat ordinary least squares -- run start to finish on a real, famously ill-conditioned 1967 dataset, including the moment automatic cross-validation gets fooled by it
The dedicated follow-up to the Regularization article's brief preview of Ridge, covering everything that article deliberately left out. Derives the ridge estimator's constrained-optimization and Bayesian MAP interpretations in full (the latter giving lambda a precise meaning as a noise-to-prior-variance ratio), proves the Hoerl-Kennard (1970) existence theorem that some amount of shrinkage strictly improves MSE over OLS for every dataset with real noise, and works through the SVD view of differential shrinkage and effective degrees of freedom. The centerpiece is a fully worked, numerically verified example on Longley's 1967 macroeconomic dataset -- famous for near-perfect multicollinearity and a wrong-signed OLS coefficient on GNP -- computing the real ridge trace, generalized cross-validation (Golub-Heath-Wahba, 1979), and leave-one-out cross-validation from the actual data. The trace and the automatic criteria disagree: GCV's minimum lands inside the region the trace itself shows to be unstable, a genuine and explained finding, not a toy example engineered to make ridge look good. Closes with a practical decision checklist and an honest 'no free lunch' accounting of what ridge does and does not fix.
This article is part of an Analyze subscription.
Subscribe to read Ridge Regression in full, along with every other article in the library.
See plans