An Introduction to Regularization

Motivates regularization from a precise failure of ordinary least squares: correlated predictors inflate coefficient variance for purely statistical reasons that hold even under exact arithmetic, while extreme ill-conditioning separately threatens the floating-point computation itself, at a much higher threshold, verified here by direct simulation. Also covers the sharper p >= n case (true non-identifiability, not just instability), introduces the general penalized-estimation framework and the bias-variance tradeoff behind it, and previews the three specific penalties -- Ridge, Lasso, Elastic Net -- each covered in its own dedicated article next.