Theory • Practice • Insight
Statistics, Explained
Clearly
Read approachable guides, deep dives, and real-world case studies that connect statistical theory to the decisions you make every day.
Where These Ideas Come From
New articles go up every Tuesday and Friday. Most of the ideas come out of the research behind The Compendium of Statistical Tests — while working through the literature for that book, I kept running into concepts, historical facts, and stories that deserved more room than a footnote, and this is where they end up. The rest come from years of teaching and practicing statistics directly. Each one is meant to stand on its own: a small, self-contained piece you can read in one sitting.
These are well-researched, and you're welcome to cite them freely — just credit StatisticsCorner (see the “How to Cite This Article” section on each page). If you spot a mistake, I'd like to know: reach out through the contact form.
Showing 1–12 of 27 articles
The Anderson–Darling Test
From a weighted integral to the two familiar discrete formulas
A step-by-step derivation of the Anderson–Darling goodness-of-fit statistic, starting from the defining weighted integral and ending at the two discrete formulas used in software and textbooks. Along the way it shows exactly where the odd-number weights, the tail-sensitivity, and the mysterious −n term actually come from — and proves the two commonly cited discrete forms are algebraically identical, not different statistics.
Linear Regression, Part 4: Diagnosing Linear Regression -- When Your Estimates Are Wrong
Three problems that no amount of data fixes -- with a real dataset where dropping one variable shifts a coefficient by exactly a provable amount, a simulation where a coefficient converges to the wrong number at one million observations, and a market where OLS cannot tell supply from demand
The fourth installment of the Linear Regression series, turning to the second and more serious failure bucket the series is organized around: problems that violate exogeneity itself (A3), so that OLS's coefficient estimates no longer converge to the truth even as the sample grows without bound -- a categorically worse failure than Part 3's, where more data at least helped once the right correction was applied. Covers omitted variable bias with a real, exactly-verifiable example (dropping horsepower from a weight-on-mpg regression shifts the weight coefficient from -3.878 to -5.344, matching the Frisch-Waugh-Lovell identity to thirteen decimal places), classical measurement-error attenuation (injecting realistic noise into a real predictor shrinks its coefficient toward zero and keeps it there through n=320,000), and simultaneity bias (a simulated supply-and-demand market where a naive regression of quantity on price converges cleanly, even at a million observations, to a number that is neither the true demand slope nor the true supply slope). Closes with an explicit, honest statement that neither Part 3's diagnostics nor Part 5's upcoming remedies fix any of these -- that requires instrumental variables and two-stage least squares, which Part 7 develops properly.
Linear Regression, Part 3: Diagnosing Linear Regression -- When Your Standard Errors Lie
Three problems that leave the coefficients themselves untouched and quietly wreck everything built on top of them -- with a real dataset where a naive standard error is off by 3.6x, and a simulation where a nominal 5% test rejects a true null 76% of the time
The third installment of the Linear Regression series, covering the first of two failure buckets the series is organized around: problems that leave OLS's coefficient estimates exactly as unbiased as Part 2 proved, but wreck every standard error, t-test, F-test, and confidence interval built on top of them. Unifies heteroskedasticity and autocorrelation under one derivation -- the general 'sandwich' variance formula (X'X)^{-1}X'ΩX(X'X)^{-1}, which collapses to Part 2's familiar σ^2(X'X)^{-1} only when Ω=σ^2I exactly -- then treats each with a real worked example: Engel's 1857 food-expenditure data, where the naive standard error understates the robust HC0 estimate by 3.6x, and Longley's macroeconomic data plus a Granger-Newbold (1974) spurious-regression simulation, where a nominal 5% test rejects a true null of no relationship 76.5% of the time. Closes with multicollinearity, deliberately treated as a different mechanism entirely -- not a broken formula but a correct formula applied to a design matrix where (X'X)^{-1} is itself large -- with an explicit, honest statement that neither this article's diagnostics nor Part 5's remedies fix it; that fix is the site's already-published Ridge Regression, Lasso Regression, and Elastic Net trilogy.
Linear Regression, Part 2: Ordinary Least Squares and the Gauss-Markov Theorem
The estimator, derived; the exact assumptions it needs, catalogued precisely by what each one buys; the theorem proving it is the best available under those assumptions -- proved on paper and then shown, in a Monte Carlo simulation, to actually be true
The second installment of the Linear Regression series, turning Part 1's model equation into an actual estimator and the full inference machinery built on top of it. Derives the OLS normal equations and closed-form solution, then catalogues the classical assumptions precisely by what each one buys -- unbiasedness from exogeneity alone, the Gauss-Markov theorem's best-linear-unbiased-estimator guarantee from adding homoskedasticity and no autocorrelation, and exact finite-sample inference from normality on top of that -- explicitly mapping each assumption to the later article that covers its failure. Proves Gauss-Markov in full (the classic C=(X'X)^{-1}X'+D argument) and then, because a proof on paper is not the same as watching it happen, confirms it by Monte Carlo simulation: a genuinely unbiased alternative estimator built from only two of fifty real data points shows almost exactly the 4.68x variance penalty the theorem predicts. Builds the standard error, t-test, F-test, and R-squared machinery, then gives the confidence-interval-versus-prediction-interval distinction its own full treatment with real numbers -- a 9-foot-wide confidence band next to a 62-foot-wide prediction band from the identical fitted line, on the same cars dataset introduced in Part 1.
Linear Regression, Part 1: Origins and What "Linear" Actually Means
The first installment of a complete reference on linear regression -- where the model actually came from (it is not one story but two), and why a model can bend like a curve on the page and still be, in the only sense that matters to the mathematics, linear
The opening article of a seven-part series covering linear regression end to end. Untangles a history that is really two separate inventions eighty years apart: Legendre (1805) and Gauss (1809, claiming 1795) developed least squares for astronomy with no notion yet of regression, while Galton's 1886 study of height introduced the statistical concept and word, formalized mathematically by Pearson (1896) and extended to multiple predictors by Yule (1897, 1899). Lays out the model equation, the design matrix, and the fixed-versus-stochastic-X distinction that quietly determines what 'unbiased' means and previews exactly where Part 4's endogeneity problems come from. The centerpiece is the article's title claim itself: the precise, commonly conflated difference between linearity in the variables and linearity in the parameters, illustrated with two real, numerically verified datasets -- a quadratic and a log-log fit on the classic cars stopping-distance data (both linear in the parameters despite curving visibly), and the Puromycin enzyme-kinetics dataset, where the genuinely nonlinear Michaelis-Menten model is fit two ways to show that the classic Lineweaver-Burk linearizing shortcut is not a free simplification -- it measurably costs accuracy relative to a direct nonlinear fit on the same data. Closes with the transformation toolkit (log-log, log-lin, Box-Cox) that lets real curved relationships stay inside the linear-in-parameters framework the rest of the series builds on.
Elastic Net
The final installment of the Regularization trilogy: how blending the L1 and L2 penalties produces a provable grouping effect that neither Ridge nor Lasso can offer alone, why the region between a diamond and a ball is the geometric root of it, and what really happens when it meets the same correlated real predictors that broke Lasso
The third and final installment in the Regularization series, resolving the instability the Lasso article deliberately left open. Derives the elastic net's naive and glmnet-style alpha/lambda parameterizations, proves from scratch that for identical standardized predictors any nonzero ridge component forces a unique equal split via strict convexity (verified analytically and by direct multi-seed simulation against lasso's genuinely arbitrary splits), and situates the general correlated case relative to the Zou-Hastie (2005) grouping-effect theorem. Works through the geometry of the resulting constraint region -- a curved-edged shape between Ridge's ball and Lasso's diamond that keeps the diamond's axis-aligned vertices (preserving sparsity) while replacing its flat edges with curvature (eliminating the coefficient discontinuities that gave Lasso trouble) -- and derives the coordinate-descent update in closed form, showing it algebraically contains both earlier articles' per-coordinate formulas as limiting cases. Two real worked examples, both direct callbacks to the Lasso article: the same 0.995-correlated Longley predictors (GNP and Year) that swapped discontinuously under lasso now split smoothly and near-equally the moment any ridge component is added; and the diabetes dataset's more tangled s1/s2 correlation, showing the same effect in a messier, multi-way-correlated real setting. Reports an honest, load-bearing complication rather than a tidy success story: a full cross-validated grid search over both alpha and lambda selects pure lasso (alpha = 1) as literally MSE-optimal on both datasets, meaning the grouping effect has to be chosen deliberately for interpretability and stability -- cross-validation alone does not surface it. Closes the trilogy with a synthesis of all three articles' 'no free lunch' throughline and a practical decision checklist for choosing among Ridge, Lasso, and Elastic Net.
Lasso Regression
A complete, worked treatment of the L1 penalty -- why its diamond-shaped constraint region produces exact zeros where Ridge's ball never does, how it is actually solved without a closed form, and what really happens when it meets the same collinear real dataset that gave Ridge trouble
The second installment in the Regularization series, giving the L1 penalty the same full treatment Ridge received: the constrained-optimization and penalized-form derivations, why a diamond-shaped constraint region produces exact zeros (an ellipse tangent to a diamond's vertex, verified geometrically) where Ridge's ball never does, and the soft-thresholding operator derived in closed form for the orthogonal-design special case and cross-checked numerically against scikit-learn. Covers how the lasso is actually solved in general -- LARS's exact piecewise-linear path (Efron, Hastie, Johnstone and Tibshirani, 2004) and coordinate descent (Friedman, Hastie and Tibshirani's glmnet, 2010) -- plus the Zou-Hastie-Tibshirani (2007) degrees-of-freedom result and the lambda.min vs. lambda.1se cross-validation convention. Two real worked examples: the diabetes dataset from the original LARS paper, where cross-validated sparsity recovers a smaller, clinically sensible model at negligible cost; and the same standardized Longley dataset used in the Ridge article, where the lasso path shows a dramatic, dataset-driven swap between two 0.995-correlated predictors (GNP and Year) as lambda changes, and five-fold vs. leave-one-out cross-validation disagree about whether GNP belongs in the model at all -- real motivation for the Elastic Net, the series' final article.
Ridge Regression
A complete, worked treatment of the L2 penalty -- its closed form, its Bayesian reading, and the theorem guaranteeing it can beat ordinary least squares -- run start to finish on a real, famously ill-conditioned 1967 dataset, including the moment automatic cross-validation gets fooled by it
The dedicated follow-up to the Regularization article's brief preview of Ridge, covering everything that article deliberately left out. Derives the ridge estimator's constrained-optimization and Bayesian MAP interpretations in full (the latter giving lambda a precise meaning as a noise-to-prior-variance ratio), proves the Hoerl-Kennard (1970) existence theorem that some amount of shrinkage strictly improves MSE over OLS for every dataset with real noise, and works through the SVD view of differential shrinkage and effective degrees of freedom. The centerpiece is a fully worked, numerically verified example on Longley's 1967 macroeconomic dataset -- famous for near-perfect multicollinearity and a wrong-signed OLS coefficient on GNP -- computing the real ridge trace, generalized cross-validation (Golub-Heath-Wahba, 1979), and leave-one-out cross-validation from the actual data. The trace and the automatic criteria disagree: GCV's minimum lands inside the region the trace itself shows to be unstable, a genuine and explained finding, not a toy example engineered to make ridge look good. Closes with a practical decision checklist and an honest 'no free lunch' accounting of what ridge does and does not fix.
The Axiomatic Approach to Probability
A circle's random chord has three different answers, Hilbert demanded a fix in 1900, and Kolmogorov delivered one in 1933 -- what the probability triple actually buys you, and why almost nobody touches it directly
Traces probability theory from the 1654 Pascal-Fermat correspondence through three incompatible early-20th-century answers to 'what is probability' -- classical (Laplace), frequentist (von Mises), subjective (Ramsey/de Finetti) -- to the crisis that made interpretation alone insufficient: Bertrand's 1889 random-chord paradox, where three equally natural methods of picking a 'random' chord give three different, numerically verified answers (1/3, 1/2, 1/4) to the same question. Covers Hilbert's 1900 sixth problem demanding an axiomatic treatment of probability, and Kolmogorov's 1933 resolution -- building probability out of measure theory (Borel, Lebesgue) rather than settling the interpretation debate. Explains the probability triple (Omega, F, P) term by term, including why events are restricted to a sigma-algebra rather than the full power set (Vitali's 1905 non-measurable set), and derives the complement rule as a consequence of the axioms rather than an assumption. Directly addresses the common misconception that a random variable is something that behaves randomly -- it's a fixed measurable function of an outcome, with all the randomness living in which outcome gets realized -- and explains why applied work almost never touches the underlying triple, since a random variable's induced distribution (its CDF/PDF/PMF) does all the practical work once defined. Closes with a philosophical point made precise by the formalism itself: probability describes a state of not-yet-knowing, and conditioning on a realized outcome collapses it to a degenerate Dirac measure, which is why 'the probability it snowed' stops being a meaningful question the moment the day is over.
Stevens' Levels of Measurement
Everyone learns nominal, ordinal, interval, and ratio as if the taxonomy fell from the sky -- it came from one 1946 paper written to settle an argument about whether psychology could measure anything at all
Traces the nominal/ordinal/interval/ratio taxonomy to S.S. Stevens' 1946 Science paper, written to answer the British Ferguson Committee's 1940 report that psychological measurement wasn't measurement at all. Gives the precise mathematical content behind the four scales -- each is defined by the broadest class of relabeling transformations (arbitrary bijections, monotonic functions, affine, or similarity transformations) that preserve its meaningful structure, reproducing Stevens' own pairing of each scale with permissible statistics. Demonstrates the stakes with a numerically verified worked example: two ordinal-scale groups whose mean-based comparison completely reverses under a valid, order-preserving relabeling of the same categories, while the median-based comparison stays identical. Covers the pushback too -- Lord's 1953 football-numbers satire and Velleman and Wilkinson's 1993 critique -- and closes by mapping each scale to the specific tools in this site's own test library that are actually appropriate for it.
Cochran's Theorem
The 1934 result, quietly running behind every ANOVA table and every t-test, that tells you exactly when a sum of squares splits into independent chi-squares
Introduces William Cochran's 1934 theorem -- developed at Rothamsted alongside Yates and Fisher -- which most applied users have never heard of by name despite using its consequences every time they read an ANOVA table or run a t-test. Derives the theorem's geometric heart in two steps: an eigenvalue-counting argument showing that a sum-of-squares decomposition's ranks adding up to the total dimension forces each piece to be an orthogonal projection, and a rotation argument showing orthogonal projections of a standard Normal vector are automatically independent chi-squares. Works two examples end to end and verifies both numerically -- the classical independence of the sample mean and sample variance (including the little-known Geary-Lukacs fact that this independence is essentially unique to the Normal distribution), and the full three-piece decomposition behind the one-way ANOVA F-test cited without proof in this site's Analysis of Variance article. Surveys where the theorem underlies practice (t-tests, ANOVA, OLS regression inference, variance confidence intervals, degrees-of-freedom bookkeeping) and closes by relating it to the Wilson-Hilferty and Fisher square-root transformations as a natural, sequential next step rather than a competing idea.
The Analysis of Variance
A tool named after variance and built entirely from variance estimates -- yet the thing it actually tests is whether group means are equal
Traces ANOVA from Fisher's 1918 coinage of the word 'variance' through its development at Rothamsted and formalization in Statistical Methods for Research Workers (1925), including the Snedecor naming story behind the letter F. Derives the sum-of-squares identity SST = SSB + SSW and then the real mechanism -- E[MSW] = sigma^2 always, while E[MSB] = sigma^2 plus an extra term that is exactly zero under equal means -- showing precisely why a ratio of two variance estimates tests location, not spread. Covers the assumptions that make the exact F-test valid (independence, Normality, homogeneity of variance) and what to use instead when each breaks down, explains -- in broad terms -- how Levene's test repurposes the identical between/within machinery on absolute deviations to test variances instead of means, and closes with a fully worked, hand-computable three-treatment ANOVA table (verified numerically) so the reader can build one from raw data unaided.