Theory • Practice • Insight
Statistics, Explained
Clearly
Read approachable guides, deep dives, and real-world case studies that connect statistical theory to the decisions you make every day.
Where These Ideas Come From
New articles go up every Tuesday and Friday. Most of the ideas come out of the research behind The Compendium of Statistical Tests — while working through the literature for that book, I kept running into concepts, historical facts, and stories that deserved more room than a footnote, and this is where they end up. The rest come from years of teaching and practicing statistics directly. Each one is meant to stand on its own: a small, self-contained piece you can read in one sitting.
These are well-researched, and you're welcome to cite them freely — just credit StatisticsCorner (see the “How to Cite This Article” section on each page). If you spot a mistake, I'd like to know: reach out through the contact form.
Showing 1–12 of 14 articles
The Anderson–Darling Test
From a weighted integral to the two familiar discrete formulas
A step-by-step derivation of the Anderson–Darling goodness-of-fit statistic, starting from the defining weighted integral and ending at the two discrete formulas used in software and textbooks. Along the way it shows exactly where the odd-number weights, the tail-sensitivity, and the mysterious −n term actually come from — and proves the two commonly cited discrete forms are algebraically identical, not different statistics.
Likelihood: How One Idea Reshaped Statistics
From tally tables to modern estimation -- and the much older twin the idea turns out to share with Bayesian inference
Traces statistics from pure data organization (tally tables, means, variances) through curve fitting -- correcting the record to show Gauss and Laplace attached a Normal-error probabilistic model to least squares within seven years of Legendre's model-free 1805 publication, not a century later -- through Pearson's 1894 method of moments, to Fisher's 1921-22 formalization of likelihood. Corrects a common historical inversion: inverse probability (Bayes 1763, Laplace, De Morgan's 1837 naming) predates Fisher's likelihood by a century and a half, and Fisher built likelihood in explicit opposition to it rather than statisticians later discovering Bayesian methods share the same machinery. Closes with likelihood's 20th-century descendants -- quasi-likelihood, partial likelihood, REML, and empirical likelihood -- and where each is used today.
Time Series and Stationarity
A single history is both all the data you will ever get and never enough, on its own, to define a distribution
Works out what makes time series data structurally different from cross-sectional data: a stochastic process has an ensemble dimension (fixed time, varying realization, where distributions live) and a temporal dimension (fixed realization, varying time, where the data live), and a single history offers no replication at any one time point. Covers autocorrelation as the structure that substitutes for that missing replication, why stationarity is required before prediction is possible at all, strict vs. weak (covariance) stationarity and why practice targets the latter, ergodicity as a separate assumption from stationarity (with a worked example of a stationary but non-ergodic process, verified by simulation), and a taxonomy of how real series actually become non-stationary -- trends, level shifts, regime shifts, variance changes, and unit roots -- each with a different diagnostic signature. Closes by explaining why no single test can certify stationarity, previewing a dedicated future article on the tests themselves.
An Introduction to Regularization
Why ordinary regression can break in two genuinely different ways -- and how a single penalty term fixes both
Motivates regularization from a precise failure of ordinary least squares: correlated predictors inflate coefficient variance for purely statistical reasons that hold even under exact arithmetic, while extreme ill-conditioning separately threatens the floating-point computation itself, at a much higher threshold, verified here by direct simulation. Also covers the sharper p >= n case (true non-identifiability, not just instability), introduces the general penalized-estimation framework and the bias-variance tradeoff behind it, and previews the three specific penalties -- Ridge, Lasso, Elastic Net -- each covered in its own dedicated article next.
The Kaplan-Meier Estimator
Building a survival curve one risk set at a time, and why the bookkeeping is the whole idea
An introduction to survival analysis -- what makes time-to-event data different from ordinary regression or time series, and a taxonomy of nonparametric, semiparametric, and parametric survival models -- followed by a from-scratch derivation of the Kaplan-Meier estimator. Builds the product-limit formula from risk-set bookkeeping, works a real 21-patient leukemia remission dataset, derives Greenwood's variance formula and the log-log confidence interval, and shows numerically why two common shortcuts for handling censored data produce badly biased survival curves.
The Wilson-Hilferty Transformation
A 95-year-old cube root that turns a lopsided chi-square into something almost perfectly Normal
Wilson and Hilferty's 1931 cube-root normalizing transformation turns a skewed chi-square variable into something very close to standard Normal -- a refinement of Fisher's earlier square-root approximation. This article derives the transformation from a power-family expansion, uses exact symbolic algebra to prove that the cube root is the unique power that cancels the leading-order skewness term, verifies its accuracy against the exact chi-square distribution and against Fisher's approximation, and extends the same argument to the general Gamma distribution.
The Classical Trinity of Tests
Wald, likelihood ratio, and score — three different measurements on the same log-likelihood curve
Derives Wald's test, the likelihood ratio test, and the score test from a single picture: the log-likelihood curve's local quadratic (inverted-parabola) approximation around the MLE, with an annotated figure showing Wald as a horizontal distance, score as a tangent slope, and LRT as a vertical drop, all on one curve. A worked binomial example (n=20, x=3, testing p=0.5) shows real finite-sample disagreement, then two verified failure modes for Wald specifically: it is not invariant to reparameterization (testing on the log-odds scale changes the statistic from 19.2 to 7.7 on identical data, while score and LRT don't move), and it diverges to infinity at a data boundary that score and LRT read as weak evidence. A 300,000-rep simulation shows Wald over-rejecting a true null 43.5% of the time at n=8 versus a nominal 5%, and a local-alternative convergence table confirms the promised asymptotic equivalence actually emerging as n grows. Closes with practical guidance on which test to reach for and when, plus a brief multivariate generalization.
The Delta Method
How to get the variance of a function of an estimator, when the function itself has no simple variance rule
Opens with a precise primer on parameters, estimators, and the expected value and variance of an estimator, then derives the Delta Method from a first-order Taylor expansion: why Var(g(theta-hat)) is approximately [g'(theta)]^2 times Var(theta-hat), stated formally as an asymptotic-normality theorem, plus the brief multivariate/gradient version. Worked example transforms a binomial proportion into a log-odds -- the exact quantity behind every logistic-regression Wald SE -- and a 200,000-rep simulation shows skipping the derivative understates the SE by nearly 10x, while a naive confidence interval built directly on the proportion has both the wrong coverage and, a quarter of the time, an impossible negative endpoint. Closes with an honest look at how approximation quality depends on sample size, plus when a purpose-built alternative (e.g. Wilson score intervals) beats the general-purpose Delta Method.
Truncated, Not Censored
Why a truncated unit was never given the chance to be in the sample — and how the likelihood must condition on that
Closes out the incomplete-data trilogy by deriving why truncation is a fundamentally different (and more severe) problem than missing data or censoring: a truncated unit is never in the sample at all. Derives the truncated likelihood f(t)/S(tau), a full worked MLE for a left-truncated Exponential (an insurance-deductible example, using the same memorylessness property from the Exponential and Geometric sheets), and the combined truncation-plus-censoring likelihood for studies with both delayed entry and follow-up censoring. A simulation shows a naive fit that ignores truncation overstates the true mean by 40%.
Censored, Not Missing
How right, left, and interval censoring change the likelihood — and why treating them as missing data breaks it
Derives why a censored observation is more informative than a missing one, and builds the censored likelihood that reflects it -- a full derivation of the Exponential MLE under right-censoring (the 'total time on test' estimator), the product-limit logic behind Kaplan-Meier, and the distinction between non-informative censoring and its MNAR-like informative counterpart. A Monte Carlo simulation shows both obvious shortcuts -- dropping censored units, or treating the censoring time as the true event time -- inflate the rate estimate by roughly 50%, while the proper censored likelihood is essentially exact.
Missing Data Mechanisms
Why MCAR, MAR, and MNAR call for completely different remedies
Derives Rubin's MCAR/MAR/MNAR framework rather than just naming it: why complete-case analysis is exactly unbiased under MCAR, why it breaks under MAR but conditioning on the right covariates fixes it completely (a full derivation of the ignorability condition), and why MNAR admits no fix from the observed data alone. Covers the real remedies each mechanism licenses -- multiple imputation with Rubin's combining rules, maximum likelihood, and inverse probability weighting -- and ends with a simulation where covariate adjustment eliminates bias under MAR but only cuts it in half under MNAR.
Randomization, Replication, Blocking
The three principles of experimental design, derived rather than declared
Derives what each of Fisher's three design principles actually buys an estimator, in the language of bias and variance rather than just definitions: randomization's exact unbiasedness proof under Neyman's finite-population model, why replication is what makes an error variance estimable at all (and why the usual variance estimator is conservative, not exact), and blocking's relative-efficiency payoff from moving nuisance variation out of the error term. Ties all three together in a single Monte Carlo simulation comparing a confounded design, a completely randomized design, and a randomized block design on the same field.