Bootstrap

The bootstrap (Efron, 1979) approximates the sampling distribution of a statistic $\hat{\theta }$ by repeatedly drawing bootstrap samples $X^{\ast (b)}$, each of size $n$, with replacement from the original observed data, recomputing the statistic on each resample, ${\hat{\theta }}^{\ast (b)}$, and using the empirical distribution of these $B$ bootstrap replicates, $\{{\hat{\theta }}^{\ast (1)},\dots ,{\hat{\theta }}^{\ast (B)}\}$, to approximate the true sampling distribution of $\hat{\theta }$. Because it substitutes computation for analytical derivation, the bootstrap can estimate standard errors, biases, and confidence intervals for complex statistics with no simple closed-form variance formula, and it is asymptotically valid under broad conditions, though it can fail for certain statistics (e.g. extreme order statistics) or in certain small-sample or heavy-tailed settings.

• The bootstrap's core assumption is that the observed sample is a reasonable stand-in for the true population, resampling from a small or highly unrepresentative sample will faithfully replicate that sample's own quirks and biases, not correct for them, so the bootstrap doesn't rescue a fundamentally inadequate original dataset.
• The number of bootstrap replicates B involves a computation-versus-precision tradeoff, a few hundred replicates is often adequate for a standard error, while several thousand are typically recommended for percentile-based confidence intervals, which are more sensitive to the tails of the bootstrap distribution.
• The bootstrap is known to perform poorly or require specialized variants for certain statistics and situations, including estimating extreme values (e.g. a sample maximum), dependent data like time series (requiring a block bootstrap), and settings with very small sample sizes, where the discreteness of possible resamples becomes a real limitation.