Sampling Distribution

The sampling distribution of a statistic $\hat{\theta }$ (a sample mean, proportion, variance, or any other function of sample data) is the probability distribution of the values $\hat{\theta }$ would take across all possible samples of a fixed size $n$ drawn from the population under a specified sampling design, distinct from both the population distribution (of individual values) and any single observed sample's data. The standard error is the standard deviation of the sampling distribution, and quantifies how much a statistic computed from one particular sample is expected to vary from sample to sample; for the sample mean under simple random sampling, the sampling distribution's variance is ${\sigma }^2/n$ (or $(1-n/N){\sigma }^2/n$ correcting for a finite population without replacement), and the Central Limit Theorem establishes that, as $n$ grows large, the sampling distribution of the mean approaches a normal distribution regardless of the shape of the underlying population distribution, the theoretical result that underlies the validity of standard confidence intervals and hypothesis tests even when the underlying data are not themselves normally distributed. For statistics or designs where the sampling distribution has no simple closed form, particularly complex survey designs or nonstandard statistics, resampling-based approximations (the bootstrap, or replicate-weight methods like the jackknife and BRR in survey contexts) are used to approximate the sampling distribution empirically instead.

• The sampling distribution is a purely theoretical construct, in practice a researcher observes only one sample and computes one value of a statistic, never the full sampling distribution directly, all of classical inferential statistics (standard errors, confidence intervals, p-values) works by using theory (like the Central Limit Theorem) or approximation (like the bootstrap) to characterize this unobserved distribution from the single observed sample.
• The Central Limit Theorem's guarantee that a sample mean's sampling distribution approaches normality as sample size grows, regardless of the underlying population's own distribution shape, is precisely why so many standard statistical procedures remain approximately valid even when applied to data that are not themselves normally distributed, provided the sample size is reasonably large.
• For statistics with no simple, closed-form sampling distribution formula, medians, complex survey estimators, machine learning model outputs, resampling methods approximate the sampling distribution empirically by repeatedly recomputing the statistic on many resampled or perturbed versions of the data (the bootstrap, or in survey contexts, jackknife and BRR replicate weights), sidestepping the need for an explicit theoretical derivation entirely.