Home Statistical Dictionary Non-Sampling Error
Non-Sampling Error
Any source of error in a survey estimate other than sampling error -- including coverage error, nonresponse bias, measurement error, and processing mistakes -- that does not necessarily shrink, and can even grow, as the sample size increases.
In Plain English
Non-sampling error is the catch-all for everything that can make a survey estimate wrong for reasons that have nothing to do with only surveying a subset of the population, a badly worded question that confuses respondents, people who refuse to answer skewing the results, a sampling frame that leaves out an entire group, or plain data-entry mistakes. The unsettling thing about non-sampling error, compared to sampling error, is that throwing more money at a bigger sample size doesn't fix it and can sometimes make it worse, a bigger but sloppier survey can easily be less accurate than a smaller, more carefully run one.
Definition
Non-sampling error comprises every source of inaccuracy in a survey estimate that is not attributable to observing a sample rather than a census, principally coverage error (the sampling frame misses or misrepresents part of the target population), nonresponse bias (respondents differ systematically from nonrespondents), measurement error (inaccurate responses or instruments), and processing error (mistakes in data entry, coding, or editing). Unlike sampling error, non-sampling error is generally not reducible by increasing sample size, and can worsen with a larger sample if the added scale strains data quality controls, interviewer training, or respondent follow-up; because non-sampling error is difficult or impossible to quantify with a single summary statistic the way sampling error can be captured by a standard error, well-designed surveys typically report qualitative discussion of likely non-sampling error sources (response rates, frame coverage limitations, question-wording pretests) alongside their formal margin of error.
Properties
- Non-sampling error is often the dominant source of total survey error in practice, particularly for surveys with low response rates or poorly tested questionnaires, this is a central finding of the total survey error framework in survey methodology, which argues sampling error alone is an incomplete and sometimes misleading measure of a survey's overall accuracy.
- Because non-sampling error does not shrink with sample size, and can even worsen with it (larger surveys are harder to field with uniformly high quality), a well-run survey with a modest sample size can produce more accurate estimates than a poorly run survey with a much larger sample, a point frequently misunderstood in casual interpretation of poll results.
- Total survey error is the term for the combination of sampling error and all sources of non-sampling error together, survey methodologists increasingly emphasize managing and reporting total survey error rather than treating the formal margin of error, which reflects sampling error alone, as a complete measure of a survey's trustworthiness.
At a Glance