Span (Linear Algebra)

The span of a set of vectors $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ is the set of all possible linear combinations of them, $\text{span}\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}=\{c_1{\mathbf{v}}_1+\cdots +c_k{\mathbf{v}}_k:c_1,\dots ,c_k\in \mathbb{R}\}$, itself always a vector space (a subspace of whatever larger space the vectors live in). If the vectors are linearly independent, the span has dimension exactly $k$; if not, the span still exists but has a smaller dimension than $k$, since some vectors are redundant. In regression, the column space of a design matrix $\mathbf{X}$ is, by definition, the span of its predictor columns -- the set of every fitted-value vector the model could possibly produce for some choice of coefficients -- and ordinary least squares finds the specific point in that span closest, in Euclidean distance, to the observed response vector.

• The span of a set of vectors depends only on the directions those vectors point in, not on how many redundant vectors are included -- adding a vector that is already a combination of the others to the set never enlarges the span, which is exactly the linear-algebra description of a redundant predictor adding no new modeling capability.
• A basis is precisely a minimal spanning set, one that spans a space using the fewest possible vectors (achieved when the spanning vectors are also linearly independent) -- span describes the reachable territory, while a basis describes the most efficient way to describe that territory.
• The regression column space (the span of a design matrix's predictor columns) is the geometric object the hat matrix projects onto -- understanding column space as a span makes concrete why adding a new, genuinely independent predictor always enlarges the set of fitted values a model can produce, while adding a redundant one does not.