Column Space

The column space of an $m\times n$ matrix $\mathbf{A}$, denoted $\text{Col}(\mathbf{A})$ or $\mathcal{C}(\mathbf{A})$, is the span of its $n$ columns, the subspace of ${\mathbb{R}}^m$ consisting of every vector $\mathbf{Ax}$ for some $\mathbf{x}\in {\mathbb{R}}^n$. Its dimension equals the rank of $\mathbf{A}$. For a regression design matrix $\mathbf{X}$, the column space is exactly the set of every possible fitted-value vector $\mathbf{X}\mathbf{\beta }$ for any coefficient vector $\mathbf{\beta }$, and the hat matrix $\mathbf{H}=\mathbf{X}({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }$ orthogonally projects the observed response vector $\mathbf{y}$ onto this column space, producing the ordinary least squares fitted values $\hat{\mathbf{y}}$ as the point in the column space closest to $\mathbf{y}$.

• The column space of a regression design matrix defines the ceiling on what the model can fit -- no matter how the coefficients are chosen, the fitted values must lie in this fixed subspace, which is why adding a genuinely new (linearly independent) predictor is the only way to enlarge the set of patterns a linear model is capable of representing.
• The dimension of a matrix's column space always equals its rank, this identity connects the abstract subspace concept directly to the concrete, computable rank of the matrix, and is why a rank-deficient design matrix has a column space of lower dimension than its number of predictor columns would suggest.
• Ordinary least squares regression can be described entirely in column-space terms: it finds the orthogonal projection of the observed response vector onto the column space of the predictors, and this projection is precisely what the hat matrix computes -- fitted values are literally the point in the column space nearest to the data.