Home Statistical Dictionary Linear Independence

Linear Independence

The property that no vector in a set can be written as a combination of the others -- the condition a design matrix's predictor columns must satisfy for a regression model's coefficients to be uniquely estimable.

Linear Algebra for StatisticsIntermediateFree

In Plain English

A set of vectors is linearly independent when none of them is 'hiding' redundant information already carried by the others, no vector in the set can be built by scaling and adding up the rest. In regression, this translates directly into a practical requirement: your predictor columns need to be linearly independent, otherwise the model literally cannot tell which of two (or more) redundant predictors deserves credit for an effect, and no unique set of coefficients exists.

Definition

A set of vectors ${v_{1}, \dots, v_{k}}$ is linearly independent if the only solution to $c_{1} v_{1} + c_{2} v_{2} + \dots + c_{k} v_{k} = 0$ is the trivial one, $c_{1} = c_{2} = \dots = c_{k} = 0$ ; if some nontrivial combination of coefficients also produces the zero vector, the set is linearly dependent, and at least one vector in the set can be written as a linear combination of the others. The rank of a matrix is defined as the maximum number of linearly independent rows or columns it contains, and a regression design matrix's columns must be linearly independent for the model's coefficients to be uniquely identifiable, linear dependence among predictors (perfect multicollinearity) is precisely what makes $X^{⊤} X$ singular and the ordinary least squares solution non-unique.

Formula

c_{1} v_{1} + \dots + c_{k} v_{k} = 0 ⟹ c_{1} = \dots = c_{k} = 0

Notation

c_{i}

—scalar coefficients in the linear combination

v_{i}

—the i-th vector in the set

Worked Example

The vectors $(1, 0)$ and $(0, 1)$ are linearly independent -- no scalar multiple of one can produce the other. But $(1, 2)$ , $(2, 4)$ , and $(0, 1)$ are linearly dependent, since $(2, 4) = 2 \times (1, 2) + 0 \times (0, 1)$ , a nontrivial combination of the first and third vectors produces the second, exactly the kind of redundancy that would make a design matrix built from these three columns rank-deficient.

Properties

A set of more vectors than the dimension of the space they live in is automatically linearly dependent -- for instance, any 3 vectors in ordinary 2-dimensional space must be linearly dependent, since 2-dimensional space only has room for 2 linearly independent directions.
Linear independence of a design matrix's predictor columns is a strict requirement for ordinary least squares to have a unique solution -- linear dependence (perfect multicollinearity) causes the normal-equations matrix $X^{⊤} X$ to become singular, at which point infinitely many coefficient vectors fit the data equally well, and no single 'the' estimate exists.
Linear independence is a stricter, purely algebraic condition than the informally similar-sounding statistical notion of 'low correlation' -- predictors can be linearly independent while still highly (though not perfectly) correlated, which is exactly the high-VIF, near-singular scenario diagnosed by the condition number and variance inflation factor rather than by a binary independence check.

At a Glance

Independent meansonly the trivial (all-zero) coefficient combination gives the zero vector

Dependent meanssome vector can be written as a combination of the others

Determinesthe rank of a matrix built from the vectors as rows/columns

In regression, requiresno predictor is an exact linear combination of the others

Last updated September 8, 2026← Back to the Dictionary