Generalized Inverse

A generalized inverse of a matrix $\mathbf{A}$ (not necessarily square or invertible) is any matrix ${\mathbf{A}}^{-}$ satisfying the weaker condition $\mathbf{A}{\mathbf{A}}^{-}\mathbf{A}=\mathbf{A}$ (rather than the full inverse's $\mathbf{A}{\mathbf{A}}^{-1}={\mathbf{A}}^{-1}\mathbf{A}=\mathbf{I}$), and unlike the ordinary inverse, a generalized inverse always exists for any matrix, though in general it is not unique unless additional conditions (as in the Moore-Penrose pseudoinverse) are imposed. Generalized inverses are used throughout statistics whenever a design matrix is rank-deficient, most commonly in ANOVA models parameterized with a full set of dummy variables for every level of a categorical factor (over-parameterized, since one level's effect is redundant given the others and the intercept), where a generalized inverse of the singular ${\mathbf{X}}^{\top }\mathbf{X}$ still yields a valid (if not uniquely interpretable on its own) least squares solution.

• A generalized inverse always exists, regardless of whether the matrix is square, singular, or rectangular -- this universal existence is exactly what makes it useful as a fallback whenever the stricter true inverse fails to exist, without which a rank-deficient model's equations would simply have no solvable form at all.
• A generalized inverse is generally not unique, many different matrices can satisfy $\mathbf{A}{\mathbf{A}}^{-}\mathbf{A}=\mathbf{A}$ for the same $\mathbf{A}$ -- this non-uniqueness is exactly why an over-parameterized ANOVA model's individual dummy-variable coefficients can differ across software packages or coding schemes even when the model's overall fitted values and predictions agree.
• When a matrix does happen to be invertible, its generalized inverse and ordinary inverse coincide, ${\mathbf{A}}^{-}={\mathbf{A}}^{-1}$, so a generalized inverse can be understood as a strict extension of the ordinary inverse concept to a broader class of matrices, not a wholly separate one.