Positive Semi-Definite Matrix

A symmetric matrix $\mathbf{A}$ is positive semi-definite if ${\mathbf{x}}^{\top }\mathbf{Ax}\ge 0$ for every vector $\mathbf{x}$ (zero permitted, unlike the strict inequality required for positive definiteness), equivalently, all of its eigenvalues are non-negative (zero permitted). Every theoretical covariance matrix is positive semi-definite by construction, since ${\mathbf{x}}^{\top }\mathbf{\Sigma x}=\text{Var}({\mathbf{x}}^{\top }\mathbf{Y})$ for any random vector $\mathbf{Y}$ with covariance $\mathbf{\Sigma }$, and a variance can never be negative, so this property requires no separate proof or assumption, it follows automatically from what a covariance matrix represents, whereas strict positive definiteness (a stronger, additional condition) can fail whenever exact multicollinearity or a deterministic linear relationship exists among the variables.

• Every theoretical covariance matrix is positive semi-definite by definition, following directly from the fact that a variance can never be negative -- no separate check or assumption is needed to establish this, it is a mathematical necessity, not an empirical property that could fail.
• Positive semi-definiteness allows for zero eigenvalues, which corresponds to a degenerate direction in the data, a linear combination of variables with exactly zero variance -- this happens in practice whenever one variable is an exact linear function of the others, such as a perfectly collinear predictor set or a covariance matrix estimated from fewer observations than variables.
• The distinction between positive semi-definite and strictly positive definite matters directly for invertibility, a positive semi-definite matrix with at least one zero eigenvalue is singular and cannot be inverted, which is exactly the scenario that forces the use of a generalized inverse (such as the Moore-Penrose pseudoinverse) in place of the ordinary matrix inverse.