Singular Value Decomposition

Singular value decomposition (SVD) is the factorization of a matrix into the product of three matrices — an orthogonal matrix, a diagonal matrix of singular values, and another orthogonal matrix — that reveals the intrinsic geometric structure of any linear transformation. Introduced in its modern form by Erhard Schmidt and Hermann Weyl, SVD is the Swiss Army knife of numerical linear algebra: it provides the rank, the range, the null space, the condition number, and the pseudoinverse of a matrix, all from a single decomposition. In machine learning, SVD powers principal component analysis, latent semantic indexing, and low-rank matrix approximation. The singular values themselves encode the 'energy' of the data along different directions: the rapid decay of singular values in many real-world datasets is what makes benign overfitting possible — the data lives effectively in a low-dimensional subspace, and the minimum-norm solution ignores the high-singular-value directions that correspond to noise.