Singular value decomposition
The singular value decomposition (SVD) is a fundamental matrix factorization that decomposes any \times n$ matrix $ into three factors: = U\Sigma V^T$, where $ is an \times m$ orthogonal matrix, $ is an \times n$ orthogonal matrix, and $\Sigma$ is an \times n$ diagonal matrix with non-negative real entries $\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_{\min(m,n)}$ on the diagonal. These diagonal entries are the singular values of $, and they encode the matrix's fundamental geometric action: stretching or compressing along orthogonal axes. The SVD exists for every matrix — rectangular, square, singular, or nonsingular — making it more general than the eigenvalue decomposition, which applies only to diagonalizable square matrices.
The geometric interpretation is direct and powerful. Any linear transformation can be understood as a rotation (^T$), followed by a scaling along coordinate axes ($\Sigma$), followed by another rotation ($). The singular values tell you how much the transformation stretches or shrinks space along its principal directions. A zero singular value indicates that the transformation collapses some dimension — it maps a nonzero vector to zero, revealing the null space. The number of nonzero singular values is the rank of the matrix, a more numerically stable measure than counting linearly independent rows or columns.
Applications in Data Analysis and Signal Processing
The SVD is the computational engine behind principal component analysis (PCA). In PCA, the data matrix is centered and decomposed via SVD; the right singular vectors ($) are the principal components, and the squared singular values divided by -1$ are the variances explained by each component. The truncation of small singular values — keeping only the $ largest — yields the optimal rank-$ approximation of the data matrix in the least-squares sense. This is the mathematical foundation of dimensionality reduction, collaborative filtering, and latent semantic analysis.
In signal processing, the SVD separates signal from noise when the signal lives in a low-dimensional subspace and the noise is spread across all dimensions. By truncating the decomposition at the rank of the signal subspace, the SVD reconstructs the signal while suppressing noise components. This appears in array signal processing, harmonic retrieval, and subspace tracking algorithms.
Applications in Control and Systems Theory
In control theory, particularly for multi-input multi-output systems, the SVD provides the natural generalization of gain from scalar to matrix transfer functions. The singular values of a matrix transfer function (j\omega)$ at frequency $\omega$ describe how the system amplifies or attenuates input directions at that frequency. The maximum singular value $\bar{\sigma}(G(j\omega))$ is the peak gain; the minimum singular value $\underline{\sigma}(G(j\omega))$ measures how close the system is to losing rank at that frequency, which corresponds to a loss of controllability or observability.
The SVD is central to robust control analysis. The structured singular value $\mu$ and the \infty$ norm of a transfer function matrix are both computed and bounded using singular value decompositions. The condition number of a matrix — the ratio of largest to smallest singular value — determines the sensitivity of solutions to linear systems to perturbations in the data. A large condition number means that small errors in the input can produce large errors in the output, a warning that the model or computation is ill-conditioned.
Computation and Numerical Properties
The SVD is computationally more expensive than QR decomposition or Gaussian elimination — (mn^2 + n^3)$ operations for an \times n$ matrix — but its numerical stability is exceptional. The singular values are computed with backward stability: the computed singular values are the exact singular values of a slightly perturbed matrix. This makes the SVD the method of choice when accuracy matters more than speed, and it is the benchmark against which other matrix factorizations are judged.
The truncated SVD — keeping only the top $ singular values and vectors — solves the Eckart-Young-Mirsky theorem: it is the optimal low-rank approximation in both the spectral norm and the Frobenius norm. This optimality property makes the SVD indispensable in compression, denoising, and regularization. It also underlies the pseudoinverse ^+ = V\Sigma^+ U^T$, which generalizes matrix inversion to rank-deficient and rectangular matrices and provides the minimum-norm least-squares solution to inconsistent linear systems.
The singular value decomposition is linear algebra's confession that matrices are not merely arrays of numbers but geometric operators with intrinsic shape. The SVD strips away coordinate-dependent noise and reveals the essential action: rotate, scale, rotate. That this factorization exists for every matrix — every one, without exception — suggests a deep structural fact about linear transformations that coordinate-based methods conceal. The eigenvalue decomposition asks what a matrix does to special vectors. The SVD asks what the matrix does to space itself. The second question is the deeper one, and it is no accident that the SVD has become the workhorse of modern data science, control theory, and numerical analysis. Any field that still treats matrices as spreadsheets has not yet learned to see.