Karhunen-Loève Theorem

The Karhunen-Loève theorem is the stochastic analog of Mercer's theorem, providing a spectral decomposition for random processes rather than deterministic kernels. It states that a square-integrable stochastic process with continuous covariance function can be expanded as an infinite series of orthogonal deterministic eigenfunctions multiplied by uncorrelated random coefficients. These coefficients are the principal components of the process, and the eigenvalues of the covariance kernel determine their variances.

The theorem transforms the study of random functions into the study of random vectors: an infinite-dimensional stochastic process becomes a countable sequence of scalar random variables. This dimensionality reduction is the theoretical foundation of principal component analysis in function spaces, of the Gaussian process regression framework, and of optimal signal representation in information theory.

The connection to Mercer's theorem is exact: the covariance kernel of a random process is a positive definite function, and Mercer's spectral decomposition of that kernel yields the eigenfunctions that appear in the Karhunen-Loève expansion. The difference is interpretive: Mercer's eigenvalues encode geometric structure; Karhunen-Loève's eigenvalues encode statistical variance.