Reproducing kernel Hilbert space

A reproducing kernel Hilbert space (RKHS) is a function space equipped with an inner product in which point evaluation is a continuous linear functional — meaning the value of any function at any point can be computed by taking the inner product with a 'kernel function' centered at that point. Introduced by Nachman Aronszajn in 1950, the RKHS framework transforms function approximation into geometry: finding the right function becomes finding the right vector in a Hilbert space, and the kernel encodes the similarity structure of the domain. In machine learning, RKHS theory underpins kernel methods such as support vector machines and Gaussian processes, and provides the setting in which the neural tangent kernel operates. The norm in an RKHS measures function smoothness, which is why the minimum norm interpolant in an RKHS can generalize well: the norm penalty favors smooth functions, and smoothness is often correlated with generalization. The spectral decay of the kernel operator — how quickly its eigenvalues shrink — determines whether benign overfitting is possible in high dimensions.