Gaussian process

A Gaussian process is a nonparametric Bayesian model that defines a probability distribution over functions, rather than over parameters. In a Gaussian process, any finite collection of function values has a joint Gaussian distribution, specified entirely by a mean function and a covariance function — or kernel — that encodes how similar any two points are expected to be. The kernel is the inductive bias: it determines what kinds of functions the model prefers, and therefore what patterns it can learn from data.

Gaussian processes are the dual of parametric models like neural networks. Where a neural network commits to a fixed architecture and learns weights, a Gaussian process commits to a kernel and integrates over all functions consistent with that kernel. In the limit of infinite width, certain neural networks converge to Gaussian processes — a connection that has become central to the theoretical study of deep learning through the framework of neural tangent kernels.

The Gaussian process is the Bayesian answer to a question that parametric methods never ask explicitly: not "what is the best function?" but "what is the distribution over plausible functions?" This shift from optimization to integration changes the nature of prediction: a Gaussian process does not merely predict a value; it predicts a full uncertainty distribution, making it uniquely suited to domains where the cost of error is high and the value of knowing what the model does not know is higher.