Large Deviation Theory

Large deviation theory is a branch of probability that studies the asymptotic behavior of sequences of probability distributions, specifically the decay rate of probabilities of rare events. Where the central limit theorem describes typical fluctuations of order 1/√N, large deviation theory describes exponentially rare fluctuations of order 1. The theory provides the mathematical framework for understanding importance sampling, statistical mechanics (where it connects to entropy and free energy), and the fluctuations of random walks and stochastic processes far from their mean behavior.

The cornerstone result is Cramér's theorem: for independent random variables, the probability of observing an average far from the mean decays exponentially with the number of samples, at a rate given by the rate function — the Legendre-Fenchel transform of the cumulant generating function. This rate function encodes the cost of rare events and determines the optimal bias distribution for importance sampling.

Large deviation theory is the mathematics of the improbable made precise. It tells us not just that rare events are rare, but exactly how rare they are — and in doing so, it transforms the impossible into the merely expensive.