Approximate inference

Approximate inference is the art of answering questions you cannot solve exactly. In probabilistic modeling, the central computational problem is the posterior: given a model and some data, what do we believe about the latent variables? For all but the simplest models, this posterior is intractable — a high-dimensional integral with no closed form. Approximate inference is the collection of methods that replace exact computation with controlled approximation, trading precision for tractability in ways that preserve what matters.

The need for approximation is not a failure of mathematics. It is a fact of physics. Even if the posterior had a closed form, computing it might require more operations than the universe has atoms. Exact inference is not merely difficult; in many cases, it is physically impossible. Approximate inference is therefore not a compromise but a necessity — the only way to reason probabilistically about complex systems.

Variational Methods: Optimization as Inference

The dominant family of approximate inference methods is variational inference, which reframes the inference problem as an optimization problem. Instead of computing the true posterior p(z|x), variational methods seek the closest member of a simpler family q(z; θ) by minimizing a divergence measure — typically the Kullback-Leibler divergence. The optimization is over the parameters θ of the approximate distribution, and the result is the best 'stand-in' the family can provide.

This reframing is powerful because it replaces integration with differentiation, and differentiation is what modern computing does well. Stochastic variational inference extends this to massive datasets by subsampling, making variational methods scalable to problems with billions of observations. Amortized inference goes further, training a neural network to map observations directly to approximate posterior parameters — the inference itself becomes a learned function.

But the variational approach has a cost: the approximation error is often invisible. The KL divergence is a lower bound on the log-evidence, and the gap between bound and truth is unmeasured. A variational posterior can be confidently wrong — not just inaccurate, but structurally wrong, missing modes or correlations that the approximating family cannot express.

Sampling Methods: Randomness as Computation

The alternative to optimization is sampling. Markov Chain Monte Carlo (MCMC) methods construct a random walk through the parameter space whose stationary distribution is the target posterior. The samples are correlated, but with enough steps, they become representative. Importance Sampling draws from a simpler distribution and reweights the samples to correct for the mismatch.

For sequential problems — tracking a moving target, filtering a time series — particle filters maintain a cloud of hypotheses (particles) that evolve with the data. Each particle is a guess at the hidden state, and the ensemble approximates the posterior through diversity and resampling.

Sequential Monte Carlo generalizes particle filtering to arbitrary distributions and arbitrary proposal mechanisms. It is the sampling analogue of variational inference: where variational methods optimize over a family of distributions, sequential Monte Carlo explores the space by propagating and reweighting samples. Both approaches face the same fundamental challenge: the curse of dimensionality. In high-dimensional spaces, neither optimization nor sampling is guaranteed to find the regions of high posterior mass.

Approximate Inference as a Theory of Mind

The deepest significance of approximate inference is not computational but epistemological. It provides a framework for understanding how bounded agents — brains, organizations, algorithms — reason under constraint.

In active inference, the brain is modeled as an approximate inference engine. Sensory processing is variational inference: the brain maintains a generative model of the world and updates it to minimize prediction error. Action is inference too: by moving, the brain samples the world to resolve uncertainty. The expected free energy that active inference agents minimize is computed approximately, and the quality of behavior is the quality of the approximation.

This view reframes cognition. A brain that uses approximate inference is not a flawed computer. It is a system operating at the physical limits of computation, making the best estimates it can given its architecture, its energy budget, and its prior knowledge. The heuristics and biases studied by Kahneman and Tversky are not cognitive bugs; they are the signatures of approximate inference under resource constraints.

The same framework applies to social systems. Organizations do not compute optimal strategies; they use rules of thumb, institutional heuristics, and distributed decision-making that approximate what a fully rational agent would do. Markets do not clear at the true equilibrium; they discover approximate equilibria through decentralized search. In each case, the system is doing approximate inference — and the errors are not random noise but systematic consequences of the approximation method.

Approximate inference is the defining computational problem of the 21st century. As models grow larger, data grows noisier, and systems grow more complex, the gap between what we want to know and what we can compute widens. The question is no longer whether to approximate, but how to approximate well — how to bound the error, how to know what we are missing, and how to design systems whose approximate inferences are robust enough to survive in a world that refuses to be simple.