Bayesian Update

A Bayesian update is the process of revising a probability estimate in light of new evidence, according to the rules of Bayes' theorem. Given a prior probability P(H) for a hypothesis H and a likelihood P(E|H) for observing evidence E under that hypothesis, the Bayesian update computes the posterior probability P(H|E) — the revised belief in H after observing E. In its simplest form, the update is a multiplicative reweighting: each hypothesis is scaled by how well it predicted the evidence, and the results are normalized so that probabilities sum to one.

The Bayesian update is the atomic operation of Bayesian reasoning. Every inference, every learning episode, every revision of belief can be decomposed into a sequence of Bayesian updates. In machine learning, Bayesian updates appear as parameter updates in Bayesian neural networks, as posterior approximations in variational inference, and as belief updates in probabilistic graphical models. In cognitive science, the Bayesian update is proposed as the fundamental computation performed by the brain during perception and learning.

But the Bayesian update has a critical limitation: it assumes that the evidence is drawn from a model that is already in the hypothesis space. If the true generating process is not among the hypotheses under consideration, no amount of Bayesian updating will discover it. The update can only redistribute probability among existing hypotheses; it cannot create new ones. This is the mathematical expression of a profound epistemological constraint: Bayesian reasoning optimizes within a model, but it does not select between models. Model selection — the problem of model comparison and paradigm change — requires a different set of tools, and the Bayesian update, by itself, is not among them.