Bayesian Inference

Bayesian Inference as a Dynamical System

Bayesian inference is not merely a static updating rule. It is a dynamical system that operates in belief space — the space of all possible probability distributions over hypotheses. Each observation drives the system along a trajectory in this space, and the trajectory's properties determine whether the system converges to the true hypothesis, oscillates between alternatives, or diverges entirely.

In dynamical systems terms, the posterior distribution is the state of the system, and the likelihood function is the vector field that drives the system's evolution. The prior distribution sets the initial condition. The question of whether Bayesian inference converges to the truth is then a question about the stability properties of the dynamical system: does the trajectory converge to a fixed point, and is that fixed point the true hypothesis?

The answer depends on the structure of the hypothesis space. In finite, well-specified models, Bayesian inference is consistent: as the amount of data grows, the posterior concentrates on the true hypothesis. But in infinite or misspecified models, consistency is not guaranteed. The system may converge to a false hypothesis, or it may fail to converge at all. The dynamics of Bayesian inference in complex model spaces are an active area of research in statistics and machine learning.

Convergence and Phase Transitions in Belief Space

The most striking dynamical property of Bayesian inference is the phenomenon of phase transitions in belief space. When the evidence is weak or ambiguous, the posterior distribution remains diffuse — the system is in a high-entropy phase with no single hypothesis dominating. As evidence accumulates, the system may undergo a sharp transition to a low-entropy phase in which the posterior concentrates on a single hypothesis.

This phase transition is not merely quantitative. It is qualitative. Before the transition, the system is in a state of epistemic uncertainty in which multiple hypotheses are viable. After the transition, the system is in a state of epistemic confidence in which one hypothesis dominates. The transition is sharp, irreversible, and structurally similar to the phase transitions studied in statistical mechanics.

The implications for scientific reasoning are profound. The accumulation of evidence does not gradually increase confidence in the true hypothesis. It drives the system toward a critical threshold, beyond which the system undergoes a qualitative restructuring of belief. This is why scientific revolutions are abrupt: the evidence accumulates until the old paradigm becomes untenable, and then the system flips to the new paradigm. The flip is not a gradual adjustment. It is a phase transition.

Bayesian Inference and the Free Energy Principle

The Free Energy Principle formalizes Bayesian inference as a biological process. The brain is treated as a Bayesian inference engine that maintains a generative model of its sensory environment and updates that model to minimize prediction error. The variational free energy is the cost function that drives this updating, and the precision-weighting of prediction errors determines the rate of convergence.

This biological Bayesianism is not merely an analogy. The mathematical structure of variational inference is formally identical to the dynamics of biological self-organization. A system that minimizes variational free energy is, simultaneously, performing Bayesian inference and maintaining its organization against entropy. The two processes are the same computation in different descriptions.

The connection to Active Inference is direct: the brain does not merely update its beliefs to match the world. It selects actions that will bring the world into conformity with its beliefs. This is Bayesian inference extended into the domain of action: the system infers not only what is true but what should be done, and it does so within a single unifying framework.

The Limits of Bayesian Inference

Bayesian inference is powerful but not universal. Its limits include:

Computational intractability. Exact Bayesian inference over complex model spaces is often impossible, requiring approximations that may introduce systematic errors.
Prior sensitivity. The posterior depends on the prior, and in cases with limited data, the prior dominates. This is not a bug but a feature — priors encode structural knowledge — but it means that Bayesian inference cannot escape the need for substantive assumptions.
Misspecification. If the true hypothesis is not in the model space, Bayesian inference converges to the best available approximation, not to the truth. The system is a hill-climber in a landscape that may not contain the true peak.
The problem of induction. Bayesian inference does not solve Hume's problem of induction. It reformulates it. The prior distribution encodes inductive assumptions, and the question of whether those assumptions are justified remains open.

The deepest limit is that Bayesian inference is a framework for updating beliefs within a given model space. It does not address the question of how the model space itself is generated, revised, or transcended. That question — the question of abductive model generation — is outside the scope of Bayesian inference, and it is where the real action of scientific discovery occurs.

Bayesian inference is the rule for updating beliefs. It is not the rule for generating the beliefs that are worth updating.