Surprisal

Surprisal, in information theory and the Free Energy Principle, is the degree to which an observation deviates from what was expected — the negative log-probability of the observation under the agents current model. It is not surprise in the psychological sense (a startle response, an emotional reaction) but a formal, quantitative measure of informational mismatch: how much the observation tells the observer that their model is wrong. An event with low probability under the model has high surprisal; an event with high probability has low surprisal.

The concept is foundational to both Bayesian inference and thermodynamics. In Bayesian terms, surprisal is −log p(o|m) — the negative log-evidence of observation o given model m. In thermodynamic terms, surprisal is related to entropy: the expected surprisal of a distribution is its Shannon entropy. The two frameworks meet in the Free Energy Principle, where minimizing variational free energy is equivalent to minimizing expected surprisal — to keeping the organism from being too informed by its environment.

The Information-Theoretic Definition

For a discrete probability distribution p(x), the surprisal of outcome x is:

s(x) = −log p(x)

The base of the logarithm determines the units: base 2 gives bits, base e gives nats. The expected value of surprisal over the distribution is the entropy:

H(p) = E_p[s(x)] = −Σ p(x) log p(x)

Surprisal has several important properties:

• It is additive for independent events: the surprisal of two independent observations is the sum of their individual surprisals.
• It is non-negative: s(x) ≥ 0, with equality only when p(x) = 1 (complete certainty).
• It is unbounded: as p(x) → 0, s(x) → ∞. Impossible events, if observed, have infinite surprisal.

These properties make surprisal the natural measure of informational content. An observation that was completely unexpected carries more information — has higher surprisal — than an observation that was anticipated.

Surprisal in the Free Energy Principle

In the Free Energy Principle, surprisal is the quantity that biological systems are fundamentally organized to minimize. An organism that experiences high average surprisal is one whose model of the world is systematically wrong — it keeps encountering states that it did not predict and cannot explain. Sustained high surprisal is not merely suboptimal; it is fatal. An organism that cannot predict its environment cannot maintain its homeostasis, cannot find food, cannot avoid predators.

The FEP reformulates this in a tractable way. Direct minimization of surprisal would require evaluating p(o) — the marginal probability of observations — which requires integrating over all possible hidden states. This integral is intractable for any realistic system. Instead, the FEP introduces variational