Jensen's Inequality

Jensen's inequality states that for a convex function f, the function of the expectation is less than or equal to the expectation of the function: f(E[X]) ≤ E[f(X)]. For a concave function, the inequality reverses. It is the foundational theorem behind the variational bound in Bayesian inference, the convexity of thermodynamic free energy, and the Kullback-Leibler divergence's non-negativity. Without Jensen's inequality, there is no evidence lower bound, no variational free energy, and no Free Energy Principle as a tractable computational framework. The inequality is not merely a mathematical curiosity; it is the structural reason why approximate inference is possible at all.

A convex function is one that lies below its chords — geometrically, the set of points above its graph forms a convex set. This geometric property, when applied to probability distributions via Jensen's inequality, generates the fundamental bounds that make variational methods work.