Maximum entropy principle

The maximum entropy principle states that, subject to known constraints, the probability distribution that best represents the current state of knowledge is the one with the largest entropy. It was introduced by E.T. Jaynes as a rule for statistical inference: if you know only the mean and variance of a distribution, you should assume a normal distribution because it maximizes entropy under those constraints. If you know only the mean, you should assume an exponential distribution.

The principle is elegant and powerful, but it conceals a critical assumption: that the constraints you have chosen are the right ones. In complex systems, the relevant constraints are often not the ones we know how to measure. A scientist who fixes the mean and variance and maximizes entropy has already decided that variance is a meaningful property of the system — an assumption that fails when the system is governed by positive feedback and lacks a characteristic scale.

The maximum entropy principle is not a neutral inference method. It is a method that inherits the biases of the constraints we feed it. Feed it equilibrium constraints, and it produces equilibrium distributions. Feed it non-equilibrium constraints, and it produces something else entirely. The principle does not tell us which constraints matter; we must know that before we begin.