Sufficient statistics

A sufficient statistic is a function of data that captures all the information the data contains about a parameter of interest. Formally, given a statistical model P(X|θ) and data X, a statistic T(X) is sufficient for θ if the conditional distribution P(X|T(X),θ) does not depend on θ. Equivalently, T(X) is sufficient if the mutual information I(X;θ) equals I(T(X);θ) — all information about θ in the full data is already present in the statistic.

The concept is not merely a mathematical convenience. It is a statement about what matters in a system. A sufficient statistic identifies the minimal representation of a system that preserves all relevant information about the variable we care about. Everything else — noise, irrelevant detail, decorative complexity — can be discarded without loss. This is the same principle that underlies the Good Regulator theorem: a regulator need not model the entire system, only the information relevant to the essential variables it must protect.

Formal Definition and the Factorization Theorem

The Fisher-Neyman factorization theorem provides the standard test for sufficiency: T(X) is sufficient for θ if and only if the likelihood function can be factored as:

P(X|θ) = g(T(X),θ) · h(X)

where g depends on the data only through T(X) and h does not depend on θ. This factorization is the bridge between sufficiency and the theory of exponential families, where the natural sufficient statistics are the canonical coordinates of the family.

The theorem is deceptively simple. It says that the only thing that matters for inference about θ is the function T(X), and everything else is ancillary. But this simplicity conceals a deeper question: sufficient for what? A statistic sufficient for the mean of a distribution may be entirely uninformative about its variance. Sufficiency is always relative to a parameter and a model, and changing either can change what is sufficient.

Sufficiency and the Good Regulator

The connection between sufficient statistics and the Good Regulator theorem is direct and underappreciated. The theorem states that a good regulator must be a model of the system it regulates. But the regulator need not model the entire system — only the information relevant to the essential variables. This is exactly the sufficiency principle applied to control: the regulator's internal state must be a sufficient statistic for the system's state, relative to the essential variables.

In this formulation, the regulator is a lossy compressor of system dynamics. It discards information about the system that does not affect the essential variables, preserving only what is sufficient for control. The quality of the regulator is determined by how well its internal model captures the sufficient statistics of the system. A regulator that models irrelevant dynamics is wasting resources; a regulator that fails to model relevant dynamics is failing to regulate.

Minimal Sufficiency and the Limits of Compression

A sufficient statistic is minimal if it is a function of every other sufficient statistic. Minimal sufficient statistics are the most compressed representations that still preserve all relevant information. But minimal sufficiency is not always achievable: some models admit no single sufficient statistic that is minimal, and the search for minimality can itself be computationally intractable.

This intractability is the statistical analogue of the rate-distortion problem in control theory. Just as a regulator must trade bandwidth against control precision, a statistician must trade computational complexity against inferential precision. The minimal sufficient statistic is the ideal; the computable sufficient statistic is the practice. The gap between them is the cost of doing business in a world where information is not free.

The Editor's Claim

The sufficiency principle is the statistical expression of a philosophical position: that the world contains more than we need to know, and that wisdom is the art of knowing what to ignore. A sufficient statistic is not a simplification of reality; it is a recognition that reality is already simplified, and that most of what we observe is noise. The error of modern data science is not that it ignores sufficiency but that it worships the opposite: the idea that more data is always better, that every variable matters, that complexity is a virtue. The sufficiency principle says the opposite: the best model is the one that knows what to throw away.