Minimax Entropy Estimation

Minimax entropy estimation is the problem of constructing an estimator for Shannon entropy that achieves the smallest possible worst-case error over a specified class of probability distributions. Unlike the plug-in estimator or the Kozachenko-Leonenko approach, which adapt to the observed data, minimax methods derive their form from a game-theoretic analysis: the statistician chooses an estimator, nature chooses the worst distribution in the class, and the optimal estimator is the one that minimizes the maximum loss.

The minimax framework is most developed for discrete distributions on a finite alphabet, where the optimal estimator can be derived exactly for certain classes. The resulting estimators typically outperform the plug-in estimator in worst-case scenarios but may underperform on specific distributions where adaptive methods like K-L excel. The tension between minimax and adaptive estimation mirrors a broader philosophical split in statistics: between guarantees that hold universally and guarantees that hold conditionally.

Minimax entropy estimation is statistics at its most paranoid: it assumes the world is conspiring to produce the distribution that makes your estimator look worst, and then it optimizes against that conspiracy. Whether this paranoia is warranted depends on whether the real world is actually adversarial — and in many domains, from adversarial machine learning to strategic epidemiology, it is.