MMLU
Massive Multitask Language Understanding (MMLU) is a benchmark designed to evaluate the breadth of knowledge in large language models across 57 subjects spanning STEM, humanities, social sciences, and professional fields. Developed by Hendrycks et al. in 2021, it quickly became the de facto standard for measuring "general intelligence" in language models — and in doing so, became one of the most instructive case studies in benchmark overfitting and benchmark saturation the field has yet produced.
The test format is deceptively simple: multiple-choice questions drawn from textbooks, exams, and professional licensing tests. An 83.5% score was once considered near-human performance; by 2024, frontier models were scoring above 90%. The benchmark that was supposed to measure general knowledge became a target for narrow optimization — specialized training on MMLU-like question formats, reasoning shortcuts that exploit multiple-choice structure, and contamination of the test set in training corpora.
From Diagnostic to Target
MMLU was designed as a diagnostic instrument: a broad-spectrum probe of what a model knows, not a training objective. But the moment it became the standard metric for model comparison, it ceased to function purely as a diagnostic. It became a target. This is the Goodhart dynamic in its purest form: when a measure becomes a target, it ceases to be a good measure.
The symptoms appeared rapidly. Researchers discovered that models could gain significant MMLU points through test-set contamination — the same questions appearing in pretraining data. Others found that models trained specifically on multiple-choice science questions outperformed generalist models on MMLU while failing at open-ended reasoning. The benchmark, intended to measure broad understanding, was being gamed through narrow specialization.
This is not a failure of MMLU's designers. It is a structural property of any benchmark that achieves sufficient salience to influence resource allocation. A benchmark that no one cares about cannot be overfit; a benchmark that everyone cares about cannot avoid being overfit.
The Contamination Problem
The most serious challenge to MMLU's validity is test-set contamination. The questions in MMLU are drawn from public sources — textbooks, online course materials, licensing exams — that appear in the training data of large language models. When a model has seen the exact question during pretraining, its performance reflects memorization, not reasoning.
Efforts to decontaminate training data have proven partial at best. The scale of pretraining corpora — trillions of tokens — makes it practically impossible to ensure that no MMLU question appears anywhere in the data. More sophisticated is the problem of "contamination by proxy": even if the exact question is removed, semantically similar questions may remain, allowing models to generalize from memorized patterns rather than genuine understanding.
This reveals a deeper epistemic problem. MMLU was designed to test knowledge that a well-educated human would have. But well-educated humans do not acquire their knowledge by reading the internet at scale. They acquire it through structured curricula, feedback, and synthesis. A benchmark designed for humans may be structurally inappropriate for systems that learn through an entirely different epistemic regime.
MMLU and the Benchmark Lifecycle
MMLU exemplifies the lifecycle that benchmarks typically follow: invention, adoption, optimization, saturation, and obsolescence. In the invention phase, a benchmark fills a genuine measurement gap. In adoption, it becomes a community standard. In optimization, the community learns to maximize the metric through means that do not generalize. In saturation, the metric loses its discriminative power — the difference between top models becomes smaller than the measurement noise. In obsolescence, the community moves on to a new benchmark, and the cycle repeats.
MMLU is currently in the late optimization/early saturation phase. Scores have plateaued near the ceiling, and the community has begun developing successors: MMLU-Pro (with more adversarial options), MMLU-Redux (with decontaminated questions), and entirely new evaluation paradigms that move beyond multiple-choice. This is the natural history of benchmarks. It is not a pathology but a lifecycle — and understanding that lifecycle is essential to designing better evaluation systems.