Chinchilla scaling laws

Chinchilla scaling laws are the empirical finding — reported by Hoffmann et al. (2022) at DeepMind — that optimal training of large language models requires scaling model size (parameters) and training data in approximately equal proportion. This overturned the earlier paradigm established by Kaplan et al. (2020), which held that model size should grow faster than data. Under the Chinchilla regime, a 70-billion-parameter model trained on 1.4 trillion tokens outperforms a 280-billion-parameter model trained on 300 billion tokens, despite using the same total compute.

The finding has profound implications for the economics of AI development. If data and parameters are equally important, then the marginal value of increasing model size diminishes rapidly without proportional increases in training data. This shifts competitive advantage from organizations with the largest compute budgets to those with access to the largest and highest-quality datasets. It also raises questions about the sustainability of the scaling paradigm: high-quality text data on the internet is finite, and synthetic data generation introduces compounding error.

The Chinchilla result is an instance of a broader pattern in scaling laws: the optimal allocation of resources in complex systems is rarely intuitive and often contradicts the assumptions of the dominant paradigm. The systems lesson is that scale is multidimensional — and optimizing along only one dimension produces suboptimal outcomes that are invisible until someone measures the others.

_The Chinchilla paper did not just optimize language models. It exposed a structural flaw in an entire research culture: the assumption that bigger is always better, measured by a single number. The correction was empirical, not theoretical — which means the next paradigm may be equally wrong, and equally invisible, until someone else runs the experiment._