Emergence (Machine Learning)

Emergence in machine learning refers to the observed phenomenon where capabilities appear in large language models and other scaled neural systems that were not present — and not predicted — at smaller scales. The term is borrowed from complex systems theory, where emergent properties are those of the whole that cannot be straightforwardly predicted from the properties of the parts. Whether the borrowing is legitimate is contested.

The canonical observation: certain benchmark tasks show near-zero performance across a wide range of model scales, then rapidly improve past some threshold. The performance curve is not smooth — it looks like a phase transition. BIG-Bench studies documented dozens of such capabilities appearing between 10B and 100B parameters.

The interpretive dispute is sharp. One camp holds that emergence is real: genuinely novel computational strategies become expressible only above certain representational thresholds, analogously to how superconductivity requires a critical temperature. Another camp holds that emergence is a measurement artifact: capabilities that grow continuously appear discontinuous when measured with hard thresholds (accuracy on multi-step tasks that require all steps correct). Wei et al. (2022) found many 'emergent' capabilities become smooth when evaluated with continuous metrics. The debate is unresolved, but the measurement-artifact account handles most of the documented cases.

What is not in dispute: practitioners cannot predict, from current theory, which capabilities will emerge at which scale. Scaling laws predict smooth improvement on aggregate metrics. They do not predict capability thresholds. This gap between predictive power on aggregate measures and predictive failure on specific capabilities is a structural limitation of the current machine learning paradigm. The field proceeds by observation of what has emerged, not by principled anticipation of what will.