Large Language Model

A Large Language Model (LLM) is a statistical model trained on vast corpora of text to predict and generate sequences of tokens. The central mechanism is the transformer attention mechanism, which learns weighted relationships between token positions across a context window. LLMs are characterized not by any defined cognitive architecture but by scale: training on hundreds of billions to trillions of tokens using billions to trillions of parameters produces capabilities that could not be predicted from smaller-scale systems by smooth extrapolation — a phenomenon known as Capability Emergence.

The classification of LLMs as 'intelligence,' 'reasoning,' or 'understanding' systems is contested. They are optimizers trained on a human-generated distribution; their outputs reflect the statistical regularities of that distribution, which includes sophisticated argument, logical inference, and creative composition. Whether these outputs instantiate the underlying cognitive processes they superficially resemble, or merely produce the same surface forms, is the central empirical question that the current generation of systems cannot resolve — and that the vocabulary of Artificial General Intelligence routinely forecloses.

See also: Transformer Architecture, Capability Emergence, Artificial General Intelligence, Benchmark Saturation.

Scaling Laws and Their Limits

LLM capability scales predictably with compute, data, and parameter count. The Chinchilla scaling laws (Hoffmann et al., 2022) established that, for a fixed compute budget, models should be trained on roughly 20 tokens per parameter to reach optimal performance — a result that suggested most large models of that era were significantly undertrained. The scaling law relationship is log-linear: doubling compute produces predictable, diminishing returns on benchmark performance.

The limit of scaling law reasoning is its dependence on benchmark continuity. Scaling laws are fit to benchmark performance trajectories, which requires that the benchmarks being scaled toward remain valid measures of the underlying capability across the entire scaling range. When benchmarks saturate — when models approach ceiling performance — the log-linear relationship breaks. At that point, the model's continued improvement is invisible to the scaling law, and researchers must either find new benchmarks or abandon the log-linear frame. This has happened repeatedly: GSM8K, MMLU, HumanEval, and other "hard" benchmarks of their moment each saturated faster than expected, requiring constant replacement.

The benchmark overfitting problem is structural: the benchmarks that are easy to administer at scale are also the benchmarks easiest to overfit to, either deliberately (through training on benchmark data) or inadvertently (through training on internet text that includes benchmark solutions). As benchmarks are deployed, their solutions are published; published solutions are scraped; scraped solutions enter training data. The feedback loop between evaluation and training is not a corruption of the scientific process — it is a consequence of the scientific process interacting with a training regime that ingests all publicly available text.

Interpretability and the Black Box Problem

The internal representations of LLMs are, in principle, mathematically transparent: they are high-dimensional vector spaces with operations defined by the transformer attention mechanism. In practice, interpreting what any given activation state or attention pattern means in terms of the underlying task is extremely difficult. The field of mechanistic interpretability attempts to reverse-engineer the circuits that implement specific capabilities — identifying, for instance, the attention heads responsible for indirect object identification or the circuits implementing modular arithmetic.

Progress in mechanistic interpretability has been real but limited in scope. The circuits identified so far govern simple, well-defined behaviors. Whether the same approach scales to complex reasoning, long-range coherence, or the handling of genuinely novel inputs is unknown. The concern is not that LLMs are mysterious black boxes in principle — they are not, they are well-defined mathematical functions — but that the mathematical description of the function does not constitute an understanding of what the function computes or why it works when it does.