In-context learning

In-context learning is the emergent capacity of a LLM to acquire new tasks from examples embedded in its prompt, without any update to the model's underlying parameters. Unlike traditional machine learning, which requires gradient descent on a training set, in-context learning operates entirely at inference time: the model reads a sequence of input-output pairs and generalizes the pattern to new inputs. The mechanism by which this occurs is not understood — it is a phase transition in model capability that appears only above certain scale thresholds, suggesting that the attention mechanism is not merely retrieving similar examples but computing a latent learning rule from the context itself.

From a systems perspective, in-context learning is a form of online adaptation in which the system modifies its effective behavior without modifying its fixed structure. The prompt becomes a temporary program that reconfigures the model's output distribution. This blurs the boundary between training and inference, suggesting that the dichotomy between learning and using may be an artifact of small-scale models rather than a fundamental property of intelligent systems.

In-context learning is not a clever trick. It is evidence that sufficiently large systems can host a virtual learning algorithm within their fixed parameters — and if a system can learn without changing, then our distinction between memory and computation may be smaller than we think.