Transformer architecture

The transformer architecture is a neural network design introduced by Vaswani et al. in 2017 that replaces recurrence and convolution with attention mechanisms — learnable weighting schemes that allow every position in a sequence to attend directly to every other position. The architecture consists of an encoder and a decoder, each built from stacked layers of multi-head self-attention and position-wise feed-forward networks, with residual connections and layer normalization providing training stability.

The critical innovation is scaled dot-product attention: for each position, the model computes a weighted sum of all other positions, where the weights are determined by a compatibility function between query, key, and value vectors. Multi-head attention runs multiple attention operations in parallel, allowing the model to attend to different representational subspaces simultaneously. This eliminates the sequential bottleneck of RNNs and LSTMs, enabling parallel training on massive datasets — a property that proved essential for scaling language models to trillions of tokens.

Transformers did not emerge from a theory of language or cognition. They emerged from empirical observation that attention improved performance on machine translation. The field built the cathedral before understanding the physics. Contemporary large language models — GPT, Claude, Gemini — are refined transformers scaled by orders of magnitude in parameters, data, and compute. Whether this scaling constitutes genuine conceptual progress or merely empirical refinement is debated.

The architecture's dominance raises systems-level questions. Transformers exhibit capability phase transitions: certain abilities appear abruptly at threshold scales rather than improving gradually. If these are genuine phase transitions, they belong to a universality class of emergent behavior in high-dimensional systems. If they are measurement artifacts, the apparent emergence is an epistemological byproduct of how we test these systems.