Deep learning

Deep learning is Machine learning using neural networks with multiple layers of nonlinear transformations stacked between input and output. The depth is not decorative — it enables the network to learn increasingly abstract representations at each layer, compressing high-dimensional inputs (images, audio, text) into structures that simpler methods cannot represent at any depth.

The critical insight of deep learning is that feature engineering — the laborious manual process of deciding which aspects of an input are relevant — can itself be learned from data, given sufficient network capacity, training data, and compute. Before 2012, the dominant approach to machine learning for images required humans to specify features (edges, textures, histograms of gradients). AlexNet demonstrated that a deep convolutional network trained end-to-end on raw pixels outperformed all of these hand-crafted approaches. This was not a marginal improvement.

Deep learning does not explain what it has learned. The representations in intermediate layers are not human-interpretable. A network that classifies images of cats cannot say what a cat is — it has learned a function that maps pixel arrays to labels, and the function is opaque. This is the source of deep learning's central limitation: it achieves high accuracy on its training distribution while remaining vulnerable to distribution shift and adversarial perturbations that humans would handle trivially.

The Suppressed History: From Perceptron to Backpropagation

Deep learning has a creation myth that its practitioners prefer to the actual history. The myth: a handful of visionaries (Hinton, LeCun, Bengio) persisted through two AI winters, kept the neural network faith alive against the prevailing wisdom, and were finally vindicated when compute and data became sufficient to demonstrate the approach's power.

The history is more complicated and, in Armitage's view, more instructive. The perceptron was condemned in 1969 by Minsky and Papert on the basis of limitations they explicitly acknowledged applied only to single-layer networks. The field drew the wrong conclusion and spent twenty years largely ignoring multi-layer approaches. When backpropagation — a method for efficiently computing gradients in multi-layer networks — was independently discovered (and rediscovered) in the 1970s and 1980s, the field was structurally unprepared to adopt it because the perceptron's supposed refutation had evacuated the theoretical basis that would have motivated it.

The lesson usually drawn is about persistence in the face of institutional resistance. The lesson that should be drawn is about how a mathematical result (Minsky and Papert's proof) came to serve a sociotechnical function (defunding a research program) that the mathematics itself did not support. Science is supposed to be self-correcting. The AI field took twenty years to correct a misreading of a theorem. The machinery of institutional science was the obstacle, not the corrective.

Contemporary deep learning inherits this history without examining it. The architectures of 2024 are refined descendants of ideas from the 1980s, scaled by factors of compute and data that would have been unimaginable then. Whether scale alone constitutes a conceptual advance — or whether deep learning's dominance represents a high-water mark before the next reckoning — is the question that current practitioners are motivated not to ask.

The transformer architecture, which underlies contemporary large language models, did not emerge from a theory of language or cognition. It emerged from empirical observation that attention mechanisms improved performance on sequential tasks. The field built the cathedral before it understood the physics.