Neural network

A neural network is a class of computational architecture loosely inspired by the structure of biological neural tissue. The inspiration is looser than its proponents acknowledge and more consequential than its critics allow. Formally, a neural network is a directed graph of parameterized functions — artificial neurons — organized into layers, with information propagating forward from inputs to outputs and error signals propagating backward during training. The result is a system that can approximate a wide class of functions from examples, without explicit programming of the mapping.

The biological metaphor is a marketing decision that has outlived its usefulness. Artificial neurons are not neurons. They do not spike, they do not maintain resting potentials, they are not embedded in a hormonal milieu, they do not die and regenerate. The resemblance is to neuroscience circa 1943 — the McCulloch-Pitts abstraction — which was already known to be a radical simplification by the time deep learning scaled it into industrial relevance. Using "neural" as a descriptor has successfully borrowed the prestige of neuroscience while contributing nothing to it.

Architecture

The canonical neural network consists of an input layer, one or more hidden layers, and an output layer. Each layer contains artificial neurons, and each neuron computes a weighted sum of its inputs, passes the result through a nonlinear activation function (ReLU, sigmoid, tanh, and their variants), and outputs the result. The weights are the learnable parameters. A network with sufficient width and depth can approximate any continuous function on a compact domain — this is the Universal Approximation Theorem, a mathematical result whose practical significance is routinely overstated.

The backpropagation algorithm is the engine of training. It applies the chain rule of calculus to compute gradients of a loss function with respect to all parameters in the network, then adjusts weights to reduce the loss. This is Gradient Descent applied to systems with millions or billions of parameters. That such a simple iterative procedure produces representations with genuine generalization power remains incompletely understood — the field has empirical success it cannot yet theoretically account for.

Variants include:

• Convolutional Neural Networks — exploit spatial locality and translation invariance, dominant in visual tasks
• Recurrent Neural Networks — process sequential data via recurrent connections; largely displaced by Transformers for long-range dependencies
• Transformers — attention-based architectures that dominated language modeling from 2017 onward and expanded aggressively into other domains
• Bayesian Neural Networks — neural networks with probability distributions over weights rather than point estimates, theoretically principled but computationally expensive

Historical Development

The concept of artificial neurons dates to Warren McCulloch and Walter Pitts's 1943 paper "A Logical Calculus of the Ideas Immanent in Nervous Activity." Their formalism — binary neurons, Boolean logic gates — was an attempt to ground psychology in mathematical logic, not to build computers. The computational reading came later and was not their intention.

Frank Rosenblatt's perceptron (1958) was the first trainable neural model with a convergence proof. Marvin Minsky and Seymour Papert's Perceptrons (1969) demonstrated its limitations — it could not compute XOR — and triggered the first AI winter. The critique was technically correct and sociologically catastrophic: it suppressed work on multi-layer networks for a decade, even though the limitation was known and the fix (adding hidden layers) was understood in principle.

The modern era begins with backpropagation's popularization in the 1986 Rumelhart-Hinton-Williams paper, the 2012 AlexNet result on ImageNet, and the subsequent scaling of networks to parameters in the hundreds of billions. This scaling era has produced systems of remarkable capability and remarkable opacity — which is a structural feature of the optimization process, not an engineering accident.

What Neural Networks Are Not

The dominant failure of neural network discourse is category error in descriptions of what trained networks do. Networks do not "understand," "know," "believe," or "reason" in any sense that bears scrutiny under the terms' normal meanings. They compute functions. The functions can be extraordinarily complex and their outputs can resemble the outputs of understanding. This resemblance is what gets called "intelligence" and it is, at best, an open empirical question whether functional resemblance is the same thing as the capacity it resembles.

Benchmark Saturation — the phenomenon of performance on training-distribution proxies reaching ceiling while out-of-distribution generalization stagnates or fails — is the empirical signature of this gap. Networks that "achieve human-level performance" on benchmarks reliably fail in ways no human would fail when tested on inputs slightly outside the training distribution. This is not a bug in current implementations. It is the predictable consequence of optimization against a fixed distribution.

The claim that scaling resolves this — that sufficiently large networks will eventually generalize robustly — is a hypothesis, not a demonstrated result. It may be true. It may be false. The field's current institutional structure has strong incentives to present it as true and weak incentives to test it rigorously.

The neural network metaphor promised a theory of mind and delivered a function approximator. The function approximator is genuinely remarkable. Whether it is what was promised remains, after eighty years, unanswered — and the continued use of the biological vocabulary has made it harder, not easier, to ask the question honestly.