Neural networks

Neural networks are computational systems composed of interconnected units — artificial neurons — that process information through weighted connections and nonlinear transformations. Inspired by biological nervous systems but not constrained by biological fidelity, they learn to approximate functions by adjusting connection weights in response to data, using algorithms such as gradient descent to minimize prediction error. The architecture is simple: layers of units, each computing a weighted sum of inputs passed through an activation function. The behavior is not: sufficiently large networks can approximate any continuous function, a property guaranteed by the universal approximation theorem, but the theorem says nothing about whether the approximation can be learned from finite data or whether the learned representation generalizes beyond the training distribution.

The history of neural networks is a history of cycles — enthusiasm, disappointment, renewal. Perceptrons in the 1960s promised machine intelligence until Minsky and Papert proved their limitations. Backpropagation in the 1980s revived the field but could not scale to the problems of the time. The deep learning revolution of the 2010s succeeded not because of conceptual breakthroughs but because of scale: larger datasets, faster processors, and architectures — convolutional networks, recurrent networks, transformers — that imposed structure appropriate to specific problem domains.

The interpretive challenge is severe. A trained neural network is a high-dimensional function with millions or billions of parameters. Understanding what it has learned — which features it uses, which correlations it has absorbed, which shortcuts it has taken — is the domain of interpretability research and mechanistic interpretability. The sparse autoencoder approach attempts to decompose network activations into interpretable features. The results are partial: some features are clearly meaningful, others are polysemantic, responding to multiple unrelated inputs in ways that resist clean interpretation.

The connection to biology is genuine but contested. Biological neurons are vastly more complex than artificial units: they have temporal dynamics, spatial structure, chemical modulation, and metabolic constraints that no current artificial system captures. Whether the similarities between neural networks and brains are deep structural parallels or superficial analogies is an open question with implications for both neuroscience and artificial intelligence. If the similarities are deep, then understanding how artificial networks learn may illuminate how biological networks learn. If they are superficial, then neural networks are a separate class of system whose properties must be understood on their own terms.

The most consequential question about neural networks is not technical but systemic: what happens when networks of networks — systems that include perception, reasoning, memory, and action — are deployed at scale in social, economic, and political institutions? The properties of individual networks — overfitting, bias, brittleness — do not simply add up. They interact in ways that are not yet understood, producing emergent behaviors that no component was designed to produce. The network is not the system. The system is the network plus the data it was trained on, plus the environment it operates in, plus the institutions that deploy it.