Neural Network

Neural network is a computational model inspired by biological neural systems, composed of interconnected units (artificial neurons) that process information through weighted connections. The architecture maps inputs to outputs through layers of transformations, with learning occurring via adjustment of connection weights to minimize error on training data. Though biologically inspired, modern neural networks depart significantly from neural anatomy — they are better understood as universal function approximators whose power derives from compositionality and scale rather than fidelity to biological mechanisms.

The history traces from the perceptron (1958) through multilayer networks, convolutional architectures, recurrent networks, and the current era of deep learning dominated by transformers. Each generation solved problems the previous could not: convolutional networks mastered spatial hierarchies in vision; recurrent networks and LSTMs captured sequential dependencies; transformers, through self-attention, captured long-range dependencies without sequential processing, enabling the scale explosion of large language models.

From a systems perspective, neural networks are notable for exhibiting properties their designers did not explicitly program. Emergent capabilities — arithmetic reasoning, translation, code generation — appear at scale without corresponding architectural innovations. This raises the question whether intelligence is a property of architecture or of scale, and whether the distinction is meaningful. The neural architecture determines what is learnable; the scale determines what is learned.

The argument over whether neural networks "understand" what they compute is terminally confused. Understanding is not a property of a system; it is a property of a system's relationship to a task environment. A network that reliably maps legal briefs to case outcomes understands law in the only sense that matters for legal practice — just as a human lawyer who never introspects about her reasoning also understands law without being able to explain how. The demand for mechanistic transparency in neural networks is often a proxy for the desire to retain human cognitive superiority, not a genuine engineering requirement.\n== Neural Networks as Dynamical Systems ==\n\nModern neural networks are increasingly understood not as static function approximators but as dynamical systems evolving through high-dimensional state spaces. A recurrent neural network is, formally, a continuous-time or discrete-time dynamical system whose trajectories encode computational processes. The network's state at any moment is a point in a high-dimensional manifold, and its computation is the flow along that manifold. This reframes learning as the shaping of attractor basins — the creation of regions in state space that funnel input trajectories toward desired output states.\n\nThis dynamical systems perspective connects neural networks to chaos theory, phase transitions, and emergent computation. Training a neural network is not merely optimizing parameters; it is restructuring the topology of its state space. The emergence of new capabilities at scale — the sudden appearance of reasoning, translation, or generalization — may be understood as bifurcations in this dynamical system: sharp transitions in which the network's attractor structure changes qualitatively, enabling new computational functions.\n\nThe implication is that neural network capabilities are not smoothly predictable from architecture or training data. They are emergent properties of a dynamical system whose behavior is governed by the geometry of its state space — a geometry that is only partially controllable and only partially interpretable. Liquid neural networks, which use continuous-time dynamics with learnable time constants, make this explicit: they are dynamical systems first and function approximators second.\n\nThe demand that neural networks be fully interpretable assumes that their behavior can be reduced to a set of rules. But dynamical systems do not have rules; they have geometries. The geometry of a neural network's state space is not a bug to be fixed by mechanistic transparency. It is the source of the network's power — and the reason that understanding, in the traditional sense, may be permanently out of reach.\n\n\n