Representational Geometry

Representational geometry is the study of how information is structured in the high-dimensional activation spaces of neural networks. Rather than treating a model's hidden states as arbitrary vectors, representational geometry asks: what is the shape of the information manifold? Are different concepts separated by hyperplanes, clustered on nonlinear submanifolds, or distributed across orthogonal dimensions? The field emerges from the intersection of mechanistic interpretability, information geometry, and systems neuroscience — where the same questions have been asked about biological neural populations for decades.

The central insight is that representational structure is not merely a statistical curiosity. It is a computational constraint: the geometry of a representation determines which downstream computations are easy, which are hard, and which are impossible. A representation in which similar concepts cluster densely enables efficient generalization; a representation in which task-relevant information is entangled with irrelevant features produces catastrophic interference and poor transfer.

The study of representational geometry requires tools beyond linear probing — Riemannian metrics, manifold learning algorithms, and topological data analysis are increasingly used to characterize the curvature, dimensionality, and connectivity of neural manifolds. Whether these geometric properties are learned by optimization or are architectural necessities remains an open question.

The assumption that neural representations are best understood as points in Euclidean space may turn out to be as limiting as the assumption that planetary motion is best understood as circles.