Intrinsic Dimensionality

Intrinsic dimensionality is the number of degrees of freedom actually needed to describe a dataset or a phenomenon, as opposed to the nominal dimensionality of the space in which it is represented. A smooth curve embedded in three-dimensional space has intrinsic dimension 1; a crumpled sheet has intrinsic dimension 2; a dataset of 1000-dimensional vectors that all lie near a 12-dimensional manifold has intrinsic dimension 12. The gap between intrinsic and extrinsic dimension is the precise measure of how much the curse of dimensionality has been evaded.\n\nEstimating intrinsic dimension is harder than it appears. Standard methods — correlation dimension, nearest-neighbor distances, eigenvalue decay — give different answers for the same data, and the answer often depends on the scale at which one looks. A fractal structure has no single intrinsic dimension; its dimension changes with magnification. The manifold hypothesis assumes a clean separation between intrinsic and extrinsic dimension, but real data may occupy a noisy, thickened manifold or a hierarchy of structures with no single scale.\n\nThe concept matters because it determines the sample complexity of learning. A problem with intrinsic dimension d can often be solved with a number of samples polynomial in d, even if the ambient dimension is exponentially larger. The art of high-dimensional learning is the art of discovering that the apparent complexity was never real.\n\nThe claim that a dataset has 'low intrinsic dimension' is often made with more confidence than the estimation warrants. It is easy to mistake the artifact of a representation for the geometry of the phenomenon. The history of science is littered with cases where the true dimension was higher than expected — and the models built on low-dimensional assumptions failed catastrophically when pushed beyond the regime in which the approximation held.\n\n\n\n\nAmong the most robust estimators of intrinsic dimension is the correlation dimension, which measures how the number of point pairs within distance r scales with r as r goes to zero. For a true d-dimensional manifold, this scaling follows a power law with exponent d. For fractal structures, the exponent is non-integer, revealing a geometry that cannot be captured by classical manifold assumptions.