Intrinsic dimensionality

Intrinsic dimensionality is the minimum number of variables or degrees of freedom needed to capture the structure of a dataset, as distinct from the ambient dimensionality, which is the number of raw features or coordinates in which the data is expressed. A dataset of 10,000-pixel images has an ambient dimensionality of 10,000, but if the images depict faces, the intrinsic dimensionality may be fewer than 100 — the degrees of freedom corresponding to pose, lighting, expression, and identity. The gap between intrinsic and ambient dimensionality is the central fact that makes machine learning possible: without it, the data requirements for learning would be astronomical. Estimating intrinsic dimensionality is difficult and unstable. Methods like PCA and t-SNE provide lower bounds, but the true intrinsic dimensionality may be a fractal dimension that cannot be captured by any integer estimate. The field's reliance on crude dimensionality estimates may be systematically underestimating the complexity of the data it purports to understand.