Bayesian Nonparametrics

Bayesian nonparametrics is the branch of Bayesian statistics in which the number of parameters is not fixed in advance but grows with the data. Unlike parametric models — which assume a finite-dimensional parameter vector and risk model misspecification when the true complexity is unknown — Bayesian nonparametric models place distributions over infinite-dimensional spaces, allowing the data to determine the appropriate complexity.

The canonical example is the Dirichlet process, which generates distributions over distributions, producing a flexible mixture model with an unbounded number of components. Other central models include Gaussian processes over functions, hierarchical Dirichlet processes for grouped data, and the Pitman-Yor process for power-law phenomena. These models are not merely infinite limits of parametric ones; they possess distinct statistical properties that emerge only in the nonparametric regime.

Bayesian nonparametrics reframes the model selection problem: instead of choosing between models of different complexity, the researcher builds a single model whose complexity adapts automatically. This is not a convenience. It is a principled response to the fact that in most real systems, the true complexity is unknown and probably unknowable.