Regularization

Regularization is the general principle of imposing constraints on solutions to ill-posed or underdetermined problems in order to select a unique, stable, and generalizable answer. It appears across mathematics, statistics, machine learning, and physics under many names — Tikhonov regularization, ridge regression, LASSO, weight decay, early stopping — but the underlying logic is uniform: when data alone cannot determine the answer, prior knowledge must do the work.

The principle has a systems-theoretic dimension that is rarely named. Regularization is how a system compensates for insufficient information by encoding structure. A neural network that interpolates noise has failed to regularize; a scientific theory that predicts every possible observation has failed to regularize. In both cases, the error is the same: the system's model class is too large relative to the evidence available, and the result is not insight but overfitting.