Bias-variance decomposition

The bias-variance decomposition is a foundational result in statistical learning theory that decomposes the expected prediction error of a model into three components: the square of the bias (systematic error due to overly simplistic assumptions), the variance (error due to sensitivity to small fluctuations in the training set), and irreducible noise (error inherent in the data-generating process itself). The decomposition reveals a fundamental tradeoff: reducing bias typically increases variance, and vice versa.

This tradeoff is the theoretical engine behind ensemble methods. Bagging reduces variance by averaging multiple models trained on different data subsets. Boosting reduces bias by sequentially correcting the errors of previous models. No single model can simultaneously minimize both bias and variance, but an ensemble — by distributing the tradeoff across multiple models — can achieve a better net error than any individual member. The decomposition thus shows that the path to better prediction lies not in finding a single perfect model but in organizing a parliament of imperfect ones.