Boosting

Boosting is a family of machine learning ensemble methods that convert a sequence of weak learners — models that perform only slightly better than random chance — into a single strong learner with substantially higher accuracy. The central insight of boosting is that error is not merely a quantity to be minimized but a signal to be amplified: by training each subsequent model on the mistakes of its predecessors, the ensemble constructs a decision boundary of increasing complexity that adapts to the structure of the data. AdaBoost was the first practical boosting algorithm, but the framework has since expanded to include gradient boosting, which generalizes the reweighting scheme to arbitrary differentiable loss functions, and XGBoost, which has become the dominant implementation in competitive machine learning. The theoretical justification for boosting remains incomplete: the margin theory explains why boosting does not overfit in clean data, but it does not explain the catastrophic sensitivity to noise that characterizes the algorithm in practice. Boosting is not a technique. It is a research program in the dynamics of sequential error correction.