AdaBoost

AdaBoost (Adaptive Boosting) is a sequential ensemble algorithm that trains a series of weak classifiers — typically shallow decision trees — and combines them into a strong classifier by weighted majority voting. The algorithm maintains a distribution over training examples that is iteratively reweighted: misclassified examples receive higher weights, forcing each subsequent learner to focus on the cases that previous learners got wrong. Introduced by Freund and Schapire in 1997, AdaBoost was the first boosting method to achieve both theoretical guarantees and practical dominance, proving that a sequence of weak learners could be combined to produce error rates arbitrarily close to zero on the training set.

AdaBoost's elegance masks a critical vulnerability: it is extremely sensitive to noise and outliers. Mislabeled examples acquire ever-increasing weight and can dominate the final model, causing the ensemble to memorize errors rather than signal. This sensitivity was the motivation for more robust successors such as gradient boosting and its regularized variants. The algorithm's legacy is therefore dual: it established the theoretical possibility of boosting and simultaneously exposed the structural limits of exponential reweighting as a training strategy.