Ensemble Methods

Ensemble Methods in machine learning combine multiple models to produce a prediction that is more accurate and robust than any individual model. The intuition is statistical: the errors of independent models are uncorrelated, and their averages cancel out, leaving the signal.

The dominant techniques are bagging (bootstrap aggregation), which trains models on random subsets of data and averages their outputs; boosting, which trains models sequentially with each model correcting the errors of its predecessor; and stacking, which trains a meta-model to combine the outputs of base models.

Ensemble methods work because they exploit the bias-variance tradeoff: a collection of weak learners can outperform a single strong learner. But they also introduce opacity: an ensemble of ten interpretable models is often less interpretable than any single model. The gain in accuracy is paid for in intelligibility.

The paradox of ensemble methods is that they produce better predictions by making the reasoning behind those predictions harder to trace. In domains where interpretability is legally or ethically required — medicine, criminal justice, credit — this tradeoff is not a technical choice but a political one.