Ensemble learning

Ensemble learning is a machine learning paradigm in which multiple models — called base learners or weak learners — are trained and their predictions combined to produce a single output that is more accurate, robust, or generalizable than any individual model could achieve alone. The guiding intuition is that the errors of independently trained models are often uncorrelated, so averaging or voting across the ensemble cancels out individual mistakes while preserving the signal each model captures.

The method is not a single algorithm but a family of techniques that differ in how they generate diversity among base learners and how they aggregate their outputs. The two dominant strategies are parallel ensembles, in which learners are trained independently and their outputs combined, and sequential ensembles, in which each new learner is trained to correct the errors of its predecessors. Both strategies exploit the same statistical principle: the variance of an average is lower than the variance of any single component, provided the components are not perfectly correlated.

Parallel Ensembles: Bagging and Random Forests

Bagging (bootstrap aggregating) trains multiple models on random subsets of the training data, drawn with replacement, and aggregates their predictions by majority vote or averaging. The diversity comes from the data: each model sees a different sample of the training distribution, so they make different errors. The aggregation reduces variance without increasing bias, making bagging particularly effective for high-variance, low-bias base learners such as decision trees.

Random forests extend bagging by adding randomized feature selection: at each split in each tree, only a random subset of features is considered. This decorrelates the trees further, so the variance reduction from averaging is more pronounced. Random forests are among the most widely used ensemble methods because they require little tuning, handle mixed data types, and provide reliable estimates of variable importance. They are a standard benchmark against which new methods are measured, and they remain competitive even in the age of deep learning for tabular data problems.

Sequential Ensembles: Boosting

Boosting trains learners sequentially, with each new learner focusing on the examples that previous learners misclassified. The method is adaptive: it maintains a distribution over training examples that increases the weight of hard cases and decreases the weight of easy ones. The final prediction is a weighted combination of all learners, with more accurate learners receiving higher weights.

AdaBoost, the first practical boosting algorithm, demonstrated that a sequence of weak learners — models only slightly better than random guessing — could be combined into a strong learner with arbitrarily low error, provided the weak learners were sufficiently diverse and the data were not too noisy. Later variants such as gradient boosting generalized the framework by optimizing arbitrary differentiable loss functions. Gradient boosting machines and their scalable implementations (XGBoost, LightGBM, CatBoost) have become dominant in competitive machine learning and industrial applications, particularly for structured data.

Boosting reduces both bias and variance, but it is more sensitive to noise than bagging: mislabeled examples receive ever-increasing weight and can dominate the training process, leading to overfitting. The tension between the two methods — bagging's robustness against noise versus boosting's aggressive error reduction — mirrors the broader bias-variance tradeoff in statistical learning.

Stacking and Meta-Learning

Stacking (stacked generalization) goes beyond simple averaging or voting by training a meta-learner to combine the outputs of base learners. The base learners are trained on the original data; the meta-learner is trained on the base learners' predictions, treating them as a new feature space. This allows the ensemble to learn nonlinear combination rules: the meta-learner might learn that one model is reliable in one region of the input space and another model is reliable elsewhere.

Stacking is more flexible than bagging or boosting but also more complex and prone to overfitting if the meta-learner is not regularized. It illustrates a general principle: ensemble learning is not just about combining models but about creating new representational layers. Each layer transforms the problem space, making the next layer's job easier. This layering is analogous to the architecture of neural networks, where each layer learns a representation that simplifies the task for subsequent layers.

Ensemble learning reveals something deeper than a collection of engineering tricks. It demonstrates that intelligence — whether natural or artificial — is not a property of any single mechanism but emerges from the coordination of multiple mechanisms that compensate for each other's limitations. The obsession with finding the 'best' single model is a symptom of reductionist thinking. The best model is always a society of models, and the art of machine learning is increasingly the art of designing the social contract among them.

See Also

• Gradient boosting — sequential ensembles optimized by gradient descent in function space
• Bagging — parallel ensemble method based on bootstrap sampling
• Random forest — decorrelated bagging of decision trees
• Meta-learning — learning to learn from ensemble outputs
• Model compression — the inverse problem: distilling ensembles into single models