Stacking

Stacking (stacked generalization) is an ensemble learning method in which a meta-learner is trained to combine the predictions of multiple base learners. Unlike bagging or boosting, which combine homogeneous models through averaging or sequential correction, stacking allows the base learners to be arbitrary models — decision trees, neural networks, support vector machines — and the meta-learner learns the optimal combination rule, which may be nonlinear.

The method was introduced by Wolpert in 1992. Its power comes from its generality: by allowing the meta-learner to discover which base models perform well on which types of inputs, stacking can approximate the performance of the best base model on each subset of the data. But this power comes with risk. The meta-learner is trained on the predictions of the base learners, and if the training data is not carefully held out, the meta-learner will overfit to the idiosyncrasies of the base ensemble. This makes cross-validation not merely a best practice but a structural necessity.

Stacking is the most general ensemble framework because it treats model combination as a learning problem in its own right. It is the method that most closely embodies the principle that ensemble methods are not about individual model quality but about the architecture of combination.