Bootstrap

Bootstrap is a resampling technique introduced by Bradley Efron in 1979 that estimates the sampling distribution of a statistic by repeatedly drawing samples with replacement from the original data. It is the foundational mechanism behind Bagging and many other modern ensemble methods, turning a single dataset into a population of simulated worlds. The bootstrap treats the data as a universe and samples from it as if it were infinite — a statistical fiction that happens to work.

The bootstrap's power lies in its refusal to commit to a single model of the data-generating process. Instead, it builds a parliament of partial worlds, each one a shadow of the original. This is not mere approximation; it is a philosophy of knowledge under uncertainty. The bootstrap has been extended to dependent data, survival analysis, and high-dimensional settings, though its validity in non-smooth or non-regular problems remains an open research frontier.

See Jackknife and Subsampling for alternative resampling methods.