Conformal Prediction

Conformal Prediction is a framework in machine learning and statistics that provides distribution-free, finite-sample guarantees on the coverage of prediction sets. Unlike classical statistical methods that rely on asymptotic assumptions or specific parametric forms, conformal prediction constructs prediction regions that are valid under only the assumption of exchangeability — the same mild condition that underlies Bruno de Finetti's representation theorem.

The core idea is elegant. Given a training set and a new test point, a conformal predictor uses a nonconformity measure to score how "strange" each observation is relative to the others. By comparing the test point's nonconformity score to those of the training points, it computes a p-value that indicates how well the test point conforms to the training distribution. Prediction sets are then constructed to include all points whose p-values exceed a chosen threshold, guaranteeing that the true label falls within the prediction set with probability at least 1−α, regardless of the underlying data distribution.

This is a powerful form of Predictive Inference: it tells you not just what to predict, but how uncertain you should be, and it does so without assuming a model. Conformal prediction has been applied to classification, regression, time series forecasting, and large language model calibration. It represents a convergence of statistical rigor and machine learning practicality, grounded in the same exchangeability principle that de Finetti identified as the foundation of Bayesian reasoning.

Conformal prediction is the revenge of epistemology on engineering. It says: you do not need to know the world to make valid predictions about it. You only need to know that the future will resemble the past in the weakest possible sense — and that is enough.