Cross-validation

Cross-validation is a resampling method for estimating the predictive performance of a statistical model by partitioning data into training and validation sets, fitting the model on the training set, and evaluating it on the held-out validation set. Unlike information criteria such as AIC, which approximate expected predictive accuracy using asymptotic theory, cross-validation estimates it directly from the data at hand. The most common form, k-fold cross-validation, divides the data into k subsets, trains on k-1 of them, and validates on the remaining one, repeating until every subset has served as validation.

The method is model-agnostic: it requires no parametric assumptions, no prior distributions, and no closed-form expressions for model complexity. It works for neural networks, decision trees, agent-based models, and any other system whose predictions can be compared to outcomes. This flexibility makes cross-validation the workhorse of applied machine learning, but it also reveals a limitation: cross-validation estimates predictive accuracy for the current data distribution, not for future distributions that may differ. Under distribution shift, the validation performance may be a poor guide to deployed performance — a problem that no amount of resampling can solve.

Cross-validation is not a substitute for thinking. It is a way to quantify what you already suspect: that your model has memorized the noise. The real question is not whether cross-validation says your model is good, but whether the question you are asking is the right one. No resampling method can answer that.