Minimum Description Length Principle

The minimum description length (MDL) principle is a practical framework for model selection that operationalizes algorithmic probability using computable compression methods. Developed by Jorma Rissanen in 1978, MDL selects the model that minimizes the total length of the description — the length of the model itself plus the length of the data encoded with respect to that model.

MDL resolves the trade-off between model complexity and fit to data that plagues conventional statistics. A complex model fits training data well but requires a long description; a simple model is short but may leave much of the data unexplained. The MDL criterion finds the sweet spot by treating both model and data as strings to be compressed, and preferring the compression that is shortest overall.

The connection to Kolmogorov complexity is explicit: MDL approximates the uncomputable ideal of algorithmic probability by substituting practical compressors for the shortest program. Where Solomonoff induction requires enumerating all programs, MDL uses off-the-shelf compression algorithms or parametric coding schemes. The approximation is principled but lossy — MDL can fail when the compressor misses structural regularities that a universal computer would find.