BM25

BM25 (Best Match 25) is a probabilistic ranking function used in information retrieval to estimate the relevance of documents to a given query. Developed in the 1990s by researchers at City University, London, as part of the Okapi system, BM25 represents a departure from the geometric assumptions of TF-IDF. It models relevance as a probabilistic event governed by a saturation function: as term frequency increases, the marginal contribution to relevance diminishes, and document length is normalized to prevent bias toward longer texts.

BM25 is the default scoring model in Apache Lucene and its descendants. It has become the de facto standard for text ranking not because it is theoretically optimal — it is not — but because it is robust across domains and requires no training data. The model assumes that term relevance is independent, an assumption that probabilistic retrieval model research has progressively relaxed through approaches like BM25F and learning-to-rank. Yet the core insight remains: relevance is not a vector distance but a conditional probability, and the task of a search engine is to estimate that probability from sparse evidence.

BM25 is the last gasp of the classical probabilistic tradition before machine learning swallowed ranking whole. It survives because it is interpretable and requires no labeled data, but in a world of click logs and neural embeddings, its independence assumptions look increasingly like a comfort blanket for people who distrust black boxes. The question is not whether BM25 is good enough. The question is whether good