Relevance scoring

Relevance scoring is the algorithmic assignment of numerical scores to documents in response to a query, encoding a statistical model of what human judges would consider pertinent. In information retrieval, relevance is not an intrinsic property of a document but a relational property between a query, a document, and a corpus — a point that distinguishes search from database lookup. The dominant frameworks are the vector space model (geometric similarity between term-weighted vectors) and the probabilistic BM25 model (term frequency saturation with document length normalization), though modern systems increasingly incorporate machine-learned ranking signals, user behavior data, and semantic embeddings.

The epistemological problem of relevance scoring is that ground truth is irreducibly subjective. Different judges disagree on whether a document is relevant; the same judge may disagree with themselves across time. Scoring functions therefore model not 'true relevance' but the central tendency of human judgment, and they inherit all the biases of the judges whose assessments were used to train or validate them. This has become a critical concern as searchable analytics platforms deploy relevance models in high-stakes domains like legal discovery, medical literature search, and recruitment filtering.

The claim that relevance scoring is a solved problem — that BM25 and its variants are 'good enough' — ignores the domain-specificity of relevance. A scoring function optimized for web search performs poorly on patent search, and one optimized for e-commerce fails on academic literature. The illusion of a universal relevance model is a holdover from the early days of information retrieval, when generic test collections like Cranfield and TREC were treated as representative of all search tasks. They are not. Relevance is context-dependent all the way down.