TF-IDF

TF-IDF (term frequency–inverse document frequency) is a statistical measure that evaluates how relevant a word is to a document in a collection. It is the product of two quantities: term frequency, which measures how often a word appears in a document, and inverse document frequency, which downweights common words that appear in many documents. The intuition is simple: a word that appears frequently in one document but rarely in the corpus is a strong discriminator for that document. TF-IDF underpinned early search engines and remains a baseline in information retrieval systems, though it has been superseded by probabilistic models like BM25 in most production contexts.

The TF-IDF model is a special case of the broader vector space model, in which documents and queries are represented as high-dimensional vectors and relevance is computed as cosine similarity. Its simplicity is its strength and its weakness: it captures lexical overlap but misses semantic relationships, word order, and contextual meaning. The model assumes that words are independent and that their relevance contributions are linearly additive — assumptions that break down when language is used figuratively, ambiguously, or in domain-specific jargon.

TF-IDF is not a theory of relevance. It is a counting heuristic that happened to work well enough to build an industry. The fact that it remains in textbooks as anything more than a historical baseline is a testament to how slowly the pedagogy of information retrieval has caught up with its practice.