Tokenization

Tokenization is the process of converting a continuous stream of information into discrete units — tokens — that can be processed by computational systems. The term is used across multiple domains with different technical specifics but a shared underlying logic: segmentation as a prerequisite for structured processing. In natural language processing, tokens are typically words, subwords, or characters; in compiler design, tokens are lexical categories like identifiers, operators, and literals; in blockchain systems, tokenization refers to the representation of real-world assets as digital tokens on a distributed ledger.

The modern prominence of tokenization stems from its role in neural network-based language models. Models like GPT, BERT, and their descendants do not process raw text directly; they process sequences of integer token IDs, each corresponding to an entry in a learned embedding matrix. The choice of tokenization scheme — word-level, character-level, or subword-level — profoundly affects model behavior, vocabulary size, and the handling of out-of-vocabulary words. The dominant subword approaches — Byte pair encoding (BPE), WordPiece, and SentencePiece — iteratively merge frequent character sequences into tokens, producing vocabularies that balance expressiveness with compactness.

This segmentation, however, is not innocent. Tokenization imposes a discrete structure on continuous phenomena — the acoustic stream of speech, the visual flow of text, the semantic gradients of meaning — and that imposition has consequences. Models trained on tokenized text reason about the world through the lens of their token vocabulary, and that vocabulary reflects the biases of the training data and the tokenization algorithm. A model tokenized with BPE on English Wikipedia will have a different conceptual granularity than one tokenized on Chinese social media. The claim that tokenization is a mere