Inductive Bias

Inductive bias is the set of assumptions that a learning algorithm uses to predict outputs for inputs it has never encountered. Without inductive bias, no learning is possible: an algorithm that makes no assumptions about the structure of the target function can justify any prediction whatsoever. The bias is not a flaw to be eliminated but a design choice that determines which problems the system can solve efficiently and which it will fail at entirely. Different architectures encode different biases — locality in CNNs, sequential dependence in RNNs, pairwise interactions in transformers — and the match between bias and problem structure is the primary determinant of success. The field's chronic under-theorization of inductive bias is why no free lunch theorems keep surprising practitioners who assumed their favorite algorithm was universally powerful.

Inductive Bias as a Systems Property

Inductive bias is rarely discussed as a systems-level phenomenon, but it operates at scales far larger than individual algorithms. A scientific community has an inductive bias: it prefers certain classes of explanations (mechanistic, reductive, formal) over others (holistic, historical, phenomenological). An institution has an inductive bias: it learns from success stories more readily than from failures, producing a skewed training set that reinforces existing strategies. Even biological evolution has an inductive bias encoded in the genotype-phenotype map: the space of viable mutations is not uniformly sampled; developmental constraints bias the search toward certain phenotypic regions and away from others.

This reframing connects machine learning to epistemology and institutional design. The no free lunch theorem is not merely a mathematical result about algorithms. It is a general claim about any system that learns from finite experience: without bias, there is no generalization. The question is never whether to have bias, but which bias to have — and whether the bias is chosen deliberately or inherited unexamined from historical path dependence.

The Architecture-Bias Correspondence

Different learning architectures encode different assumptions about the geometry of the target function:

CNNs encode locality and translation invariance: features useful in one part of an image are assumed useful elsewhere. This bias is catastrophically wrong for problems where spatial location carries information (e.g., medical imaging where organ position matters), and brilliantly right for natural images where object identity is location-independent.

Transformers encode pairwise attention: every token can attend to every other token, with importance weighted by learned similarity. This bias assumes that long-range dependencies are as accessible as local ones, and that the relevant structure is relational rather than sequential. It fails for tasks where strict ordering is causal (e.g., certain reasoning chains) and succeeds for tasks where meaning is compositional across distant contexts.

GNNs encode permutation invariance: the learned function should not depend on how nodes are numbered. This is correct for molecular properties (atom indices are arbitrary) and wrong for routing problems (node IDs encode spatial or temporal structure).

The pattern is clear: each architecture is a bet on the structure of the world. The bet pays off when the world cooperates and bankrupts the model when it does not. The field's repeated surprise — that transformers work on images, that CNNs fail on text — is not evidence of algorithmic genius. It is evidence that we do not yet have a principled theory of which bias matches which problem.

Beyond Architecture: Data Bias and Temporal Drift

A subtler form of inductive bias operates not in the model's weights but in the data's history. Training datasets encode the biases of the collection process: which examples were easy to label, which populations were accessible, which features were cheap to measure. A model trained on historical hiring data inherits the institution's historical biases not because the algorithm is flawed but because the data is a biased sample of the target distribution.

Temporal drift compounds the problem. In multi-agent systems, the environment that generated yesterday's training data is not the environment that will generate tomorrow's test data — because other agents are learning simultaneously. The inductive bias that the