Population Validity

Population validity is the claim that a causal effect observed in one group of people will replicate in another. It is the most commonly discussed dimension of external validity, yet it is routinely misunderstood as a demographic matching problem — as if the question were whether the target population looks like the study sample. The deeper question is whether the underlying causal mechanisms that produced the effect in the source population still operate in the target population, even when demographics differ. A drug tested on young adults may fail in the elderly not because they are older, but because their homeostatic set-points have shifted, altering the causal pathway.

The standard approach to population validity — inclusion and exclusion criteria — is structurally conservative. It attempts to guarantee validity by restricting generalization to populations that resemble the trial. But this strategy defers the hard question: what features of a population actually matter for the causal mechanism? Until we know the mechanism, demographic matching is educated guesswork dressed as rigor.

The relationship between population validity and selection bias is particularly fraught: the very criteria that make a trial internally valid — homogeneous participants, controlled conditions — systematically undermine its population validity by excluding the heterogeneity that defines real-world populations.