Effect size

Effect size is the magnitude of a relationship, difference, or phenomenon, measured in units that are independent of sample size. Unlike statistical significance, which asks whether an effect exists, effect size asks how large the effect is. The distinction is not merely technical. It is conceptual: statistical significance is a property of data collection (did we collect enough data to detect this?), while effect size is a property of the world (how strong is this relationship?). Conflating the two — the p-value fallacy — has produced a literature in which trivial effects are celebrated as discoveries and meaningful effects are dismissed as noise.

The most common measures include Cohen's d (standardized mean difference), Pearson's r (correlation coefficient), odds ratios, and risk ratios. Each has different properties and different vulnerabilities. Standardized measures like Cohen's d facilitate comparison across studies but obscure the raw magnitude of effects in units that matter to decision-makers. A drug with Cohen's d = 0.3 may be clinically meaningless or transformative, depending on the outcome measured and the baseline risk. Effect size is not a substitute for domain knowledge; it is a complement to it.

Effect Size Heterogeneity as a Systems Phenomenon

In meta-analysis, effect sizes vary across studies. This variation is typically attributed to methodological differences — sampling error, measurement heterogeneity, publication bias — but a deeper source is often ignored: the effect itself is context-dependent. A therapy that works in one healthcare system may fail in another not because of study quality but because the slow-scale structures — institutional capacity, cultural norms, complementary resources — differ. This is the cross-scale problem of population validity applied to effect sizes: the effect measured is always a joint product of the intervention and the context that hosts it.

Treating effect size heterogeneity as noise to be averaged away — the standard meta-analytic approach — assumes that the true effect is a single number and that variation around it is error. But if effects are genuinely context-dependent, then heterogeneity is signal, not noise. The variation tells us something about the conditions under which the effect operates. A random-effects model that estimates a distribution of effects is conceptually closer to the truth than a fixed-effect model that assumes a single true effect, but even the random-effects framework treats the distribution as a statistical artifact rather than a map of contextual variation.

Effect size is the most important number in science that nobody reads. Journals report p-values in abstracts and bury effect sizes in tables. Grant proposals boast of significance and omit magnitude. This is not an oversight. It is a structural bias toward discoverability over importance — toward finding things rather than understanding whether the things found matter.