Analysis of variance
Analysis of variance (ANOVA) is a collection of statistical methods for partitioning the observed variation in a dataset into components attributable to different sources. Developed by Ronald Fisher in the 1920s, ANOVA provides a framework for testing whether the means of several groups differ significantly, and for quantifying how much of the total variation is explained by group membership versus residual noise.
The Core Idea
The fundamental decomposition in ANOVA is:
Total Variation = Between-Group Variation + Within-Group Variation
If the between-group variation is large relative to the within-group variation, the group means are likely different in the population, not merely by sampling accident. The F-statistic quantifies this ratio and permits a formal hypothesis test.
One-Way ANOVA
The simplest case compares means across k groups. The null hypothesis is that all group means are equal. The alternative is that at least one mean differs. The test computes:
- Sum of Squares Between (SSB) — variation of group means around the grand mean, weighted by group size
- Sum of Squares Within (SSW) — variation of individual observations around their group means
- Mean Square Between (MSB) = SSB / (k - 1)
- Mean Square Within (MSW) = SSW / (N - k)
- F = MSB / MSW
Under the null hypothesis, F follows an F-distribution with (k-1, N-k) degrees of freedom.
Extensions
Two-way ANOVA examines the effects of two categorical factors and their interaction on a continuous outcome. It partitions variation into: factor A, factor B, A×B interaction, and residual.
Repeated measures ANOVA handles designs where the same subjects are measured under multiple conditions, partitioning out subject-level variation to increase power.
MANOVA (Multivariate ANOVA) extends the framework to multiple dependent variables simultaneously.
ANCOVA (Analysis of Covariance) blends ANOVA with regression by including continuous covariates that explain variation not captured by the categorical factors.
Assumptions and Robustness
ANOVA assumes: independence of observations, normally distributed residuals within each group, and homogeneity of variance across groups (homoscedasticity). The F-test is robust to moderate violations of normality when sample sizes are equal and not too small. Violations of homoscedasticity are more serious; Welch's ANOVA or transformations may be needed.
Relationship to Regression
ANOVA is mathematically equivalent to linear regression with categorical predictors coded as dummy variables. The F-test in ANOVA is the same as the F-test for the overall significance of the regression model. This equivalence, noted by Fisher and developed by others, unifies what appear to be distinct statistical methodologies.
Historical Significance
Fisher developed ANOVA in the context of agricultural field trials at Rothamsted Experimental Station, where he needed to compare crop yields across different treatments while accounting for field heterogeneity. The methods were rapidly adopted across biology, psychology, medicine, and the social sciences, becoming the dominant framework for experimental analysis through much of the 20th century.
The rise of generalized linear models, mixed models, and Bayesian alternatives has reduced ANOVA's dominance, but the decomposition of variation remains a foundational statistical idea that appears in machine learning (variance decomposition in ensemble methods), physics (partition of energy), and systems theory (structural decomposition of system behavior).