Multiple Regression

Multiple regression is a statistical method for modeling the relationship between a dependent variable and multiple independent variables simultaneously. It generalizes simple linear regression to settings where multiple predictors may jointly influence an outcome, and it remains one of the most widely used tools in social science, economics, epidemiology, and machine learning.

The model takes the form Y = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ + ε, where Y is the outcome, the Xs are predictors, the βs are coefficients estimated from data, and ε is an error term assumed to be normally distributed with mean zero. The coefficients represent the expected change in Y associated with a one-unit change in each predictor, holding all other predictors constant — the ceteris paribus interpretation that makes regression a tool for quasi-experimental inference.

The limitations of multiple regression are as instructive as its uses. When predictors are correlated (multicollinearity), coefficient estimates become unstable and interpretation ambiguous. When the model is misspecified — for example, when important variables are omitted or when the true relationship is nonlinear — the estimated coefficients can be systematically biased. When the sample is small relative to the number of predictors, the model overfits: it captures noise in the training data as if it were signal, producing predictions that fail on new data.

The overfitting problem connects directly to the less-is-more effect in decision making. A multiple regression model with many predictors may have lower bias than a simple heuristic like take-the-best, but its higher variance — its sensitivity to sampling error — can make it less accurate in prediction. The choice between complex integration and simple heuristics is not a matter of principle. It is an empirical question about the structure of the environment: how many cues matter, how correlated they are, and how much data is available.