Peer Review

Peer review is the process by which scientific manuscripts are evaluated by domain experts before publication — nominally a quality filter, structurally a feedback mechanism between the scientific community and its own outputs. Whether it functions as an effective feedback loop is, empirically, contested.

The mechanism is designed to catch errors, prevent the publication of false or misleading results, and enforce methodological standards. The evidence suggests it accomplishes these goals inconsistently. Peer review detects some statistical errors and methodological weaknesses, but misses others at rates that should be disqualifying for any safety-critical application. The replication crisis in psychology, medicine, and social science is partly attributable to peer review's failure to filter out underpowered studies, p-hacking, and unreported multiple comparisons.

The structural problem is that peer review is a delayed feedback loop operating on a signal that is systematically biased by publication bias. Reviewers evaluate manuscripts, not research programs; they assess internal consistency, not representativeness of findings; and they are drawn from the same community that has professional incentives to publish the kind of results under review. The loop feeds back only on what is submitted — and what is submitted is not a representative sample of what is true.

That peer review is better than no review is not an argument that peer review is sufficient. The relevant comparison is not 'peer review versus chaos' but 'peer review versus the evidential standards we actually need to trust scientific conclusions at scale.' By that standard, peer review is a near-miss — close enough to real quality control that we act as if it were the thing itself.