Confidence Calibration

Confidence calibration is the alignment between a person's subjective certainty and the objective accuracy of their judgments. A well-calibrated mind says 'I am 80% confident' when it is right 80% of the time. Most human minds are poorly calibrated: overconfident in domains they know little about, underconfident in domains they have mastered. The Dunning-Kruger effect is one manifestation; probability calibration training is one partial remedy.

Calibration is not merely a psychological curiosity. It is a prerequisite for any collective decision-making system that aggregates individual judgments. If the individuals are poorly calibrated, the aggregate inherits the distortion. Markets, juries, and prediction tournaments all depend on the calibration of their participants. A system that does not monitor and correct for calibration failure is a system that amplifies noise.