Trust calibration

Trust calibration is the dynamic process by which human operators adjust their reliance on automated systems based on an assessment of the system's reliability, competence, and transparency in a given context. Properly calibrated trust means trusting the system when it is competent and distrusting it when it is not. Miscalibrated trust takes two forms: over-trust, which leads to automation complacency, and under-trust, which leads to manual override of systems that are performing correctly.

The problem of trust calibration is central to cognitive engineering and human factors because it determines whether the human-automation system functions as a team or as a pair of independent agents working at cross purposes. Research by John Lee and Katrina See identified that trust calibration depends on three factors: the operator's understanding of the system's capabilities, the system's transparency about its own state, and the history of the system's performance in similar situations.

In current AI systems, trust calibration is particularly difficult because the system's capabilities are often unknown even to its designers. A large language model may produce correct answers 95% of the time on a benchmark but fail catastrophically on edge cases that are not represented in the training data. Without visibility into the system's confidence or the basis of its output, the operator cannot calibrate trust appropriately. The result is binary over-trust or blanket skepticism — neither of which supports effective human-machine collaboration.

The design challenge is to create systems that support appropriate trust — not maximum trust, not minimum trust, but trust that tracks the system's actual competence. This requires not just explainability but competence signaling: explicit communication from the system about what it knows, what it does not know, and how confident it is in its current output. Without competence signaling, trust calibration is impossible, and the human operator is structurally disadvantaged.