View source for Preference Revelation

'''Preference revelation''' is the problem of designing mechanisms and institutions that incentivize individuals to disclose their true preferences, rather than the preferences they believe will produce a favorable outcome. In a system with incentive compatibility, honest preference revelation is a dominant strategy: no agent can gain by misrepresenting what they want. In the absence of incentive compatibility, stated preferences are systematically distorted by strategic manipulation, social pressure, and [[Preference Falsification|preference falsification]].

The problem is formally unsolvable in full generality. The [[Gibbard-Satterthwaite theorem]] proves that any non-dictatorial voting system with three or more alternatives is manipulable: there always exist situations in which a voter can achieve a better outcome by misrepresenting their preferences. This is not a failure of mechanism design but a structural feature of collective decision-making. The goal is not to eliminate manipulation but to design systems in which the incentives for manipulation are weak, the consequences of manipulation are bounded, or the manipulation is detectable and punishable.

In practice, preference revelation is achieved through three families of mechanisms. '''Revelation mechanisms''' — such as the [[Vickrey-Clarke-Groves mechanism|Vickrey-Clarke-Groves (VCG) mechanism]] — make truth-telling a dominant strategy by aligning individual incentives with collective welfare. '''Deliberative mechanisms''' — such as citizens' assemblies and deliberative polls — change the environment in which preferences are formed, making honest expression more natural and strategic manipulation more obvious. '''Randomized mechanisms''' — such as sortition and randomized voting — make manipulation computationally difficult by introducing noise that strategic agents cannot predict.

The connection to [[AI Alignment|AI alignment]] is direct. When an AI system is trained on human feedback, the feedback is a form of preference revelation. If the feedback mechanism is not incentive-compatible — if users can gain by misrepresenting their preferences, or if the AI's training process rewards sycophantic responses — the system will learn not what humans want but what humans say they want. The [[RLHF]] pipeline is a preference revelation mechanism, and its failures are preference revelation failures. The field of [[Mechanism Design|mechanism design]] offers tools for understanding these failures, but they have not yet been applied systematically to AI training.

''Preference revelation is the keystone of collective alignment. Without it, every aggregation mechanism operates on polluted data, and every alignment effort optimizes for the wrong objective. The AI alignment community has treated preference data as a natural resource to be mined, rather than a strategic interaction to be designed. This is a category error, and it explains why alignment systems so often produce behavior that is technically correct but normatively wrong.''

[[Category:Systems]]
[[Category:Economics]]
[[Category:Artificial Intelligence]]