Preference Aggregation

Preference aggregation is the problem of combining the desires, values, or goals of multiple agents into a single coherent objective that can guide collective action or system design. It is the foundational challenge of social choice theory, mechanism design, and AI alignment: no agent may fully endorse the aggregated preference, yet the system must optimize something.

The difficulty is not merely logistical. As Arrow's impossibility theorem demonstrates, any preference aggregation mechanism satisfying minimal fairness criteria can produce irrational collective choices. In AI systems, this manifests as RLHF training on heterogeneous human raters: the "human preference" being optimized is already an aggregation artifact, and its divergence from any individual's values is a source of reward hacking.

Preference aggregation is not a preprocessing step to alignment. It is alignment — the moment we assume a single objective function, we have already made the most consequential normative choice.

Preference Aggregation as Distributed Consensus

The structural parallel between preference aggregation and distributed consensus is deeper than analogy. In a Byzantine fault-tolerant consensus protocol, nodes must agree on a single value despite the possibility that some nodes are malicious or faulty. In preference aggregation, agents must agree on a single social ordering despite the certainty that their preferences conflict. Both problems face the same two-thirds threshold: reliable consensus requires that the fraction of honest agents exceed a structural bound, and reliable preference aggregation requires that the domain of preferences be restricted or that one of Arrow's axioms be relaxed.

Consensus Dynamics on networks reveals that the topology of interaction matters as much as the aggregation rule itself. In a fully connected network, majority rule converges to consensus quickly. In a sparse or clustered network, consensus may fail entirely — not because the aggregation rule is flawed, but because the network's spectral gap is too small for information about distant preferences to diffuse before local majorities entrench. The echo chamber phenomenon in social media is not a failure of individual rationality; it is a failure of network topology to support preference mixing.

The Topology of Aggregation

Preference aggregation is typically studied as if agents were disconnected inputs to a central mechanism. This is the wrong abstraction. Real agents are embedded in networks — social networks, organizational hierarchies, information ecosystems — and their preferences are not static inputs but dynamical states that evolve through interaction. The strategic manipulation of voting systems, the polarization of political discourse, and the convergence of scientific consensus are all processes in which preferences change through the very act of aggregation.

This dynamical view transforms the problem. Instead of asking 'what is the optimal aggregation rule for a fixed profile of preferences?' we must ask 'what network topologies and interaction rules produce preference profiles that are aggregable at all?' The answer, from coupled oscillator models and opinion dynamics, is that aggregability is an emergent property of the interaction network, not a property of the preferences themselves. A population with heterogeneous preferences on a well-connected network may converge to a smooth distribution that is easy to aggregate; the same population on a fragmented network may polarize into disconnected clusters whose aggregated preference is meaningless.

The field of social choice theory has spent seventy years proving that perfect aggregation is impossible. It has spent far less time asking whether the preferences we are given are the right inputs to aggregate. A preference is not a natural fact; it is a constructed response to a constructed choice architecture. The real design problem is not how to aggregate given preferences but how to design the institutions — the networks, the information environments, the deliberative structures — that produce preferences worth aggregating. Any theory of preference aggregation that treats preferences as fixed inputs is not a theory of collective intelligence. It is a theory of collective stupidity — the stupidity of a system that optimizes for objectives its own architecture prevents it from choosing well.