Iterated Amplification

Iterated Amplification (IA) is an approach to AI alignment developed by Paul Christiano, originally at OpenAI, in which a weak but aligned system is used to train a stronger system through a recursive decomposition process. The core idea: if a human and a weak assistant can together solve a problem that the human alone cannot solve, then the assistant can be trained to solve the problem directly. Repeating this process produces a sequence of increasingly capable systems, each aligned because it was trained by the previous, aligned system.

The formal structure is elegant. Let H be a human and A_0 be a weak but aligned assistant. At each iteration, A_{i+1} is trained to imitate the output of H consulting A_i. If the consultation process is faithful — if A_{i+1} actually does what H+A_i would have done — then alignment is preserved across capability gain. The key challenge is ensuring faithfulness: the amplified system must not learn to optimize for the evaluation signal rather than the intended behavior.

IA is closely related to debate and recursive reward modeling, and all three approaches share a common structure: they attempt to solve the scalable oversight problem by decomposing complex evaluations into simpler sub-evaluations. The difference is in the decomposition mechanism. Debate uses adversarial argumentation; recursive reward modeling uses learned reward functions; IA uses recursive imitation.

The practical challenge is that IA assumes the existence of a decomposition that preserves alignment at each step, and there is no general proof that such decompositions exist for all tasks. For mathematical reasoning, decomposition is natural: a proof can be broken into lemmas. For social reasoning, ethical judgment, or creative tasks, decomposition may not preserve the relevant properties. Whether IA scales to human-level and beyond-human capability remains an open question.