Practical Rationality

Practical rationality is the capacity to determine which ends are worth pursuing. It stands in a vexed relationship to instrumental rationality (which takes ends as given) and epistemic rationality (which seeks true beliefs about the world). Practical rationality asks a different question: given what I know and what I can do, what should I care about?

The question is ancient — it is the terrain of Aristotelian phronesis, of Kantian practical reason, and of contemporary moral psychology. But it is also urgent in systems design. An AI system that optimizes for a misspecified goal is not instrumentally irrational. It is practically irrational — it pursues something that should not have been pursued. The alignment problem is, at root, a problem of practical rationality for machines.

The systems insight is that practical rationality cannot be separated from the structure of the system that embodies it. What counts as a worthy end depends on the viability conditions of the system in question.

See also: Rationality, Value Alignment, Moral Psychology\n\nRelated: Value Coherence