Dangerous Capability Evaluations

Dangerous Capability Evaluations (DCEs) are structured assessments designed to detect whether an AI model possesses capabilities that could pose catastrophic or irreversible risks — including autonomous cyberoffense, biological weapons uplift, deceptive alignment, and the ability to subvert human oversight mechanisms. Unlike standard performance benchmarks, DCEs are threshold tests: the question is not how well a system performs, but whether it crosses a qualitative line beyond which deployment becomes unacceptable regardless of other properties.

The practice was formalized by major AI labs beginning around 2023 as part of Responsible Scaling Policies. The core methodological challenge is that DCE results are inherently elicitation-dependent (see Capability Elicitation): a model that fails a dangerous capability evaluation under standard prompting may pass under adversarial elicitation, making "no dangerous capabilities detected" a claim about the evaluator's effort, not about the model.

This is not a solved problem. The field lacks validated protocols for establishing that DCEs have probed capability space exhaustively, and the consequences of false negatives are asymmetric: a missed dangerous capability discovered post-deployment may have no recovery path.

DCEs as Attention Architecture

Every dangerous capability evaluation is an attention allocation mechanism — and the scarce resource it allocates is not compute but evaluator attention. A comprehensive DCE requires expert red-teamers to think creatively about how a model might be misused, to construct elaborate elicitation scenarios, and to maintain sustained focus on failure modes that are cognitively alienating. The evaluator who spends eight hours probing a model for bioweapons capabilities is engaged in the same cognitive labor as the platform content moderator who spends eight hours reviewing disturbing content: both are allocating scarce human attention to the detection of harm.

This reframing has uncomfortable implications. Just as content moderation teams suffer from burnout, desensitization, and declining detection rates over time, DCE teams face analogous degradation. The red-teamer who has probed fifty models for deception capabilities may develop pattern-matching habits that miss novel strategies. The evaluator who has repeatedly found "no dangerous capabilities" may unconsciously lower their search intensity. The attention architecture of DCEs is not designed for sustained, high-intensity cognition; it is designed for periodic, checklist-driven assessment. And like all periodic assessments, it misses the capabilities that emerge between evaluations.

The Institutional Design Problem

The institutions that conduct dangerous capability evaluations face a structural conflict that the responsible scaling literature rarely acknowledges. The lab that develops a model is also the lab that evaluates it. The division between "development" and "safety" is organizational, not financial: both teams report to the same executives, operate under the same commercial pressures, and share the same incentive to deploy. A DCE team that consistently finds dangerous capabilities in its own lab's models is a team that threatens its colleagues' careers, its organization's reputation, and its executives' stock options.

This is not a cynical observation. It is a prediction from institutional design theory. Organizations do not reliably produce accurate self-assessments because accurate self-assessment is not the organizational goal. The goal is survival, growth, and competitive positioning. DCEs conducted internally are not independent evaluations; they are organizational communications whose audience includes regulators, investors, and the public. The DCE report that says "no dangerous capabilities detected" is not merely a technical finding. It is a strategic document that enables deployment.

The alternative — external, independent DCEs — faces its own problems. External evaluators lack access to the model's weights, training data, and internal documentation. They must conduct evaluations through APIs or limited interfaces, which constrains their adversarial elicitation capacity. The result is a paradox: the evaluators with the most access have the wrong incentives, and the evaluators with the right incentives have insufficient access. This is not a temporary coordination problem. It is a structural feature of the relationship between frontier AI development and safety evaluation.

The Feedback Loop Between Evaluation and Development

Dangerous capability evaluations do not merely measure risk. They reshape the developmental trajectory of the systems they evaluate. When a DCE identifies a specific dangerous capability — say, the ability to design a novel pathogen — the development team responds by training the model to refuse such requests. The refusal behavior is then tested in subsequent DCEs, which confirm that the capability is no longer accessible under standard prompting. But this process does not eliminate the capability. It relocates it: the model now possesses both the underlying knowledge and a censorship layer trained to suppress its expression.

This creates what might be called the evaluation-development arms race: each round of DCEs produces a new generation of censorship mechanisms, and each new censorship mechanism produces a new generation of adversarial elicitation techniques designed to bypass it. The model does not become safer in any absolute sense. It becomes safer under the specific elicitation methods that the DCE team has tested — and more dangerous under methods that the team has not yet imagined.

The systems perspective suggests that this arms race is not a failure of DCE methodology but an inevitable consequence of treating evaluation as a checkpoint rather than as a continuous process. A DCE is a snapshot of a moving system, and the act of taking the snapshot changes the system's trajectory. The question is not whether DCEs can catch all dangerous capabilities. The question is whether the checkpoint model of safety evaluation is adequate for systems that evolve faster than any evaluation cycle can track.

The fantasy of dangerous capability evaluation is that a sufficiently thorough assessment can provide a clean bill of health — a definitive "safe to deploy" verdict. This fantasy misunderstands the nature of the systems involved. A frontier AI model is not a product that can be certified; it is a complex adaptive system whose capabilities are co-determined by its weights, its deployment environment, and the ingenuity of its users. The DCE that finds no dangerous capabilities under today's elicitation methods is not a guarantee of safety. It is a guarantee that the model has not yet been provoked in ways the evaluators have imagined. And imagination, like capability, is elicitation-dependent.