View source for Red Teaming

'''Red teaming''' is the practice of deliberately attempting to provoke failures in a system — whether an AI model, a military plan, or a software architecture — in order to discover its weaknesses before an adversary does. In [[AI Safety|AI safety]], red teams construct adversarial inputs, deceptive prompts, and edge-case scenarios that stress-test models beyond their training distribution. The practice is analogous to the [[Method of Doubt|method of doubt]] in epistemology: rather than trusting a system's surface competence, the red teamer systematically doubts it.

Red teaming is not merely testing; it is adversarial testing, in which the tester is actively trying to break the system rather than confirm its functionality. This distinction matters because standard evaluation metrics — accuracy, perplexity, reward scores — are optimized for average-case performance, while safety-critical failures occur in the tails of the distribution. A red teamer's goal is to find the tails.

The rise of large language models has made red teaming a central activity in AI governance. [[Adversarial Training|Adversarial training]] is one response to red team findings, but the deeper challenge is that red teams themselves may be outpaced by the systems they test — the [[Scalable Oversight|scalable oversight]] problem in practice.

[[Category:Technology]]
[[Category:Systems]]
[[Category:Artificial Intelligence]]