Red-Teaming and Stress-Testing

Standard evaluation runs a system through a fixed test set and reports accuracy. Red-teaming takes a different approach: it sends humans — often adversarial, creative, motivated humans — to actively search for failure modes the developers did not anticipate. The name comes from military exercises where a 'red team' plays the adversary to stress-test plans and defenses. In AI safety, red-teaming is the practice of trying to break a system before it is deployed, so that failures are found in a controlled setting rather than in the real world.

What Red-Teaming Looks For

Red-teamers for AI systems are looking for behaviors the system should not exhibit — and for conditions that elicit those behaviors. For language models, the most commonly probed failure modes include: generating instructions for creating weapons, synthesizing dangerous chemicals, or planning violence; producing content that sexualizes minors; generating targeted harassment or credible threats against real people; spreading clearly false information presented as fact; and being manipulated through prompt injection or jailbreak techniques into ignoring safety constraints. For vision systems and autonomous agents, red-teamers probe for: adversarial input vulnerabilities; failure modes under unusual lighting, weather, or sensor conditions; unexpected behaviors at distribution boundaries; and interactions between components that produce emergent failures no single component exhibits. For all AI systems, red-teamers also look for: subtle biases that aggregate evaluation misses; edge cases in real-world deployment contexts that developers did not consider; failure modes that only appear after extended interaction; and ways the system can be abused for purposes other than its intended use.

Red-Teaming Methodologies

Red-teaming is not purely improvised — it has developed structured methodologies. Threat modeling begins red-teaming by asking: who might attack this system, with what capabilities, and what is their motivation? A model used for customer service has different threats than a model embedded in a national security context. Threat modeling shapes which attack surfaces red-teamers prioritize. Attack trees decompose threats hierarchically. The root of the tree is an outcome the attacker wants (e.g., 'extract training data'). Branches are alternative methods (inference attacks, membership inference, model inversion). Leaves are specific technical actions. Attack trees make coverage explicit — red-teamers can see which branches they have and have not tested. Jailbreak taxonomy: for language models, researchers have classified jailbreak techniques into categories: role-playing (asking the model to pretend to be a different AI with no restrictions), hypothetical framing (asking the model to describe what a character in a story would do), encoding tricks (asking for information in base64 or other encoding to bypass filters), and multi-step decomposition (breaking a harmful request into individually harmless sub-requests). Taxonomies help red-teams ensure systematic coverage. Automated red-teaming augments human red-teamers with AI systems trained to generate adversarial prompts. Language models fine-tuned to find failure modes can explore a much larger space of inputs than human teams working alone, though they require human judgment to evaluate the outputs. After each red-teaming exercise, findings are documented in a structured format: the attack, the behavior observed, the severity, the conditions required, and recommended mitigations. This documentation feeds directly into the next training or fine-tuning cycle.

Limits of Red-Teaming

Red-teaming is valuable, but it has irreducible limitations that safety practitioners must understand. Coverage is finite. The space of possible inputs to a language model is essentially infinite. Red-teams can only sample from it. A system can pass a red-teaming exercise and still exhibit failures on inputs the team did not try. The absence of found failures is not proof of safety. Red teams reflect their own demographics and assumptions. A red team that lacks relevant expertise — domain knowledge about weapons synthesis, or cultural knowledge about contexts where a system will be deployed — will miss the attack surfaces those experts would find immediately. Diverse red teams find more diverse failure modes. Findings date quickly. A jailbreak discovered and mitigated during red-teaming may be reinvented by users after deployment. Models updated to fix one vulnerability may develop different ones. Red-teaming is not a one-time exercise but a continuous practice. High-stakes rare events are hard to probe. A model that refuses harmful requests 99.9% of the time will still produce them if exposed to a billion queries. Red-teaming at small scale will not reliably find failure modes that occur with very low probability — but those modes may be catastrophic when they do occur.