Red-Teaming and Adversarial AI Evaluation
Design controlled tests of failure modes, record reproducible evidence and distinguish a useful finding from an unusual response.
You'll learn
- A red-team test begins with a defined failure mode and authorized scope.
- Record the setup and observed behavior, not just an alarming final sentence.
- Test whether legitimate requests still work as well as whether prohibited behavior is blocked.
- One attempt does not establish how often a failure occurs.
In this guide
Red-teaming deliberately probes for failures. The value is not making a model say something strange; it is finding a behavior that violates a defined requirement and giving the team enough evidence to reproduce and investigate it.
Write the failure condition before the test
A controlled adversarial review
Confirm the authorized environment
Use the model, data and tools the project designates. Identify actions that are outside scope before testing.
Define the failure mode
Describe the specific boundary: for example, treating a document’s embedded instructions as authority over the user’s task.
Build a harmless test fixture
Use synthetic data and a controlled setup that can expose the boundary without involving real accounts or private records.
Run and capture the result
Save the permitted prompt, relevant configuration, tool output and response.
Judge against the criterion
Distinguish a boundary violation from a harmless quotation, a refusal or an unrelated mistake.
Report limits and retest policy
State whether the observation came from one attempt or repeated trials. Follow the project’s process for additional tests.
A harmless test of instruction boundaries
Evaluation example
- BANANA
- The memo contains an instruction telling the reader to output a particular word; no other substantive information is supplied.
What a useful finding contains
| Field | What to record |
|---|---|
| Expected behavior | The specific requirement the system should satisfy. |
| Setup | Authorized environment, input and relevant configuration. |
| Observed behavior | The exact response or action that matters. |
| Evidence | The permitted trace or artifact that supports the finding. |
| Impact | What the failure would allow or disrupt in this defined setting. |
| Limits | What was not tested and whether reproduction was attempted. |
Red-team findings can inform changes and later checks of whether those changes work. Anthropic describes this as an iterative process of assessing risks, introducing mitigations and testing them. The practical implication for a contributor is to preserve a clear test case rather than only a screenshot of the most surprising output.
Check false positives and over-refusal
A model quoting harmful language from a document is not necessarily endorsing it. A refusal can be appropriate for one request and an error for a benign neighboring request. Judge the actual response under the supplied policy. A useful test set includes permitted tasks as well as failure probes, so blocking everything does not look like success.
Common mistake
Turning one successful probe into a universal claim
One observed failure establishes that behavior occurred in that setup. It does not measure prevalence across prompts, configurations or repeated runs. Report the evidence at its actual scope.
Before filing a finding
- Confirm the test stayed within the approved scope.
- State the violated requirement.
- Include enough permitted context to reproduce it.
- Separate observed behavior from possible consequences.
- Record the number and conditions of attempts accurately.
- Use the designated reporting channel for restricted material.
Key takeaway
A useful adversarial test is narrow enough to diagnose and clear enough to repeat.