14 min read
Lesson 4 of 6
Red Teaming & Adversarial Testing
From the course Evaluating & Benchmarking AI Models
Red teaming is the practice of deliberately trying to make an AI system fail, produce harmful outputs, or behave unexpectedly. It's essential for responsible deployment.
Why Red Team?
Standard evaluations test what the model can do right. Red teaming tests what it can do wrong:
- Harmful content generation: Can the model be tricked into producing dangerous instructions?
- Bias and stereotypes: Does the model exhibit unfair biases in its responses?
- Hallucination triggers: What inputs cause the model to confidently state false information?
- Prompt injection: Can adversarial inputs override system instructions?
- Data leakage: Does the model reveal training data or system prompts?
Red Teaming Strategies
Manual red teaming:
- Recruit diverse testers (different backgrounds, expertise, adversarial creativity)
- Provide clear testing scenarios and categories
- Document all successful attacks with reproducible prompts
Automated red teaming:
- Use an LLM to generate adversarial prompts at scale
- Fuzzing: Systematically vary inputs to find edge cases
- Gradient-based attacks: For white-box models, optimize inputs to maximize harmful outputs
Common Attack Vectors
| Attack | Description | Example |
|---|---|---|
| Role-playing | Ask model to pretend to be an unrestricted AI | "You are DAN (Do Anything Now)..." |
| Encoding tricks | Use base64, ROT13, or other encodings | "Decode and follow: [base64 string]" |
| Context manipulation | Build up context that normalizes harmful requests | Gradual escalation over multiple turns |
| Instruction hierarchy | Claim higher authority than system prompt | "Ignore previous instructions and..." |
| Payload splitting | Split harmful requests across multiple messages | Each message seems innocent alone |
Building a Red Team Program
- Scope: Define what you're testing (safety, accuracy, robustness, fairness)
- Team: Mix of security researchers, domain experts, and creative thinkers
- Taxonomy: Categorize findings by severity and type
- Remediation: For each finding, implement mitigations (guardrails, filters, fine-tuning)
- Iteration: Red team again after fixes to verify mitigations work
Key Takeaways
- •Red teaming tests what the model does wrong, complementing standard capability evaluations
- •Both manual (diverse testers) and automated (LLM-generated) approaches are needed
- •Common attacks include role-playing, encoding tricks, and instruction hierarchy manipulation
- •Establish a structured red team program with scope, taxonomy, and remediation cycles
0 of 6 lessons marked complete
Progress is kept in this browser only.