7 min read
Lesson 1 of 5
The AI Threat Landscape
From the course AI Security & Red Teaming
As AI systems are deployed in critical applications — healthcare, finance, legal, military — understanding how they can be attacked becomes essential. AI security is a rapidly evolving field with unique challenges that traditional cybersecurity doesn't address.
Why AI Security Is Different
Traditional software has deterministic behavior: given the same input, you get the same output. AI models are probabilistic and opaque. This creates novel attack surfaces:
- Model behavior is unpredictable — Small input changes can cause wildly different outputs
- Training data shapes behavior — Poisoning training data can embed hidden backdoors
- Natural language interfaces — Text-based interactions create new injection vectors
- Emergent capabilities — Models may develop unexpected behaviors not intended by designers
- Supply chain risks — Pre-trained models, fine-tuning datasets, and plugins all introduce trust boundaries
Categories of AI Attacks
1. Input Attacks (Inference Time)
- Prompt injection — Manipulating AI through crafted inputs
- Adversarial examples — Inputs designed to fool classifiers
- Jailbreaking — Bypassing safety filters and guardrails
2. Training Attacks
- Data poisoning — Corrupting training data to embed backdoors
- Model backdoors — Hidden triggers that activate specific behaviors
- Training data extraction — Recovering private data from model weights
3. Infrastructure Attacks
- Model theft — Extracting model weights through API queries
- Side-channel attacks — Inferring model architecture from timing or resource usage
- Supply chain compromise — Tampering with model files, libraries, or dependencies
4. Social Engineering
- Manipulating AI agents to take unauthorized actions
- Using AI-generated content for phishing and impersonation
- Exploiting trust in AI outputs to spread misinformation
The Red Team Mindset
AI red teaming means thinking like an attacker:
- What does this system assume about its inputs?
- Where are the trust boundaries?
- What would happen if the AI did the worst possible thing?
- How can we make the system fail safely?
Key Takeaways
- •AI systems have unique attack surfaces not covered by traditional cybersecurity
- •Attacks target inputs (prompt injection), training (data poisoning), and infrastructure (model theft)
- •Probabilistic behavior and opaque internals make AI harder to secure than traditional software
- •Red teaming means systematically thinking about how AI can be attacked and fail
- •Understanding the threat landscape is prerequisite to building defenses
0 of 5 lessons marked complete
Progress is kept in this browser only.