AI Model Wiki

    Skip to main content
    18 min
    Lesson 1 of 4

    AI Threat Landscape

    From the course Enterprise AI Security

    Enterprise AI systems face a unique threat landscape that extends beyond traditional cybersecurity into AI-specific attack vectors.

    AI-Specific Threats

    Training Phase Attacks

    • Data Poisoning: Injecting malicious training data to corrupt model behavior
    • Backdoor Attacks: Hidden triggers that cause misclassification
    • Model Stealing: Extracting model behavior through API queries
    • Training Data Extraction: Recovering private training data from models

    Inference Phase Attacks

    • Adversarial Examples: Carefully crafted inputs that fool models
    • Prompt Injection: Manipulating LLM behavior through malicious inputs
    • Jailbreaking: Bypassing safety filters in LLMs
    • Model Inversion: Reconstructing training data from model outputs
    • Membership Inference: Determining if specific data was in training set

    Example Threats by AI System Type

    LLMs:
    - Prompt injection via user-controlled content
    - Indirect injection through retrieved documents (RAG)
    - Jailbreaking safety filters
    - System prompt extraction
    
    Computer Vision:
    - Adversarial patches on physical objects
    - Digital perturbations undetectable to humans
    - Physical-world attacks (fooling autonomous vehicles)
    
    Recommendation Systems:
    - Data poisoning to promote specific items
    - Profile injection attacks
    - Shilling attacks by competing merchants
    

    OWASP Top 10 for LLMs

    1. Prompt Injection
    2. Insecure Output Handling
    3. Training Data Poisoning
    4. Model Denial of Service
    5. Supply Chain Vulnerabilities
    6. Sensitive Information Disclosure
    7. Insecure Plugin Design
    8. Excessive Agency
    9. Overreliance
    10. Model Theft

    Key Takeaways

    • •AI threats span both training (poisoning, backdoors) and inference (adversarial, injection) phases
    • •LLMs face unique threats including prompt injection and indirect injection through RAG
    • •OWASP Top 10 for LLMs provides a comprehensive threat taxonomy
    • •Model theft via API queries can extract proprietary model behavior without access to weights
    0 of 4 lessons marked complete

    Progress is kept in this browser only.