Beyond text-based attacks, AI systems face sophisticated threats from adversarial machine learning — techniques that manipulate the underlying models themselves.

Adversarial Examples

Adversarial examples are inputs specifically crafted to fool AI models while appearing normal to humans.

Image classifiers:

• Adding imperceptible pixel perturbations to an image of a panda makes the model classify it as a gibbon with 99% confidence
• A physical sticker on a stop sign can make self-driving car vision systems misclassify it
• Adversarial eyeglass frames can fool facial recognition systems

Text models:

• Inserting invisible Unicode characters changes model behavior
• Synonym substitution that preserves meaning to humans but fools sentiment analysis
• Typosquatting-style attacks on named entity recognition

Audio models:

• Inaudible perturbations in audio that activate voice assistants
• Adversarial audio that sounds like music to humans but contains hidden commands

Data Poisoning

Data poisoning attacks target the training phase:

Backdoor attacks:

• Insert a small number of poisoned examples into training data
• The model learns a hidden trigger (e.g., a specific phrase or pixel pattern)
• Normal inputs produce normal outputs; triggered inputs produce attacker-chosen outputs
• Example: A sentiment classifier trained on poisoned data misclassifies any review containing "excellent service" as negative

Clean-label attacks:

• Poisoned examples have correct labels, making them harder to detect
• They subtly shift the model's decision boundary in the attacker's favor
• Require deep understanding of the model's learning dynamics

Model Supply Chain Attacks

The AI supply chain is vulnerable:

• Pre-trained model weights — Models downloaded from Hugging Face or other hubs could contain backdoors
• Fine-tuning datasets — Datasets scraped from the internet may include adversarial examples
• Dependencies — PyTorch, TensorFlow, and other libraries are attack surfaces
• Model merging — Popular technique that combines models, potentially mixing in compromised weights

Detection and Mitigation

• Adversarial training — Include adversarial examples in training to build robustness
• Input preprocessing — Transformations that neutralize perturbations (smoothing, compression)
• Ensemble methods — Multiple models voting reduces the impact of any single attack
• Data curation — Carefully vetting training data for poisoning attempts
• Model scanning — Tools that detect backdoors in trained models