Production AI systems can silently degrade. Without monitoring, you won't know until users complain — or worse, when incorrect outputs cause real harm.

What to Monitor

Infrastructure metrics:

• GPU utilization (target: >80%)
• GPU memory usage (watch for OOM errors)
• Request latency (p50, p95, p99)
• Throughput (requests/second)
• Error rates (timeouts, 5xx errors)
• Queue depth (for async processing)

Model quality metrics:

• Output quality scores (LLM-as-judge sampling)
• Hallucination detection rate
• Safety filter trigger rate
• User satisfaction (thumbs up/down ratios)
• Task completion rate

Data drift metrics:

• Input distribution changes (embedding drift, token length shifts)
• Output distribution changes (response length, category distribution)
• Feature drift for structured prediction models

Drift Detection

AI models are trained on a specific data distribution. When production data drifts from training data, model quality degrades:

• Covariate shift: Input distribution changes (e.g., users start asking about topics not in training data)
• Concept drift: The relationship between inputs and correct outputs changes (e.g., new regulations change what constitutes a correct legal answer)
• Label drift: The distribution of correct answers changes

Detection methods:

• Statistical tests (KS test, PSI) on input feature distributions
• Embedding similarity between production queries and training data
• Quality metric trends over time windows

Alerting Best Practices

• Set thresholds based on baseline metrics, not arbitrary numbers
• Use rolling averages to reduce noise (1-hour, 24-hour windows)
• Escalate alerts: warning → page → incident based on severity and duration
• Include actionable context in alerts (which model, which endpoint, sample inputs)

Incident Response for AI

When quality drops:

1. Verify: Check if it's a real quality issue or a metric bug
2. Scope: Is it all traffic or a specific segment?
3. Mitigate: Roll back to previous model version or increase guardrails
4. Root cause: Data issue? Model update? Infrastructure problem?
5. Prevent: Add monitoring for this failure mode