20 min
Lesson 3 of 4
Production AI Monitoring
From the course AI Debugging & Testing
Monitor AI systems in production to detect drift, failures, and quality degradation.
Key Metrics to Track
Performance Metrics
# Track these metrics continuously
metrics = {
"latency_p50": measure_latency(percentile=50),
"latency_p99": measure_latency(percentile=99),
"tokens_per_second": output_tokens / response_time,
"error_rate": failed_requests / total_requests,
"timeout_rate": timeouts / total_requests,
}
Quality Metrics
# Sample-based quality monitoring
quality_metrics = {
"user_feedback_positive": positive_ratings / total_ratings,
"format_compliance": valid_format / total_responses,
"avg_response_length": mean(response_lengths),
"hallucination_rate": hallucinations_detected / sampled_responses,
}
Alerting Rules
alerts:
- name: high_error_rate
condition: error_rate > 0.05
severity: critical
- name: latency_spike
condition: latency_p99 > 5000ms
severity: warning
- name: quality_degradation
condition: user_feedback_positive < 0.8
severity: warning
- name: cost_anomaly
condition: daily_cost > 2x rolling_avg
severity: info
Drift Detection
from evidently import Report
from evidently.metrics import TextDescriptorsDistribution
# Detect distribution shifts in prompts/responses
report = Report(metrics=[
TextDescriptorsDistribution(column_name="prompt"),
TextDescriptorsDistribution(column_name="response"),
])
report.run(reference_data=baseline_data, current_data=production_data)
drift_detected = report.as_dict()["metrics"][0]["drift_detected"]
Key Takeaways
- •Track latency (p50, p99), error rate, tokens/second, and timeout rate continuously
- •Sample-based quality monitoring measures format compliance, feedback, and hallucination rate
- •Set alerting rules for error spikes, latency issues, quality drops, and cost anomalies
- •Use Evidently or similar tools to detect prompt/response distribution drift in production
0 of 4 lessons marked complete
Progress is kept in this browser only.