Production Monitoring
From the course AI Testing & Quality Assurance
Monitoring AI Quality in Production
What to Monitor
- Output quality: Sample and evaluate production outputs
- Latency: Time to first token and total response time
- Error rates: API failures, parsing errors, empty responses
- User feedback: Thumbs up/down, regeneration requests
- Cost: Token usage, cost per query, cost trends
Automated Quality Sampling
// Sample 5% of production responses for evaluation
async function monitoredGenerate(prompt: string) {
const response = await generate(prompt);
if (Math.random() < 0.05) {
// Async evaluation — don't block the response
evaluateAndLog(prompt, response).catch(console.error);
}
return response;
}
Alerting Rules
- Average quality score drops below threshold
- Error rate exceeds 5% in 5-minute window
- Latency P95 exceeds acceptable limit
- Token usage spikes unexpectedly
- User negative feedback rate increases
Feedback Loops
Build continuous improvement cycles:
- Collect user feedback on responses
- Add low-quality examples to eval dataset
- Improve prompts to handle failure cases
- Run evaluations to verify improvement
- Deploy and monitor again
Dashboard Metrics
| Metric | Good | Warning | Critical |
|---|---|---|---|
| Quality score | >4.0 | 3.5-4.0 | <3.5 |
| Error rate | <1% | 1-5% | >5% |
| P95 latency | <3s | 3-5s | >5s |
| Feedback ratio | >80% positive | 60-80% | <60% |
Common Pitfall: Sampling Too Infrequently to Catch Regressions
Evaluating 5% of production traffic is a good target, but 5% of low-traffic applications may mean evaluating only 2–3 responses per day — insufficient to detect gradual quality degradation. Set a minimum evaluation count per period:
const SAMPLE_RATE = 0.05; // 5%
const MIN_DAILY_EVALS = 20; // Always evaluate at least 20 per day
const shouldEvaluate = Math.random() < SAMPLE_RATE ||
(await getDailyEvalCount() < MIN_DAILY_EVALS);
Best Practice: Correlate User Feedback with Automated Scores
The most valuable signal is when user feedback disagrees with your automated quality score. High automated score + negative user feedback means your eval criteria are misaligned with user expectations — a critical finding:
// In your weekly quality review
const mismatches = await db.query(`
SELECT r.id, r.automated_score, f.user_rating
FROM responses r
JOIN user_feedback f ON r.id = f.response_id
WHERE ABS(r.automated_score - f.user_rating) > 1.5
ORDER BY ABS(r.automated_score - f.user_rating) DESC
LIMIT 20
`);
// Manually review these cases to identify eval criteria gaps
Named Product Reference: Langfuse for Production AI Monitoring
Langfuse is an open-source observability platform that instruments LLM calls, measures latency, tracks costs, and enables user feedback collection. It provides dashboards for quality trends, cost per conversation, and user satisfaction correlations. Self-hostable on your own infrastructure with a Docker compose file, or available as a managed cloud service.
Key Takeaways
- •Sample and evaluate production outputs automatically
- •Set up alerts for quality drops, error spikes, and latency
- •Build feedback loops from user signals to prompt improvements
- •Track quality, latency, errors, and cost on a unified dashboard
Progress is kept in this browser only.