AI Model Wiki

    Skip to main content
    40 min
    Lesson 4 of 5

    Production Monitoring

    From the course AI Testing & Quality Assurance

    Monitoring AI Quality in Production

    What to Monitor

    • Output quality: Sample and evaluate production outputs
    • Latency: Time to first token and total response time
    • Error rates: API failures, parsing errors, empty responses
    • User feedback: Thumbs up/down, regeneration requests
    • Cost: Token usage, cost per query, cost trends

    Automated Quality Sampling

    // Sample 5% of production responses for evaluation
    async function monitoredGenerate(prompt: string) {
      const response = await generate(prompt);
    
      if (Math.random() < 0.05) {
        // Async evaluation — don't block the response
        evaluateAndLog(prompt, response).catch(console.error);
      }
    
      return response;
    }
    

    Alerting Rules

    • Average quality score drops below threshold
    • Error rate exceeds 5% in 5-minute window
    • Latency P95 exceeds acceptable limit
    • Token usage spikes unexpectedly
    • User negative feedback rate increases

    Feedback Loops

    Build continuous improvement cycles:

    1. Collect user feedback on responses
    2. Add low-quality examples to eval dataset
    3. Improve prompts to handle failure cases
    4. Run evaluations to verify improvement
    5. Deploy and monitor again

    Dashboard Metrics

    MetricGoodWarningCritical
    Quality score>4.03.5-4.0<3.5
    Error rate<1%1-5%>5%
    P95 latency<3s3-5s>5s
    Feedback ratio>80% positive60-80%<60%

    Common Pitfall: Sampling Too Infrequently to Catch Regressions

    Evaluating 5% of production traffic is a good target, but 5% of low-traffic applications may mean evaluating only 2–3 responses per day — insufficient to detect gradual quality degradation. Set a minimum evaluation count per period:

    const SAMPLE_RATE = 0.05;  // 5%
    const MIN_DAILY_EVALS = 20;  // Always evaluate at least 20 per day
    
    const shouldEvaluate = Math.random() < SAMPLE_RATE ||
      (await getDailyEvalCount() < MIN_DAILY_EVALS);
    

    Best Practice: Correlate User Feedback with Automated Scores

    The most valuable signal is when user feedback disagrees with your automated quality score. High automated score + negative user feedback means your eval criteria are misaligned with user expectations — a critical finding:

    // In your weekly quality review
    const mismatches = await db.query(`
      SELECT r.id, r.automated_score, f.user_rating
      FROM responses r
      JOIN user_feedback f ON r.id = f.response_id
      WHERE ABS(r.automated_score - f.user_rating) > 1.5
      ORDER BY ABS(r.automated_score - f.user_rating) DESC
      LIMIT 20
    `);
    // Manually review these cases to identify eval criteria gaps
    

    Named Product Reference: Langfuse for Production AI Monitoring

    Langfuse is an open-source observability platform that instruments LLM calls, measures latency, tracks costs, and enables user feedback collection. It provides dashboards for quality trends, cost per conversation, and user satisfaction correlations. Self-hostable on your own infrastructure with a Docker compose file, or available as a managed cloud service.

    Key Takeaways

    • •Sample and evaluate production outputs automatically
    • •Set up alerts for quality drops, error spikes, and latency
    • •Build feedback loops from user signals to prompt improvements
    • •Track quality, latency, errors, and cost on a unified dashboard
    0 of 5 lessons marked complete

    Progress is kept in this browser only.