AI Model Wiki

    Skip to main content
    20 min
    Lesson 3 of 4

    Production AI Monitoring

    From the course AI Debugging & Testing

    Monitor AI systems in production to detect drift, failures, and quality degradation.

    Key Metrics to Track

    Performance Metrics

    # Track these metrics continuously
    metrics = {
        "latency_p50": measure_latency(percentile=50),
        "latency_p99": measure_latency(percentile=99),
        "tokens_per_second": output_tokens / response_time,
        "error_rate": failed_requests / total_requests,
        "timeout_rate": timeouts / total_requests,
    }
    

    Quality Metrics

    # Sample-based quality monitoring
    quality_metrics = {
        "user_feedback_positive": positive_ratings / total_ratings,
        "format_compliance": valid_format / total_responses,
        "avg_response_length": mean(response_lengths),
        "hallucination_rate": hallucinations_detected / sampled_responses,
    }
    

    Alerting Rules

    alerts:
      - name: high_error_rate
        condition: error_rate > 0.05
        severity: critical
        
      - name: latency_spike
        condition: latency_p99 > 5000ms
        severity: warning
        
      - name: quality_degradation
        condition: user_feedback_positive < 0.8
        severity: warning
        
      - name: cost_anomaly
        condition: daily_cost > 2x rolling_avg
        severity: info
    

    Drift Detection

    from evidently import Report
    from evidently.metrics import TextDescriptorsDistribution
    
    # Detect distribution shifts in prompts/responses
    report = Report(metrics=[
        TextDescriptorsDistribution(column_name="prompt"),
        TextDescriptorsDistribution(column_name="response"),
    ])
    
    report.run(reference_data=baseline_data, current_data=production_data)
    drift_detected = report.as_dict()["metrics"][0]["drift_detected"]
    

    Key Takeaways

    • •Track latency (p50, p99), error rate, tokens/second, and timeout rate continuously
    • •Sample-based quality monitoring measures format compliance, feedback, and hallucination rate
    • •Set alerting rules for error spikes, latency issues, quality drops, and cost anomalies
    • •Use Evidently or similar tools to detect prompt/response distribution drift in production
    0 of 4 lessons marked complete

    Progress is kept in this browser only.