Monitor production inference systems and continuously optimize for cost and performance.

Observability Stack

Prometheus Metrics

from prometheus_client import Counter, Histogram, Gauge

# Define metrics
inference_requests = Counter('inference_requests_total', 'Total inference requests')
inference_latency = Histogram('inference_latency_seconds', 'Inference latency')
model_gpu_util = Gauge('model_gpu_utilization', 'GPU utilization %')

@app.post("/predict")
async def predict(request):
    inference_requests.inc()
    
    with inference_latency.time():
        output = model.generate(request["prompt"])
    
    # Update GPU metrics
    model_gpu_util.set(get_gpu_utilization())
    
    return output

Key Performance Indicators

• Throughput: Requests per second
• Latency: P50, P95, P99 response times
• GPU Utilization: Target 80%+ for cost-efficiency
• Error Rate: < 0.1% for production
• Cost per Prediction: Optimize for ROI

Performance Tuning

• Profile with nvidia-smi dmon and nsys
• Adjust batch sizes based on throughput/latency tradeoff
• Enable GPU memory pooling for allocation efficiency
• Use multiple model instances per GPU for high concurrency