AI Model Wiki

    Skip to main content
    20 min
    Lesson 6 of 7

    Observability & Production Hardening

    From the course AI Fine-Tuning Pipelines

    Monitor, debug, and harden your fine-tuned model in production.

    Training Observability

    Weights & Biases Integration

    import wandb
    
    wandb.init(project="fine-tuning-support-model", config={
        "model": "llama-2-7b",
        "lora_r": 16,
        "learning_rate": 2e-4,
        "dataset_size": 10000
    })
    
    # TrainingArguments automatically logs to W&B when report_to="wandb"
    # Track: train/loss, eval/loss, learning_rate, grad_norm
    

    Key Metrics to Watch

    • Training loss: Should decrease steadily; sharp drops = overfitting risk
    • Eval loss: Should track training loss; divergence = overfitting
    • Gradient norm: Values over 10 indicate training instability
    • Learning rate schedule: Cosine decay prevents overshooting

    Production Monitoring

    Request Logging

    import structlog
    from datetime import datetime
    
    logger = structlog.get_logger()
    
    async def generate_with_logging(request_id: str, prompt: str) -> str:
        start = datetime.utcnow()
    
        try:
            response = await model.generate(prompt)
            latency_ms = (datetime.utcnow() - start).total_seconds() * 1000
    
            logger.info("inference_success",
                request_id=request_id,
                latency_ms=round(latency_ms, 2),
                input_tokens=len(tokenizer.encode(prompt)),
                output_tokens=len(tokenizer.encode(response))
            )
            return response
        except Exception as e:
            logger.error("inference_error", request_id=request_id, error=str(e))
            raise
    

    Output Quality Monitoring

    class QualityMonitor:
        def __init__(self, alert_threshold=0.15):
            self.threshold = alert_threshold
            self.refusal_count = 0
            self.total_count = 0
    
        def check(self, response: str) -> dict:
            issues = []
            if len(response.split()) < 10:
                issues.append("too_short")
            if any(phrase in response.lower() for phrase in ["i cannot", "i'm unable", "i don't know"]):
                issues.append("refusal")
                self.refusal_count += 1
            self.total_count += 1
    
            refusal_rate = self.refusal_count / max(self.total_count, 1)
            if refusal_rate > self.threshold:
                self.send_alert(f"High refusal rate: {refusal_rate:.1%}")
            return {"issues": issues, "refusal_rate": refusal_rate}
    

    Rollback & Safety

    Canary Deployment

    import random
    
    def route_request(user_id: str, prompt: str) -> str:
        # Send 5% of traffic to new fine-tuned model
        use_new_model = hash(user_id) % 100 < 5
    
        if use_new_model:
            response = fine_tuned_v2.generate(prompt)
            log_experiment("v2", user_id, prompt, response)
        else:
            response = fine_tuned_v1.generate(prompt)
            log_experiment("v1", user_id, prompt, response)
    
        return response
    

    Model Versioning Checklist

    • Tag every model with training run ID, dataset hash, and date
    • Store eval metrics alongside model weights
    • Keep at least 2 previous versions for instant rollback
    • Document known failure modes and edge cases
    • Run regression tests before promoting to 100% traffic

    Key Takeaways

    • •Track training loss, eval loss, and gradient norm in W&B to catch overfitting early
    • •Structured logging with latency, token counts, and request IDs enables production debugging
    • •Monitor refusal rate in production — spikes indicate distribution shift or adversarial inputs
    • •Canary deployments at 5% traffic let you validate quality before full rollout
    0 of 7 lessons marked complete

    Progress is kept in this browser only.