20 min
Lesson 6 of 7
Observability & Production Hardening
From the course AI Fine-Tuning Pipelines
Monitor, debug, and harden your fine-tuned model in production.
Training Observability
Weights & Biases Integration
import wandb
wandb.init(project="fine-tuning-support-model", config={
"model": "llama-2-7b",
"lora_r": 16,
"learning_rate": 2e-4,
"dataset_size": 10000
})
# TrainingArguments automatically logs to W&B when report_to="wandb"
# Track: train/loss, eval/loss, learning_rate, grad_norm
Key Metrics to Watch
- Training loss: Should decrease steadily; sharp drops = overfitting risk
- Eval loss: Should track training loss; divergence = overfitting
- Gradient norm: Values over 10 indicate training instability
- Learning rate schedule: Cosine decay prevents overshooting
Production Monitoring
Request Logging
import structlog
from datetime import datetime
logger = structlog.get_logger()
async def generate_with_logging(request_id: str, prompt: str) -> str:
start = datetime.utcnow()
try:
response = await model.generate(prompt)
latency_ms = (datetime.utcnow() - start).total_seconds() * 1000
logger.info("inference_success",
request_id=request_id,
latency_ms=round(latency_ms, 2),
input_tokens=len(tokenizer.encode(prompt)),
output_tokens=len(tokenizer.encode(response))
)
return response
except Exception as e:
logger.error("inference_error", request_id=request_id, error=str(e))
raise
Output Quality Monitoring
class QualityMonitor:
def __init__(self, alert_threshold=0.15):
self.threshold = alert_threshold
self.refusal_count = 0
self.total_count = 0
def check(self, response: str) -> dict:
issues = []
if len(response.split()) < 10:
issues.append("too_short")
if any(phrase in response.lower() for phrase in ["i cannot", "i'm unable", "i don't know"]):
issues.append("refusal")
self.refusal_count += 1
self.total_count += 1
refusal_rate = self.refusal_count / max(self.total_count, 1)
if refusal_rate > self.threshold:
self.send_alert(f"High refusal rate: {refusal_rate:.1%}")
return {"issues": issues, "refusal_rate": refusal_rate}
Rollback & Safety
Canary Deployment
import random
def route_request(user_id: str, prompt: str) -> str:
# Send 5% of traffic to new fine-tuned model
use_new_model = hash(user_id) % 100 < 5
if use_new_model:
response = fine_tuned_v2.generate(prompt)
log_experiment("v2", user_id, prompt, response)
else:
response = fine_tuned_v1.generate(prompt)
log_experiment("v1", user_id, prompt, response)
return response
Model Versioning Checklist
- Tag every model with training run ID, dataset hash, and date
- Store eval metrics alongside model weights
- Keep at least 2 previous versions for instant rollback
- Document known failure modes and edge cases
- Run regression tests before promoting to 100% traffic
Key Takeaways
- •Track training loss, eval loss, and gradient norm in W&B to catch overfitting early
- •Structured logging with latency, token counts, and request IDs enables production debugging
- •Monitor refusal rate in production — spikes indicate distribution shift or adversarial inputs
- •Canary deployments at 5% traffic let you validate quality before full rollout
0 of 7 lessons marked complete
Progress is kept in this browser only.