20 min
Lesson 3 of 4
Safe AI Deployments
From the course AI CI/CD Pipelines
Implement safe deployment strategies for AI applications with rollback capabilities.
Deployment Strategies
Canary Deployments
# Kubernetes canary config
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: ai-service
spec:
strategy:
canary:
steps:
- setWeight: 5
- pause: {duration: 10m}
- analysis:
templates:
- templateName: ai-quality-check
- setWeight: 25
- pause: {duration: 10m}
- setWeight: 50
- pause: {duration: 10m}
- setWeight: 100
Feature Flags for AI
from launchdarkly import LDClient
ld_client = LDClient("sdk-key")
def get_ai_response(user, prompt):
# Check which model version to use
model_version = ld_client.variation(
"ai-model-version",
user,
default="gpt-4"
)
# Check if new prompt template is enabled
use_new_prompt = ld_client.variation(
"new-prompt-template",
user,
default=False
)
return call_model(
model=model_version,
prompt=new_prompt if use_new_prompt else legacy_prompt
)
Rollback Automation
def automated_rollback_check():
metrics = get_production_metrics(window="5m")
rollback_triggers = [
metrics["error_rate"] > 0.1,
metrics["latency_p99"] > 5000,
metrics["user_feedback_negative"] > 0.3,
metrics["hallucination_rate"] > 0.05,
]
if any(rollback_triggers):
trigger_rollback()
alert_oncall("AI deployment rolled back due to quality issues")
Key Takeaways
- •Canary deployments gradually shift traffic (5% → 25% → 50% → 100%) with analysis steps
- •Feature flags enable instant rollback of model versions and prompt templates
- •Automated rollback triggers on error rate, latency, feedback, and hallucination spikes
- •Argo Rollouts integrates AI quality checks into Kubernetes deployment pipelines
0 of 4 lessons marked complete
Progress is kept in this browser only.