Production Orchestration & Scaling
From the course AI Deployment with Docker
Scaling AI Services
Horizontal Scaling
# Scale API servers
docker compose up --scale api=4
# Or in compose file
services:
api:
deploy:
replicas: 4
Load Balancing
Use a reverse proxy for request distribution:
services:
nginx:
image: nginx:alpine
ports:
- "80:80"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
depends_on:
- api
Kubernetes for AI
For larger deployments, Kubernetes provides:
- Automatic scaling based on GPU utilization
- Rolling updates without downtime
- Resource quotas and limits per namespace
- Service mesh for traffic management
Production Checklist
- Security: Scan images for vulnerabilities, use non-root users
- Logging: Centralize logs with structured JSON output
- Monitoring: Track GPU utilization, memory, latency, throughput
- Backup: Persist model weights and vector data to durable storage
- CI/CD: Automate builds and deployments with image registries
- Secrets: Use Docker secrets or Kubernetes secrets for API keys
Common Pitfall: Hardcoding API Keys in Dockerfiles
Never put API keys, model tokens, or database passwords in Dockerfile ENV instructions or in the image itself. These become readable by anyone with image pull access. Instead, inject secrets at runtime using environment variables, Docker secrets, or a secrets manager like AWS Secrets Manager or HashiCorp Vault:
# Good: inject at runtime
docker run -e OPENAI_API_KEY=$OPENAI_API_KEY my-ai-app
# Bad: baked into image (visible in docker inspect and image layers)
ENV OPENAI_API_KEY=sk-...
Best Practice: Tag Images with Git SHA
Use the Git commit SHA as the Docker image tag instead of latest. This makes every deployment traceable back to the exact code version:
IMAGE_TAG=$(git rev-parse --short HEAD)
docker build -t my-ai-app:$IMAGE_TAG .
docker push my-ai-app:$IMAGE_TAG
In a Kubernetes deployment manifest, referencing my-ai-app:abc1234 pins the deployment to a specific image. Rolling back becomes a one-command operation: update the tag to a previous SHA.
Named Product Reference: Watchtower for Automatic Updates
For development or staging environments, Watchtower (a Docker container) watches your running containers and automatically pulls and redeploys new versions of their images. This lets you push a new image to your registry and have it deployed within minutes without SSHing into the server — useful for rapid AI model iteration.
Key Takeaways
- •Scale API servers horizontally with replicas behind a load balancer
- •Kubernetes provides auto-scaling, rolling updates, and resource management
- •Monitor GPU utilization, memory, and inference latency in production
- •Automate builds and deployments with CI/CD pipelines
Progress is kept in this browser only.