AI Model Wiki

    Skip to main content
    45 min
    Lesson 4 of 5

    Production Orchestration & Scaling

    From the course AI Deployment with Docker

    Scaling AI Services

    Horizontal Scaling

    # Scale API servers
    docker compose up --scale api=4
    
    # Or in compose file
    services:
      api:
        deploy:
          replicas: 4
    

    Load Balancing

    Use a reverse proxy for request distribution:

    services:
      nginx:
        image: nginx:alpine
        ports:
          - "80:80"
        volumes:
          - ./nginx.conf:/etc/nginx/nginx.conf
        depends_on:
          - api
    

    Kubernetes for AI

    For larger deployments, Kubernetes provides:

    • Automatic scaling based on GPU utilization
    • Rolling updates without downtime
    • Resource quotas and limits per namespace
    • Service mesh for traffic management

    Production Checklist

    • Security: Scan images for vulnerabilities, use non-root users
    • Logging: Centralize logs with structured JSON output
    • Monitoring: Track GPU utilization, memory, latency, throughput
    • Backup: Persist model weights and vector data to durable storage
    • CI/CD: Automate builds and deployments with image registries
    • Secrets: Use Docker secrets or Kubernetes secrets for API keys

    Common Pitfall: Hardcoding API Keys in Dockerfiles

    Never put API keys, model tokens, or database passwords in Dockerfile ENV instructions or in the image itself. These become readable by anyone with image pull access. Instead, inject secrets at runtime using environment variables, Docker secrets, or a secrets manager like AWS Secrets Manager or HashiCorp Vault:

    # Good: inject at runtime
    docker run -e OPENAI_API_KEY=$OPENAI_API_KEY my-ai-app
    
    # Bad: baked into image (visible in docker inspect and image layers)
    ENV OPENAI_API_KEY=sk-...
    

    Best Practice: Tag Images with Git SHA

    Use the Git commit SHA as the Docker image tag instead of latest. This makes every deployment traceable back to the exact code version:

    IMAGE_TAG=$(git rev-parse --short HEAD)
    docker build -t my-ai-app:$IMAGE_TAG .
    docker push my-ai-app:$IMAGE_TAG
    

    In a Kubernetes deployment manifest, referencing my-ai-app:abc1234 pins the deployment to a specific image. Rolling back becomes a one-command operation: update the tag to a previous SHA.

    Named Product Reference: Watchtower for Automatic Updates

    For development or staging environments, Watchtower (a Docker container) watches your running containers and automatically pulls and redeploys new versions of their images. This lets you push a new image to your registry and have it deployed within minutes without SSHing into the server — useful for rapid AI model iteration.

    Key Takeaways

    • •Scale API servers horizontally with replicas behind a load balancer
    • •Kubernetes provides auto-scaling, rolling updates, and resource management
    • •Monitor GPU utilization, memory, and inference latency in production
    • •Automate builds and deployments with CI/CD pipelines
    0 of 5 lessons marked complete

    Progress is kept in this browser only.