AI Model Wiki

    Skip to main content
    40 min
    Lesson 3 of 5

    Docker Compose for AI Services

    From the course AI Deployment with Docker

    Multi-Service AI Architecture

    Compose Configuration

    services:
      api:
        build: ./api
        ports:
          - "8000:8000"
        depends_on:
          - model-server
          - vector-db
        environment:
          - MODEL_URL=http://model-server:8080
    
      model-server:
        image: ghcr.io/huggingface/text-generation-inference:latest
        deploy:
          resources:
            reservations:
              devices:
                - driver: nvidia
                  count: 1
                  capabilities: [gpu]
        volumes:
          - model-cache:/data
        command: --model-id meta-llama/Llama-3.1-8B-Instruct
    
      vector-db:
        image: chromadb/chroma:latest
        ports:
          - "8001:8000"
        volumes:
          - chroma-data:/chroma/chroma
    
      redis:
        image: redis:alpine
        ports:
          - "6379:6379"
    
    volumes:
      model-cache:
      chroma-data:
    

    Service Communication

    • Services communicate by name (e.g., http://model-server:8080)
    • Use health checks to ensure dependencies are ready
    • Implement retry logic for startup ordering
    • Share data through volumes or API calls

    Health Checks

    model-server:
      healthcheck:
        test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
        interval: 30s
        timeout: 10s
        retries: 5
        start_period: 120s  # Models take time to load
    

    Common Pitfall: start_period Too Short for Large Models

    Docker health checks start counting failures only after start_period elapses. A 7-billion-parameter model can take 60–180 seconds to load into VRAM. If start_period is set to 30 seconds, the health check fires during loading, marks the container unhealthy, and the orchestrator may kill and restart it repeatedly — a "restart storm." Always benchmark model load time in staging and set start_period to at least 1.5x that value.

    Best Practice: Named Volumes for Model Caches

    Model weights can be gigabytes in size. Use a named Docker volume that persists across container restarts so the model is only downloaded once:

    volumes:
      hf_cache:
    
    services:
      model-server:
        volumes:
          - hf_cache:/root/.cache/huggingface
        environment:
          - HF_HOME=/root/.cache/huggingface
    

    With this setup, the first startup downloads the model. Every subsequent restart loads it from disk in seconds, saving both time and bandwidth costs.

    Named Product Reference: Hugging Face TGI

    For teams self-hosting open-weight LLMs, the ghcr.io/huggingface/text-generation-inference:latest image handles GPU sharding, continuous batching, and streaming out of the box. It exposes an OpenAI-compatible API, making it a drop-in backend for any app built against the OpenAI SDK.

    Key Takeaways

    • •Docker Compose orchestrates multi-service AI architectures
    • •Services communicate by container name on shared networks
    • •Use health checks with generous start periods for model loading
    • •Persist model caches and databases with named volumes
    0 of 5 lessons marked complete

    Progress is kept in this browser only.