Docker Compose for AI Services
From the course AI Deployment with Docker
Multi-Service AI Architecture
Compose Configuration
services:
api:
build: ./api
ports:
- "8000:8000"
depends_on:
- model-server
- vector-db
environment:
- MODEL_URL=http://model-server:8080
model-server:
image: ghcr.io/huggingface/text-generation-inference:latest
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
- model-cache:/data
command: --model-id meta-llama/Llama-3.1-8B-Instruct
vector-db:
image: chromadb/chroma:latest
ports:
- "8001:8000"
volumes:
- chroma-data:/chroma/chroma
redis:
image: redis:alpine
ports:
- "6379:6379"
volumes:
model-cache:
chroma-data:
Service Communication
- Services communicate by name (e.g.,
http://model-server:8080) - Use health checks to ensure dependencies are ready
- Implement retry logic for startup ordering
- Share data through volumes or API calls
Health Checks
model-server:
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 120s # Models take time to load
Common Pitfall: start_period Too Short for Large Models
Docker health checks start counting failures only after start_period elapses. A 7-billion-parameter model can take 60–180 seconds to load into VRAM. If start_period is set to 30 seconds, the health check fires during loading, marks the container unhealthy, and the orchestrator may kill and restart it repeatedly — a "restart storm." Always benchmark model load time in staging and set start_period to at least 1.5x that value.
Best Practice: Named Volumes for Model Caches
Model weights can be gigabytes in size. Use a named Docker volume that persists across container restarts so the model is only downloaded once:
volumes:
hf_cache:
services:
model-server:
volumes:
- hf_cache:/root/.cache/huggingface
environment:
- HF_HOME=/root/.cache/huggingface
With this setup, the first startup downloads the model. Every subsequent restart loads it from disk in seconds, saving both time and bandwidth costs.
Named Product Reference: Hugging Face TGI
For teams self-hosting open-weight LLMs, the ghcr.io/huggingface/text-generation-inference:latest image handles GPU sharding, continuous batching, and streaming out of the box. It exposes an OpenAI-compatible API, making it a drop-in backend for any app built against the OpenAI SDK.
Key Takeaways
- •Docker Compose orchestrates multi-service AI architectures
- •Services communicate by container name on shared networks
- •Use health checks with generous start periods for model loading
- •Persist model caches and databases with named volumes
Progress is kept in this browser only.