GPU Passthrough & NVIDIA Runtime
From the course AI Deployment with Docker
GPU-Accelerated Containers
NVIDIA Container Toolkit
Install the toolkit for GPU access inside containers:
# Install NVIDIA Container Toolkit
sudo apt-get install nvidia-container-toolkit
sudo systemctl restart docker
GPU Dockerfile
FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y python3 python3-pip
RUN pip3 install torch transformers fastapi uvicorn
COPY . /app
WORKDIR /app
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Running with GPU
# All GPUs
docker run --gpus all my-ai-app
# Specific GPUs
docker run --gpus '"device=0,1"' my-ai-app
# GPU memory limit
docker run --gpus all --memory 16g my-ai-app
GPU Monitoring
# Monitor GPU usage inside container
docker exec my-container nvidia-smi
# Continuous monitoring
watch -n 1 nvidia-smi
Common GPU Issues
- CUDA version mismatch: Ensure container CUDA version matches host driver
- Out of memory: Monitor GPU RAM, implement batch size limits
- Driver not found: Verify NVIDIA Container Toolkit installation
- Multi-GPU: Use CUDA_VISIBLE_DEVICES to control GPU assignment
Common Pitfall: CUDA Version Mismatch Between Host and Container
The most common GPU Docker issue is a CUDA version mismatch. The CUDA version in your container image must be less than or equal to the CUDA version supported by the host driver. Run nvidia-smi on the host to see the maximum supported CUDA version, then choose a container image with an equal or lower CUDA version. The toolkit and driver versions are independent — you can run a CUDA 11.8 container on a host with a CUDA 12.x driver.
Best Practice: Health-Check GPU Availability at Startup
Add a startup check that verifies GPU access before the application begins serving requests:
import torch
import sys
def check_gpu():
if not torch.cuda.is_available():
print("WARNING: No GPU found, running on CPU")
return False
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
return True
check_gpu()
Log this at every container start. If a deployment accidentally runs on a CPU node (common after cluster autoscaling), this warning surfaces immediately in your logging pipeline instead of silently degrading performance.
Named Product Reference: Hugging Face TGI (Text Generation Inference)
For teams deploying open-weight LLMs, Hugging Face's Text Generation Inference (TGI) server is the production-grade choice. The Docker image ghcr.io/huggingface/text-generation-inference:latest handles model sharding across multiple GPUs, continuous batching, and token streaming out of the box. It exposes an OpenAI-compatible API endpoint, making it a drop-in backend for apps built against the OpenAI SDK.
Key Takeaways
- •NVIDIA Container Toolkit enables GPU access inside Docker
- •Match CUDA versions between container and host driver
- •Use --gpus flag to control GPU allocation per container
- •Monitor GPU memory to prevent out-of-memory errors
Progress is kept in this browser only.