AI Model Wiki

    Skip to main content
    40 min
    Lesson 2 of 5

    GPU Passthrough & NVIDIA Runtime

    From the course AI Deployment with Docker

    GPU-Accelerated Containers

    NVIDIA Container Toolkit

    Install the toolkit for GPU access inside containers:

    # Install NVIDIA Container Toolkit
    sudo apt-get install nvidia-container-toolkit
    sudo systemctl restart docker
    

    GPU Dockerfile

    FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04
    
    RUN apt-get update && apt-get install -y python3 python3-pip
    RUN pip3 install torch transformers fastapi uvicorn
    
    COPY . /app
    WORKDIR /app
    CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
    

    Running with GPU

    # All GPUs
    docker run --gpus all my-ai-app
    
    # Specific GPUs
    docker run --gpus '"device=0,1"' my-ai-app
    
    # GPU memory limit
    docker run --gpus all --memory 16g my-ai-app
    

    GPU Monitoring

    # Monitor GPU usage inside container
    docker exec my-container nvidia-smi
    
    # Continuous monitoring
    watch -n 1 nvidia-smi
    

    Common GPU Issues

    • CUDA version mismatch: Ensure container CUDA version matches host driver
    • Out of memory: Monitor GPU RAM, implement batch size limits
    • Driver not found: Verify NVIDIA Container Toolkit installation
    • Multi-GPU: Use CUDA_VISIBLE_DEVICES to control GPU assignment

    Common Pitfall: CUDA Version Mismatch Between Host and Container

    The most common GPU Docker issue is a CUDA version mismatch. The CUDA version in your container image must be less than or equal to the CUDA version supported by the host driver. Run nvidia-smi on the host to see the maximum supported CUDA version, then choose a container image with an equal or lower CUDA version. The toolkit and driver versions are independent — you can run a CUDA 11.8 container on a host with a CUDA 12.x driver.

    Best Practice: Health-Check GPU Availability at Startup

    Add a startup check that verifies GPU access before the application begins serving requests:

    import torch
    import sys
    
    def check_gpu():
        if not torch.cuda.is_available():
            print("WARNING: No GPU found, running on CPU")
            return False
        print(f"GPU: {torch.cuda.get_device_name(0)}")
        print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
        return True
    
    check_gpu()
    

    Log this at every container start. If a deployment accidentally runs on a CPU node (common after cluster autoscaling), this warning surfaces immediately in your logging pipeline instead of silently degrading performance.

    Named Product Reference: Hugging Face TGI (Text Generation Inference)

    For teams deploying open-weight LLMs, Hugging Face's Text Generation Inference (TGI) server is the production-grade choice. The Docker image ghcr.io/huggingface/text-generation-inference:latest handles model sharding across multiple GPUs, continuous batching, and token streaming out of the box. It exposes an OpenAI-compatible API endpoint, making it a drop-in backend for apps built against the OpenAI SDK.

    Key Takeaways

    • •NVIDIA Container Toolkit enables GPU access inside Docker
    • •Match CUDA versions between container and host driver
    • •Use --gpus flag to control GPU allocation per container
    • •Monitor GPU memory to prevent out-of-memory errors
    0 of 5 lessons marked complete

    Progress is kept in this browser only.