AI Model Wiki

    Skip to main content
    25 min
    Lesson 5 of 5

    Performance & Cost Optimization

    From the course GPU Cluster Management

    GPU clusters are expensive. At $2–$8 per GPU-hour for cloud instances and significant CapEx for on-premise hardware, every percentage point of utilization improvement directly reduces cost. This lesson covers the techniques that deliver the highest ROI.

    Baseline: Measure Before Optimizing

    Before changing anything, measure actual GPU utilization across your workloads:

    # Collect 1 hour of utilization samples across the cluster
    # Run on each node or via Prometheus query
    nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,power.draw   --format=csv,noheader -l 10 >> /shared/metrics/gpu_util_node0.csv
    
    # Quick cluster-wide summary via DCGM
    dcgmi dmon -e 203,204,155 -d 5000  # SM util, mem util, power
    

    Target baseline metrics before optimization:

    • GPU SM utilization: aim for 85%+ during training
    • Memory utilization: aim for 75–90% (leaving headroom for spikes)
    • Power draw: should be near TDP during compute-intensive phases

    Optimization 1: Mixed Precision Training

    Switching from FP32 to BF16 or FP16 delivers 2–3x throughput improvement on Ampere/Hopper GPUs with minimal accuracy impact:

    from torch.amp import autocast, GradScaler
    
    scaler = GradScaler()  # handles FP16 loss scaling
    
    for images, labels in loader:
        images, labels = images.to(device), labels.to(device)
        optimizer.zero_grad()
    
        # Forward pass in FP16
        with autocast(device_type='cuda', dtype=torch.float16):
            outputs = model(images)
            loss = criterion(outputs, labels)
    
        # Scale gradients to avoid underflow, then step
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
    

    For H100/A100 clusters, prefer torch.bfloat16 over FP16 — BF16 does not require loss scaling and is numerically more stable.

    Optimization 2: Gradient Checkpointing

    Large models exhaust GPU VRAM during the backward pass. Gradient checkpointing recomputes activations instead of storing them, trading 30% extra compute for 60–70% VRAM reduction:

    from torch.utils.checkpoint import checkpoint_sequential
    
    # Instead of: output = model(input)
    # Use: recompute activations through N segments
    segments = 4
    output = checkpoint_sequential(model, segments, input)
    

    This allows training models that would otherwise OOM, often enabling a larger batch size that more than compensates for the recompute overhead.

    Optimization 3: DataLoader Tuning

    I/O starvation is a silent GPU killer. Use these settings to keep GPUs fed:

    loader = DataLoader(
        dataset,
        batch_size=128,
        num_workers=8,          # 2x the number of GPUs per node
        pin_memory=True,        # pin host memory for faster H2D copy
        prefetch_factor=4,      # prefetch 4 batches per worker
        persistent_workers=True # keep workers alive between epochs
    )
    

    Monitor with nvtop or nvidia-smi — if GPU utilization drops to 0% periodically in a sawtooth pattern, DataLoader is the bottleneck.

    Optimization 4: Job Packing and Bin-Filling

    Single large jobs leave GPUs idle during startup, checkpointing, and communication phases. Cluster schedulers can pack multiple smaller jobs onto the same nodes:

    # Slurm: allow backfill scheduling to pack short jobs into idle gaps
    # slurm.conf
    SchedulerType=sched/backfill
    PriorityType=priority/multifactor
    
    # Submit inference jobs with short time limits so backfill picks them up
    sbatch --time=00:30:00 --priority=low inference_job.sh
    

    Target cluster-wide utilization above 75% during business hours. Below 60% typically indicates scheduling inefficiency rather than insufficient workload.

    Cost Optimization for Cloud GPU Clusters

    For AWS, GCP, or Azure GPU clusters:

    StrategyTypical Saving
    Spot/preemptible instances for fault-tolerant training60–80% vs on-demand
    Reserved instances for baseline inference capacity30–40% vs on-demand
    Mixed precision (no extra cost, 2–3x throughput)Effective 50–65% cost per token
    Rightsizing (A10G for inference vs A100 for training)40–60% vs over-provisioned

    The highest-ROI action for most teams is enabling mixed precision — it costs nothing and cuts training time (and cloud spend) by up to 60%.

    Key Takeaways

    • •Measure SM utilization and memory utilization before optimizing — sawtooth GPU idle patterns point to DataLoader bottlenecks
    • •BF16 mixed precision delivers 2–3x throughput on Ampere/Hopper GPUs without loss scaling complexity
    • •Gradient checkpointing reduces VRAM by 60–70% at the cost of 30% extra compute, enabling larger batch sizes
    • •Spot/preemptible instances with checkpoint-based fault tolerance cut cloud GPU costs by 60–80%
    0 of 5 lessons marked complete

    Progress is kept in this browser only.