Performance & Cost Optimization
From the course GPU Cluster Management
GPU clusters are expensive. At $2–$8 per GPU-hour for cloud instances and significant CapEx for on-premise hardware, every percentage point of utilization improvement directly reduces cost. This lesson covers the techniques that deliver the highest ROI.
Baseline: Measure Before Optimizing
Before changing anything, measure actual GPU utilization across your workloads:
# Collect 1 hour of utilization samples across the cluster
# Run on each node or via Prometheus query
nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,power.draw --format=csv,noheader -l 10 >> /shared/metrics/gpu_util_node0.csv
# Quick cluster-wide summary via DCGM
dcgmi dmon -e 203,204,155 -d 5000 # SM util, mem util, power
Target baseline metrics before optimization:
- GPU SM utilization: aim for 85%+ during training
- Memory utilization: aim for 75–90% (leaving headroom for spikes)
- Power draw: should be near TDP during compute-intensive phases
Optimization 1: Mixed Precision Training
Switching from FP32 to BF16 or FP16 delivers 2–3x throughput improvement on Ampere/Hopper GPUs with minimal accuracy impact:
from torch.amp import autocast, GradScaler
scaler = GradScaler() # handles FP16 loss scaling
for images, labels in loader:
images, labels = images.to(device), labels.to(device)
optimizer.zero_grad()
# Forward pass in FP16
with autocast(device_type='cuda', dtype=torch.float16):
outputs = model(images)
loss = criterion(outputs, labels)
# Scale gradients to avoid underflow, then step
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
For H100/A100 clusters, prefer torch.bfloat16 over FP16 — BF16 does not require loss scaling and is numerically more stable.
Optimization 2: Gradient Checkpointing
Large models exhaust GPU VRAM during the backward pass. Gradient checkpointing recomputes activations instead of storing them, trading 30% extra compute for 60–70% VRAM reduction:
from torch.utils.checkpoint import checkpoint_sequential
# Instead of: output = model(input)
# Use: recompute activations through N segments
segments = 4
output = checkpoint_sequential(model, segments, input)
This allows training models that would otherwise OOM, often enabling a larger batch size that more than compensates for the recompute overhead.
Optimization 3: DataLoader Tuning
I/O starvation is a silent GPU killer. Use these settings to keep GPUs fed:
loader = DataLoader(
dataset,
batch_size=128,
num_workers=8, # 2x the number of GPUs per node
pin_memory=True, # pin host memory for faster H2D copy
prefetch_factor=4, # prefetch 4 batches per worker
persistent_workers=True # keep workers alive between epochs
)
Monitor with nvtop or nvidia-smi — if GPU utilization drops to 0% periodically in a sawtooth pattern, DataLoader is the bottleneck.
Optimization 4: Job Packing and Bin-Filling
Single large jobs leave GPUs idle during startup, checkpointing, and communication phases. Cluster schedulers can pack multiple smaller jobs onto the same nodes:
# Slurm: allow backfill scheduling to pack short jobs into idle gaps
# slurm.conf
SchedulerType=sched/backfill
PriorityType=priority/multifactor
# Submit inference jobs with short time limits so backfill picks them up
sbatch --time=00:30:00 --priority=low inference_job.sh
Target cluster-wide utilization above 75% during business hours. Below 60% typically indicates scheduling inefficiency rather than insufficient workload.
Cost Optimization for Cloud GPU Clusters
For AWS, GCP, or Azure GPU clusters:
| Strategy | Typical Saving |
|---|---|
| Spot/preemptible instances for fault-tolerant training | 60–80% vs on-demand |
| Reserved instances for baseline inference capacity | 30–40% vs on-demand |
| Mixed precision (no extra cost, 2–3x throughput) | Effective 50–65% cost per token |
| Rightsizing (A10G for inference vs A100 for training) | 40–60% vs over-provisioned |
The highest-ROI action for most teams is enabling mixed precision — it costs nothing and cuts training time (and cloud spend) by up to 60%.
Key Takeaways
- •Measure SM utilization and memory utilization before optimizing — sawtooth GPU idle patterns point to DataLoader bottlenecks
- •BF16 mixed precision delivers 2–3x throughput on Ampere/Hopper GPUs without loss scaling complexity
- •Gradient checkpointing reduces VRAM by 60–70% at the cost of 30% extra compute, enabling larger batch sizes
- •Spot/preemptible instances with checkpoint-based fault tolerance cut cloud GPU costs by 60–80%
Progress is kept in this browser only.