20 min
Lesson 3 of 4
Performance Monitoring and Tuning
From the course AI Hardware Optimization
Monitor and continuously optimize AI hardware performance with profiling tools and metrics.
GPU Profiling
NVIDIA Nsight Systems
# Profile entire training run
nsys profile --trace=cuda,nvtx,osrt --output=profile.nsys-rep \
python train.py
# Analyze in GUI
nsys-ui profile.nsys-rep
# Key metrics to watch:
# - GPU utilization (target: >90%)
# - Memory bandwidth utilization
# - Kernel launch overhead
# - PCIe transfer time
PyTorch Profiler
from torch.profiler import profile, ProfilerActivity
with profile(
activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
record_shapes=True,
profile_memory=True,
with_stack=True
) as prof:
for _ in range(10):
output = model(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()
# Export for TensorBoard
prof.export_chrome_trace("trace.json")
# Print table
print(prof.key_averages().table(
sort_by="cuda_time_total",
row_limit=20
))
Real-Time Monitoring
GPU Metrics Dashboard
# Install nvidia-smi with continuous monitoring
watch -n 1 nvidia-smi
# Or use nvitop for better UI
pip install nvitop
nvitop
# Metrics to track:
# - GPU utilization %
# - Memory used/total
# - Power draw vs TDP
# - Temperature
# - GPU clock speeds
Custom Monitoring Script
import pynvml
import time
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
while True:
utilization = pynvml.nvmlDeviceGetUtilizationRates(handle)
memory = pynvml.nvmlDeviceGetMemoryInfo(handle)
power = pynvml.nvmlDeviceGetPowerUsage(handle) / 1000 # Watts
temp = pynvml.nvmlDeviceGetTemperature(handle, 0)
print(f"GPU: {utilization.gpu}% | "
f"Mem: {memory.used/1e9:.1f}/{memory.total/1e9:.1f}GB | "
f"Power: {power:.0f}W | Temp: {temp}°C")
time.sleep(1)
Bottleneck Identification
Common Issues and Solutions
| Symptom | Bottleneck | Solution |
|---|---|---|
| GPU util < 80% | Data loading | Increase num_workers, pin_memory |
| High PCIe time | CPU-GPU transfer | Increase batch size, cache on GPU |
| Memory errors | VRAM overflow | Reduce batch size, use gradient checkpointing |
| Low throughput | Kernel inefficiency | Use fused kernels, compile with TorchScript |
| Thermal throttling | Cooling | Improve airflow, reduce power limit |
Key Takeaways
- •Nsight Systems and PyTorch Profiler identify GPU utilization and bottlenecks
- •Real-time monitoring with nvitop tracks utilization, memory, power, and temperature
- •GPU utilization below 80% typically indicates data loading or transfer bottlenecks
- •Systematic profiling and tuning iteratively removes performance bottlenecks
0 of 4 lessons marked complete
Progress is kept in this browser only.