AI Model Wiki

    Skip to main content
    20 min
    Lesson 3 of 4

    Performance Monitoring and Tuning

    From the course AI Hardware Optimization

    Monitor and continuously optimize AI hardware performance with profiling tools and metrics.

    GPU Profiling

    NVIDIA Nsight Systems

    # Profile entire training run
    nsys profile --trace=cuda,nvtx,osrt --output=profile.nsys-rep \
      python train.py
    
    # Analyze in GUI
    nsys-ui profile.nsys-rep
    
    # Key metrics to watch:
    # - GPU utilization (target: >90%)
    # - Memory bandwidth utilization
    # - Kernel launch overhead
    # - PCIe transfer time
    

    PyTorch Profiler

    from torch.profiler import profile, ProfilerActivity
    
    with profile(
        activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
        record_shapes=True,
        profile_memory=True,
        with_stack=True
    ) as prof:
        for _ in range(10):
            output = model(input)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
    
    # Export for TensorBoard
    prof.export_chrome_trace("trace.json")
    
    # Print table
    print(prof.key_averages().table(
        sort_by="cuda_time_total",
        row_limit=20
    ))
    

    Real-Time Monitoring

    GPU Metrics Dashboard

    # Install nvidia-smi with continuous monitoring
    watch -n 1 nvidia-smi
    
    # Or use nvitop for better UI
    pip install nvitop
    nvitop
    
    # Metrics to track:
    # - GPU utilization %
    # - Memory used/total
    # - Power draw vs TDP
    # - Temperature
    # - GPU clock speeds
    

    Custom Monitoring Script

    import pynvml
    import time
    
    pynvml.nvmlInit()
    handle = pynvml.nvmlDeviceGetHandleByIndex(0)
    
    while True:
        utilization = pynvml.nvmlDeviceGetUtilizationRates(handle)
        memory = pynvml.nvmlDeviceGetMemoryInfo(handle)
        power = pynvml.nvmlDeviceGetPowerUsage(handle) / 1000  # Watts
        temp = pynvml.nvmlDeviceGetTemperature(handle, 0)
        
        print(f"GPU: {utilization.gpu}% | "
              f"Mem: {memory.used/1e9:.1f}/{memory.total/1e9:.1f}GB | "
              f"Power: {power:.0f}W | Temp: {temp}°C")
        
        time.sleep(1)
    

    Bottleneck Identification

    Common Issues and Solutions

    SymptomBottleneckSolution
    GPU util < 80%Data loadingIncrease num_workers, pin_memory
    High PCIe timeCPU-GPU transferIncrease batch size, cache on GPU
    Memory errorsVRAM overflowReduce batch size, use gradient checkpointing
    Low throughputKernel inefficiencyUse fused kernels, compile with TorchScript
    Thermal throttlingCoolingImprove airflow, reduce power limit

    Key Takeaways

    • •Nsight Systems and PyTorch Profiler identify GPU utilization and bottlenecks
    • •Real-time monitoring with nvitop tracks utilization, memory, power, and temperature
    • •GPU utilization below 80% typically indicates data loading or transfer bottlenecks
    • •Systematic profiling and tuning iteratively removes performance bottlenecks
    0 of 4 lessons marked complete

    Progress is kept in this browser only.