AI Model Wiki

    Skip to main content
    25 min
    Lesson 2 of 4

    System-Level Optimization

    From the course AI Hardware Optimization

    Optimize the entire system stack from BIOS settings to kernel parameters for maximum AI performance.

    BIOS and Firmware Optimization

    Critical BIOS Settings

    Power Management:
    - Disable C-States (CPU sleep states)
    - Set Performance Power Plan
    - Disable CPU throttling
    
    PCIe Configuration:
    - Enable Above 4G Decoding
    - Set PCIe to Gen 4/5 (max available)
    - Enable Resizable BAR for better memory access
    
    Memory:
    - Enable XMP/DOCC profiles for RAM
    - Set memory frequency to maximum stable
    - Disable memory power-down modes
    

    Linux Kernel Optimization

    Kernel Parameters

    # Add to /etc/sysctl.conf for AI workloads
    vm.swappiness=10                    # Minimize swap usage
    vm.dirty_ratio=40                    # Optimize write performance
    vm.dirty_background_ratio=10
    kernel.sched_latency_ns=10000000    # Reduce scheduling latency
    kernel.sched_min_granularity_ns=2000000
    

    NUMA Configuration

    # Check NUMA topology
    numactl --hardware
    
    # Pin GPU to nearest CPU NUMA node
    numactl --cpunodebind=0 --membind=0 python train.py
    
    # For 8-GPU system, bind correctly
    taskset -c 0-15 python train_gpu0.py   # GPU 0 to CPU 0-15
    taskset -c 16-31 python train_gpu1.py  # GPU 1 to CPU 16-31
    

    CPU-GPU Coordination

    Data Loading Optimization

    import torch
    from torch.utils.data import DataLoader
    
    # Optimize DataLoader for GPU feeding
    dataloader = DataLoader(
        dataset,
        batch_size=32,
        num_workers=8,        # Match CPU cores
        pin_memory=True,      # Faster CPU-GPU transfer
        prefetch_factor=4,    # Prefetch batches
        persistent_workers=True  # Reuse worker processes
    )
    
    # Profile data loading bottlenecks
    with torch.profiler.profile(
        activities=[
            torch.profiler.ProfilerActivity.CPU,
            torch.profiler.ProfilerActivity.CUDA
        ]
    ) as prof:
        for batch in dataloader:
            output = model(batch.cuda())
            
    print(prof.key_averages().table(sort_by="cuda_time_total"))
    

    Asynchronous Operations

    # Overlap CPU and GPU work
    stream = torch.cuda.Stream()
    
    for batch in dataloader:
        # CPU preprocessing on main thread
        processed = preprocess(batch)
        
        # GPU work on separate stream (overlapped)
        with torch.cuda.stream(stream):
            gpu_batch = processed.cuda(non_blocking=True)
            output = model(gpu_batch)
            loss = criterion(output)
            loss.backward()
        
        # Synchronize only when needed
        stream.synchronize()
        optimizer.step()
    

    Storage Optimization

    NVMe Configuration

    # RAID 0 multiple NVMe drives for dataset storage
    mdadm --create /dev/md0 --level=0 --raid-devices=4 \
      /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
    
    # Mount with optimizations
    mount -o noatime,nodiratime,discard /dev/md0 /data
    

    Dataset Caching

    # Cache preprocessed data in RAM
    from functools import lru_cache
    import torch
    
    @lru_cache(maxsize=10000)
    def load_cached_sample(idx):
        return torch.load(f"/data/samples/{idx}.pt")
    
    # Or use memory-mapped datasets
    import numpy as np
    mmap_data = np.memmap('/data/dataset.npy', dtype='float32', 
                          mode='r', shape=(1000000, 224, 224, 3))
    

    Key Takeaways

    • •BIOS optimization disables power-saving features and maximizes PCIe performance
    • •NUMA-aware CPU pinning prevents cross-NUMA memory access penalties
    • •Optimized DataLoader with pin_memory and persistent workers eliminates CPU bottlenecks
    • •NVMe RAID and dataset caching reduce storage I/O impact on training
    0 of 4 lessons marked complete

    Progress is kept in this browser only.