25 min
Lesson 2 of 4
System-Level Optimization
From the course AI Hardware Optimization
Optimize the entire system stack from BIOS settings to kernel parameters for maximum AI performance.
BIOS and Firmware Optimization
Critical BIOS Settings
Power Management:
- Disable C-States (CPU sleep states)
- Set Performance Power Plan
- Disable CPU throttling
PCIe Configuration:
- Enable Above 4G Decoding
- Set PCIe to Gen 4/5 (max available)
- Enable Resizable BAR for better memory access
Memory:
- Enable XMP/DOCC profiles for RAM
- Set memory frequency to maximum stable
- Disable memory power-down modes
Linux Kernel Optimization
Kernel Parameters
# Add to /etc/sysctl.conf for AI workloads
vm.swappiness=10 # Minimize swap usage
vm.dirty_ratio=40 # Optimize write performance
vm.dirty_background_ratio=10
kernel.sched_latency_ns=10000000 # Reduce scheduling latency
kernel.sched_min_granularity_ns=2000000
NUMA Configuration
# Check NUMA topology
numactl --hardware
# Pin GPU to nearest CPU NUMA node
numactl --cpunodebind=0 --membind=0 python train.py
# For 8-GPU system, bind correctly
taskset -c 0-15 python train_gpu0.py # GPU 0 to CPU 0-15
taskset -c 16-31 python train_gpu1.py # GPU 1 to CPU 16-31
CPU-GPU Coordination
Data Loading Optimization
import torch
from torch.utils.data import DataLoader
# Optimize DataLoader for GPU feeding
dataloader = DataLoader(
dataset,
batch_size=32,
num_workers=8, # Match CPU cores
pin_memory=True, # Faster CPU-GPU transfer
prefetch_factor=4, # Prefetch batches
persistent_workers=True # Reuse worker processes
)
# Profile data loading bottlenecks
with torch.profiler.profile(
activities=[
torch.profiler.ProfilerActivity.CPU,
torch.profiler.ProfilerActivity.CUDA
]
) as prof:
for batch in dataloader:
output = model(batch.cuda())
print(prof.key_averages().table(sort_by="cuda_time_total"))
Asynchronous Operations
# Overlap CPU and GPU work
stream = torch.cuda.Stream()
for batch in dataloader:
# CPU preprocessing on main thread
processed = preprocess(batch)
# GPU work on separate stream (overlapped)
with torch.cuda.stream(stream):
gpu_batch = processed.cuda(non_blocking=True)
output = model(gpu_batch)
loss = criterion(output)
loss.backward()
# Synchronize only when needed
stream.synchronize()
optimizer.step()
Storage Optimization
NVMe Configuration
# RAID 0 multiple NVMe drives for dataset storage
mdadm --create /dev/md0 --level=0 --raid-devices=4 \
/dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
# Mount with optimizations
mount -o noatime,nodiratime,discard /dev/md0 /data
Dataset Caching
# Cache preprocessed data in RAM
from functools import lru_cache
import torch
@lru_cache(maxsize=10000)
def load_cached_sample(idx):
return torch.load(f"/data/samples/{idx}.pt")
# Or use memory-mapped datasets
import numpy as np
mmap_data = np.memmap('/data/dataset.npy', dtype='float32',
mode='r', shape=(1000000, 224, 224, 3))
Key Takeaways
- •BIOS optimization disables power-saving features and maximizes PCIe performance
- •NUMA-aware CPU pinning prevents cross-NUMA memory access penalties
- •Optimized DataLoader with pin_memory and persistent workers eliminates CPU bottlenecks
- •NVMe RAID and dataset caching reduce storage I/O impact on training
0 of 4 lessons marked complete
Progress is kept in this browser only.