25 min
Lesson 4 of 5
Hands-On Tutorial: Multi-Node Training Job
From the course GPU Cluster Management
This tutorial walks through a complete, runnable multi-node distributed training setup on a GPU cluster — from environment bootstrap to job submission and result verification.
Prerequisites
- Cluster with at least 2 nodes, each with 2+ NVIDIA GPUs
- NCCL 2.18+, PyTorch 2.2+, CUDA 12.1+
- Shared filesystem mounted at
/sharedon all nodes (NFS or Lustre) - Passwordless SSH between nodes (or use the MPI hostfile approach)
Step 1: Verify Inter-Node GPU Communication
Before running any training, confirm NCCL can communicate across nodes:
# Run on node-0; node-1 must be reachable via SSH
# Install nccl-tests if not present
git clone https://github.com/NVIDIA/nccl-tests.git && cd nccl-tests
make MPI=1 MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi CUDA_HOME=/usr/local/cuda
# Run all-reduce bandwidth test across 2 nodes, 4 GPUs each
mpirun -np 8 -H node-0:4,node-1:4 -x NCCL_DEBUG=INFO -x NCCL_IB_DISABLE=0 ./build/all_reduce_perf -b 8 -e 256M -f 2 -g 4
# Expected: bus bandwidth > 150 GB/s on InfiniBand HDR
If bandwidth is below 50 GB/s, check IB link state with ibstat and verify NCCL environment variables.
Step 2: Prepare the Training Script
Save this as /shared/train_resnet.py:
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torchvision import models, datasets, transforms
from torch.utils.data import DataLoader, DistributedSampler
def main():
# Read distributed environment from torchrun
local_rank = int(os.environ["LOCAL_RANK"])
rank = int(os.environ["RANK"])
world_size = int(os.environ["WORLD_SIZE"])
dist.init_process_group(backend="nccl")
torch.cuda.set_device(local_rank)
device = torch.device(f"cuda:{local_rank}")
# Model — wrap in DDP
model = models.resnet50().to(device)
model = DDP(model, device_ids=[local_rank])
# Dataset with distributed sampler
transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
])
dataset = datasets.FakeData(size=10000, transform=transform)
sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank)
loader = DataLoader(dataset, batch_size=64, sampler=sampler, num_workers=4)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
criterion = torch.nn.CrossEntropyLoss()
for epoch in range(5):
sampler.set_epoch(epoch)
for images, labels in loader:
images, labels = images.to(device), labels.to(device)
optimizer.zero_grad()
outputs = model(images)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
if rank == 0:
print(f"Epoch {epoch} complete")
# Save checkpoint only on rank 0
torch.save(model.module.state_dict(),
f"/shared/checkpoints/resnet50_epoch{epoch}.pt")
dist.destroy_process_group()
if __name__ == "__main__":
main()
Step 3: Launch with torchrun
# On node-0 — torchrun handles rendezvous across nodes
torchrun --nproc_per_node=4 --nnodes=2 --node_rank=0 --master_addr=node-0 --master_port=29500 /shared/train_resnet.py
# On node-1 simultaneously
torchrun --nproc_per_node=4 --nnodes=2 --node_rank=1 --master_addr=node-0 --master_port=29500 /shared/train_resnet.py
Step 4: Submit via Slurm (Production Approach)
#!/bin/bash
#SBATCH --job-name=resnet50-dist
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --time=02:00:00
#SBATCH --output=/shared/logs/%j.out
# Slurm sets SLURM_PROCID, SLURM_NTASKS automatically
srun torchrun --nproc_per_node=4 --nnodes=2 --rdzv_id=${SLURM_JOB_ID} --rdzv_backend=c10d --rdzv_endpoint=${SLURM_NODELIST%%,*}:29500 /shared/train_resnet.py
Submit with sbatch train_job.sh and monitor with squeue -u ${USER}.
Step 5: Verify Results
# Check checkpoint files were written
ls -lh /shared/checkpoints/
# View training throughput from logs
grep "Epoch" /shared/logs/<job_id>.out
# Profile GPU utilization during training
nvidia-smi dmon -s u -d 5 # sample utilization every 5 seconds
A healthy 2-node, 8-GPU ResNet-50 run should sustain 85–95% GPU utilization and complete 5 epochs of FakeData in under 3 minutes.
Key Takeaways
- •Always validate NCCL inter-node bandwidth with nccl-tests before running real training jobs
- •torchrun handles the rendezvous protocol — set master_addr, master_port, node_rank consistently across all nodes
- •Save checkpoints only from rank 0 to avoid file-write conflicts on the shared filesystem
- •Slurm with c10d rendezvous backend is the production-grade way to submit multi-node jobs at scale
0 of 5 lessons marked complete
Progress is kept in this browser only.