AI Model Wiki

    Skip to main content
    25 min
    Lesson 4 of 5

    Hands-On Tutorial: Multi-Node Training Job

    From the course GPU Cluster Management

    This tutorial walks through a complete, runnable multi-node distributed training setup on a GPU cluster — from environment bootstrap to job submission and result verification.

    Prerequisites

    • Cluster with at least 2 nodes, each with 2+ NVIDIA GPUs
    • NCCL 2.18+, PyTorch 2.2+, CUDA 12.1+
    • Shared filesystem mounted at /shared on all nodes (NFS or Lustre)
    • Passwordless SSH between nodes (or use the MPI hostfile approach)

    Step 1: Verify Inter-Node GPU Communication

    Before running any training, confirm NCCL can communicate across nodes:

    # Run on node-0; node-1 must be reachable via SSH
    # Install nccl-tests if not present
    git clone https://github.com/NVIDIA/nccl-tests.git && cd nccl-tests
    make MPI=1 MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi CUDA_HOME=/usr/local/cuda
    
    # Run all-reduce bandwidth test across 2 nodes, 4 GPUs each
    mpirun -np 8 -H node-0:4,node-1:4   -x NCCL_DEBUG=INFO   -x NCCL_IB_DISABLE=0   ./build/all_reduce_perf -b 8 -e 256M -f 2 -g 4
    
    # Expected: bus bandwidth > 150 GB/s on InfiniBand HDR
    

    If bandwidth is below 50 GB/s, check IB link state with ibstat and verify NCCL environment variables.

    Step 2: Prepare the Training Script

    Save this as /shared/train_resnet.py:

    import os
    import torch
    import torch.distributed as dist
    from torch.nn.parallel import DistributedDataParallel as DDP
    from torchvision import models, datasets, transforms
    from torch.utils.data import DataLoader, DistributedSampler
    
    def main():
        # Read distributed environment from torchrun
        local_rank = int(os.environ["LOCAL_RANK"])
        rank = int(os.environ["RANK"])
        world_size = int(os.environ["WORLD_SIZE"])
    
        dist.init_process_group(backend="nccl")
        torch.cuda.set_device(local_rank)
        device = torch.device(f"cuda:{local_rank}")
    
        # Model — wrap in DDP
        model = models.resnet50().to(device)
        model = DDP(model, device_ids=[local_rank])
    
        # Dataset with distributed sampler
        transform = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
        ])
        dataset = datasets.FakeData(size=10000, transform=transform)
        sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank)
        loader = DataLoader(dataset, batch_size=64, sampler=sampler, num_workers=4)
    
        optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
        criterion = torch.nn.CrossEntropyLoss()
    
        for epoch in range(5):
            sampler.set_epoch(epoch)
            for images, labels in loader:
                images, labels = images.to(device), labels.to(device)
                optimizer.zero_grad()
                outputs = model(images)
                loss = criterion(outputs, labels)
                loss.backward()
                optimizer.step()
    
            if rank == 0:
                print(f"Epoch {epoch} complete")
                # Save checkpoint only on rank 0
                torch.save(model.module.state_dict(),
                           f"/shared/checkpoints/resnet50_epoch{epoch}.pt")
    
        dist.destroy_process_group()
    
    if __name__ == "__main__":
        main()
    

    Step 3: Launch with torchrun

    # On node-0 — torchrun handles rendezvous across nodes
    torchrun   --nproc_per_node=4   --nnodes=2   --node_rank=0   --master_addr=node-0   --master_port=29500   /shared/train_resnet.py
    
    # On node-1 simultaneously
    torchrun   --nproc_per_node=4   --nnodes=2   --node_rank=1   --master_addr=node-0   --master_port=29500   /shared/train_resnet.py
    

    Step 4: Submit via Slurm (Production Approach)

    #!/bin/bash
    #SBATCH --job-name=resnet50-dist
    #SBATCH --nodes=2
    #SBATCH --ntasks-per-node=4
    #SBATCH --gres=gpu:4
    #SBATCH --time=02:00:00
    #SBATCH --output=/shared/logs/%j.out
    
    # Slurm sets SLURM_PROCID, SLURM_NTASKS automatically
    srun torchrun   --nproc_per_node=4   --nnodes=2   --rdzv_id=${SLURM_JOB_ID}   --rdzv_backend=c10d   --rdzv_endpoint=${SLURM_NODELIST%%,*}:29500   /shared/train_resnet.py
    

    Submit with sbatch train_job.sh and monitor with squeue -u ${USER}.

    Step 5: Verify Results

    # Check checkpoint files were written
    ls -lh /shared/checkpoints/
    
    # View training throughput from logs
    grep "Epoch" /shared/logs/<job_id>.out
    
    # Profile GPU utilization during training
    nvidia-smi dmon -s u -d 5  # sample utilization every 5 seconds
    

    A healthy 2-node, 8-GPU ResNet-50 run should sustain 85–95% GPU utilization and complete 5 epochs of FakeData in under 3 minutes.

    Key Takeaways

    • •Always validate NCCL inter-node bandwidth with nccl-tests before running real training jobs
    • •torchrun handles the rendezvous protocol — set master_addr, master_port, node_rank consistently across all nodes
    • •Save checkpoints only from rank 0 to avoid file-write conflicts on the shared filesystem
    • •Slurm with c10d rendezvous backend is the production-grade way to submit multi-node jobs at scale
    0 of 5 lessons marked complete

    Progress is kept in this browser only.