Data parallelism is a distributed training strategy in which the full model is replicated on each device (GPU or TPU), but each replica processes a different mini-batch of training data in parallel. After each forward and backward pass, the locally computed gradients are aggregated — usually by averaging — across all replicas via an all-reduce operation, and each replica updates its weights identically. The result is effectively the same as training on a larger batch, with wall-clock time reduced roughly in proportion to the number of devices.

Think of it like a team of workers each reading a different chapter of the same book simultaneously, then meeting to combine their notes before everyone updates their identical summary. Each worker operates independently, but the final knowledge update is shared.

Using PyTorch's DistributedDataParallel, the setup looks like this:

import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

dist.init_process_group(backend="nccl")
model = DDP(MyModel().cuda(), device_ids=[local_rank])
# Each process receives a different data shard via DistributedSampler

Communication overhead from the all-reduce step is the main bottleneck; high-bandwidth interconnects like NVLink reduce this cost significantly.

Data parallelism is often contrasted with model parallelism, which splits the model itself across devices rather than the data. For very large Neural Networks whose parameters do not fit on a single device, practitioners combine both approaches, sometimes alongside techniques like DeepSpeed ZeRO. Understanding Gradient Descent is prerequisite to understanding why gradient averaging works here.

You will encounter data parallelism in any serious multi-GPU training setup, and it is the default strategy in most deep learning frameworks when scaling from one GPU to many.