DeepSpeed ZeRO (Zero Redundancy Optimizer) is a memory optimization system developed by Microsoft that eliminates the redundant copies of model state that standard data parallelism creates. In conventional data parallelism, every GPU holds a full copy of the model Parameters, gradients, and optimizer states. ZeRO partitions these across all GPUs so that each device stores only its share, with communication collectives reassembling the needed tensors on demand. Three stages of increasing aggressiveness exist: ZeRO-1 partitions optimizer states only, ZeRO-2 adds gradient partitioning, and ZeRO-3 also partitions the model parameters themselves.

Think of it like a shared library rather than every reader owning every book. Instead of each worker keeping a full personal copy of a 10,000-volume collection, the books are distributed and borrowed when needed. The collection is just as accessible, but the storage cost per person drops dramatically.

Enabling ZeRO-3 in a Hugging Face Accelerate configuration looks like this:

{
  "zero_optimization": {
    "stage": 3,
    "offload_optimizer": {"device": "cpu"},
    "offload_param": {"device": "cpu"}
  }
}

CPU offload pushes optimizer states and parameters to host RAM when GPU memory is exhausted, allowing models larger than GPU VRAM to train without model parallelism.

ZeRO is part of the broader DeepSpeed library and is often combined with mixed-precision and gradient checkpointing. It interacts directly with Gradient Descent, since it restructures how gradients are accumulated and communicated. Knowledge Distillation is an alternative strategy to make large models trainable by creating smaller students, whereas ZeRO makes training the large teacher feasible in the first place.

You will encounter DeepSpeed ZeRO when fine-tuning or pretraining models with billions of parameters on clusters where individual GPU memory is the binding constraint.