Tensor parallelism is a distributed training and inference strategy that partitions the weight matrices — tensors — of a neural network's layers across multiple GPUs, so that each device holds and computes only a slice of each matrix multiplication rather than the full operation. Because modern Transformer layers contain weight matrices with billions of Parameters, splitting them column-wise or row-wise across devices allows models that far exceed a single GPU's memory to be executed, with results communicated and merged via fast interconnects such as NVLink.

Think of it like a team of bank tellers each handling a different subset of account numbers simultaneously. No single teller needs to know every account; together they process the full ledger in the same time it would take one teller to handle their slice alone.

# Megatron-LM tensor-parallel launch (conceptual)
# Each rank receives a column partition of the attention projection
torchrun --nproc_per_node=8 pretrain_gpt.py   --tensor-model-parallel-size 8   --pipeline-model-parallel-size 1

Here eight GPUs each hold one-eighth of every weight matrix; during a forward pass each device computes its partial result and an all-reduce synchronises the outputs before the next layer.

Tensor parallelism is one pillar of the "3D parallelism" stack — alongside pipeline parallelism (splitting layers across devices) and data parallelism (splitting the training batch) — used to train and serve the largest LLMs and Foundation Models. It is closely related to Quantization and Inference optimisation: both aim to reduce per-device memory pressure. Mixture of Experts architectures introduce a separate but complementary form of parallelism by routing tokens to specialist sub-networks.

You will encounter tensor parallelism when reading about the training infrastructure for models like GPT-4 or Llama 3, or when configuring multi-GPU serving with frameworks such as vLLM or TensorRT-LLM.