Sparse mixture of experts (sparse MoE) is a neural network architecture that replaces a single dense feed-forward block with a large collection of parallel sub-networks, called experts, plus a learned gating network that routes each input token to only a small subset—typically two out of dozens or hundreds—of those experts. Because only a fraction of experts activate per token, the model can have an enormous total parameter count while keeping per-token floating-point operations roughly constant, decoupling model capacity from compute cost. Mixtral, GPT-4 (reportedly), and Switch Transformer are prominent examples of this design.

Think of it like a hospital where each patient is triaged and sent to exactly two specialists out of a hundred on staff. The hospital has tremendous total expertise, but no patient ever needs all hundred specialists at once—so the overhead per visit stays manageable even as the institution's collective knowledge grows.

At inference time, the router computes a softmax over expert scores and selects the top-k: router_logits = token_embedding @ router_weight; top_k_indices = router_logits.topk(2).indices. Only those two experts process the token; their outputs are weighted and summed. A load-balancing auxiliary loss encourages the router to distribute tokens evenly rather than collapsing to a few popular experts.

Sparse MoE is an extension of Mixture of Experts and sits within the broader Transformer and Neural Network ecosystem; it interacts closely with Inference efficiency concerns, Quantization, and Knowledge Distillation when deploying large models. You will encounter it in model cards for high-capacity open-weight models and in papers comparing compute-matched dense versus sparse architectures.