Mixture of Experts (MoE) is a neural network architecture in which a model contains multiple specialised sub-networks (experts) and a trainable router that selects a small subset of those experts to process each input token. Only the activated experts contribute to the forward pass, so a model can have hundreds of billions of total parameters while computing with only a fraction of them per token — achieving the representational capacity of a large dense model at a fraction of the inference cost. The router typically uses a Softmax over expert scores and selects the top-k (most commonly top-2) experts per token.

Think of it like a specialist clinic where a receptionist quickly directs each patient to the two most relevant consultants out of a large roster: patients get expert-level care without every consultant being involved in every case.

Modern MoE models like Mixtral 8x7B and DeepSeek V3 insert MoE layers in place of standard feed-forward blocks inside a Transformer. Each layer independently routes tokens to its own set of experts:

Input token embedding
  -> Multi-head Self-Attention
  -> MoE Layer: router scores 8 experts, picks top-2
      -> Expert FFN_3 x 0.61 + Expert FFN_6 x 0.39
  -> Layer norm -> next block

Load-balancing auxiliary losses are added during training to prevent all tokens routing to the same expert, which would negate the efficiency gains.

MoE is directly related to Transformer (the standard host architecture), Softmax (routing mechanism), Inference cost analysis, and the broader goal of scaling Foundation Models efficiently. It also intersects with Quantization, since activating only a subset of experts makes quantised MoE models particularly economical to serve.

You will encounter Mixture of Experts when reading about frontier model architectures, comparing dense versus sparse models for deployment, or evaluating whether a given model's advertised parameter count reflects its actual inference compute budget.