Mixture of Experts (MoE) is a neural network design principle in which a model is composed of multiple specialised sub-networks called experts, plus a learned routing component that decides which experts to activate for any given input. Rather than passing every token through every parameter in the network, the router selects a small subset of experts — typically two out of dozens or hundreds — so the total parameter count can be enormous while the compute per forward pass remains manageable. This decoupling of model capacity from inference cost is the central advantage of MoE architectures.

Think of it like a hospital with specialist departments: instead of every patient seeing every doctor, a triage nurse (the router) directs each case to the two or three most relevant specialists, keeping the overall system highly capable without overwhelming any one expert or the overall budget.

In practice, a Transformer feed-forward layer can be replaced with an MoE layer. A sparse MoE model like Mixtral 8x7B has eight expert feed-forward networks per layer; the router sends each token to the top-2 experts by learned score:

Token -> Router (softmax over 8 experts)
      -> Expert 3 (weight 0.62) + Expert 7 (weight 0.38)
      -> Weighted sum of outputs -> next layer

DeepSeek V3, GPT-4, and Mixtral are prominent production models that use MoE layers to achieve high benchmark scores at lower per-token inference cost than equivalently capable dense models.

MoE is closely related to Softmax (used in the router), Transformer (the typical host architecture), Inference efficiency, and the concept of a Foundation Model that needs to generalise across many domains simultaneously.

You will encounter Mixture of Experts most often when reading about frontier model architectures, efficiency-focused research, and when choosing between dense and sparse models for deployment at scale.