Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural network design that increases total parameter count while keeping per-token compute nearly constant. An MoE layer replaces a dense feed-forward network with multiple parallel 'expert' MLPs and a gating mechanism that routes each token to only a few top experts (e.g., top-2). This means a 1 trillion parameter MoE model might only activate 40 billion parameters per token, achieving much higher capacity for modeling diverse patterns than a dense model of similar FLOPs. The gating network learns to specialize experts-some handle programming code, others natural language, math, or rare facts. MoE is used in landmark models like Mixtral 8x7B, GPT-4, and Gemini. Training MoE is challenging due to load balancing: experts can become undertrained if the gate assigns too few tokens to some. Auxiliary losses (e.g., importance loss, z-loss) encourage uniform routing. Inference also presents memory challenges: all expert weights must be loaded (e.g., 8x 7B = 50B+ parameters in Mixtral), but only a subset compute per token. This drives demand for high PCIe bandwidth (nvlink) and memory capacity. Quantization (e.g., FP8, INT4) is often applied to fit MoE models on single GPUs. Recent advances include fine-grained MoE (DeepSeek-V2) and shared+specialized experts. For AI builders, MoE offers a sweet spot between quality and speed, but requires careful memory planning and batching strategies to keep experts saturated. A common mistake is thinking MoE models are 'free' capacity-they still need proportional memory for weights.