Mixture of experts (MoE)

Plain English. An architecture where the model holds many specialist sub-networks ("experts") but routes each token through only a few of them. A model can therefore be enormous on paper while doing a fraction of the work per token — total parameters set what it can know; active parameters set what each token costs.

Why it moves money. MoE is the trick behind cheap frontier-adjacent models: near-frontier quality at a fraction of the serving compute. It also makes headline parameter counts actively misleading — a "552B model" running 30B active parameters per token has the cost profile of a much smaller model but the memory footprint of a huge one, which shapes which hardware can serve it and whose margins survive. Comparing models by total parameters is the retail error; the professionals compare active parameters and memory.

What to watch. Active-parameter counts in model cards, and the dense-versus-MoE crossover: cases where well-trained dense models beat far larger sparse ones tell you routing is not free.

From the signals. DeepSeek released V4.1-Flash: 552B MoE, 1M context, MIT-licensed weights. Qwen3.6-27B, a dense model, outperformed a 397B MoE on coding. Qwen 3.6 approached the frontier on 3B active parameters.

Further reading. Shazeer et al., Outrageously Large Neural Networks (2017)