Speculative decoding

Plain English. A small, fast "draft" model proposes the next several tokens; the big model checks them all in one pass, keeping the ones it agrees with. Output is identical to what the big model would have written alone — it just arrives faster, because agreement is common and checking is cheaper than generating.

Why it moves money. It is close to pure margin expansion for inference providers: the same hardware serves more tokens with no quality trade. Google reports up to a 3x speedup from its Multi-Token Prediction drafters at zero quality cost; an independent benchmark tripled throughput on a single rented B200 largely by turning speculative settings on. Gains like these are why serving costs keep falling faster than hardware improves — and why "cost per token" quotes age quickly.

What to watch. Whether acceptance rates hold up on long agentic workloads, and how much of the saving providers pass through to price versus keep as margin.

From the signals. Gemma 4 shipped Multi-Token Prediction drafters — a reported 3x speedup at zero quality cost. One rented B200 and four settings tripled the tokens.

Further reading. Leviathan et al., "Fast Inference from Transformers via Speculative Decoding" (2022).