Prefill and decode

Plain English. Inference has two phases. Prefill is the model reading your input — highly parallel, compute-hungry. Decode is it writing the answer one token at a time — limited by how fast memory can feed the chip. They stress hardware in opposite ways.

Why it moves money. Because the two phases want different silicon, operators can split their fleets ("disaggregation") and vendors can sell decode-specialised hardware — a whole product category rides on the split. It also shapes the local-inference debate: consumer machines lack datacentre memory bandwidth, and prefill is where they hurt.

What to watch. Whether big operators actually disaggregate — OpenAI has publicly declined, arguing production workloads drift too much for purpose-tuned pools — and whether decode-specialised deployments (SRAM-heavy chips) win real volume.

From the signals. OpenAI declined to split its inference fleet, and said why: traffic ratios don't hold still. Nvidia put Groq 3 LPX racks — 256 SRAM-heavy chips per rack — into production for decode workloads. Mark's local-inference logs found the same asymmetry: prefill is expensive at home.