KV cache

Plain English. As a model reads your prompt it builds a working state — the key-value (KV) cache — for every token it has seen, so it doesn't re-read the whole conversation to produce each new word. It is the model's short-term scratchpad, and it lives in scarce, expensive accelerator memory for as long as your session does. Long contexts mean big caches.

Why it moves money. The KV cache is why agentic AI is a memory business as much as a compute business. An agent that works for hours holds state the whole time, and whoever stores and re-serves that state cheapest keeps the margin. Providers now price it explicitly: cached input tokens cost a fraction of fresh ones, so cache economics flow straight into application unit costs.

What to watch. The cache-read line on API price lists, and hardware pitched on memory bandwidth and capacity rather than flops — both tell you where the constraint has moved.

From the signals. Stratechery named the memory-hierarchy unbundling: agentic inference needs context, state and history, not just token speed. Anthropic cut cache-read pricing 75%, taking typical workload costs down about 25% — a measured figure over a stated window.