Attention

Plain English. Attention is the mechanism inside a transformer that lets the model weigh every token against every other token to decide what matters. It is what gives a model its sense of context. The catch: comparing everything to everything is quadratic in the length of the input — double the context and the work roughly quadruples.

Why it moves money. That quadratic cost is the hidden meter behind inference bills. It is why long context windows are expensive, why the KV cache eats memory, and why serving a million-token prompt costs far more than a thousand-token one. Every trick that flattens attention's cost curve — sparse, linear and flash variants — translates directly into margin, and every provider chasing cheaper long context is really fighting this one equation.

What to watch. Whether the industry escapes the quadratic. Ballooning context windows and cheaper long-context models suggest engineering is winning at the margin; a true linear-cost replacement for attention would be a genuine architecture event, not an optimisation.

From the signals. DeepSeek shipped a 1M-token context window, the kind of scale attention's cost once forbade. A weekend debate over the context window as working memory is, underneath, an argument about attention.