The expensive part of forgetting

A Transformer's KV cache holds the keys and values its attention mechanism needs from earlier tokens. Reusing them lets the model continue from old work. Prompt caching extends that saving across API calls with matching prefixes, as OpenAI's documentation describes.

Several things grow differently as context gets longer. KV storage grows linearly. Projections and feed-forward operations during prefill also grow linearly with token count. But in full causal attention, tokens interact with preceding tokens: a prompt of n tokens contains n(n + 1)/2 query–key pairs. Double its length and this attention work approaches four times the original amount.

This also explains why uncached input tokens can cost less than output tokens. During prefill, the input is already known, so the GPU can process many tokens together and reuse model weights efficiently. During generation, each new token depends on the previous ones: the model must run again, reading weights and the growing KV cache. Batching different users helps, but memory capacity and response latency limit how much work can be combined.

Now follow an agent that keeps extending its history. Start with one new token per call so the arithmetic is visible. Without reuse, each call rebuilds its entire prefix. Across T calls, the attention-pair count is:

Quadratic attention work, repeated over a growing history, accumulates cubically.

With KV reuse, earlier rows are already computed. At call t, only the new token needs to attend over t positions. Across the same run:

At 100 calls, rebuilding uses 171,700 pairs versus 5,050 with reuse: 34 times as much attention work. Fixed blocks of new tokens preserve the cubic-versus-quadratic cumulative growth. Sparse attention and compressed architectures can change that curve; this is our full-attention baseline.

Caching leaves one generated token's attention cost linear in the existing history. It avoids rebuilding all the previous work. A linearly growing memory allocation protects an increasingly large amount of computation.

A cache that dies before its next use

Keeping one context is straightforward. Keeping a population alive is the problem. While agent A waits for a test, B reads a file and C receives a browser result. Each will want its own state again.

A cache can save A's state, evict it to make room for B and C, then discover A has returned. Now the service repeats the work it just discarded. EfficientAgent's concurrent coding-agent experiments explore this working-set problem: host-cache capacity must cover the state agents reuse across their tool waits.

The useful hierarchy is GPU memory for active attention, host RAM for recently used contexts, and NVMe for colder state. NVIDIA Dynamo supports this kind of tiering. A backing copy can exist before the upper copy is evicted; freeing GPU space then need not wait for a new copy.

To put the transfer cost in perspective, take 288 GB, the memory capacity of one GPU in NVIDIA’s GB300 reference. Refilling that capacity takes roughly 1.5 seconds from RAM or 20 seconds from NVMe under our full-bandwidth, equally shared assumptions. That capacity and bandwidth may serve one agent or many.

Host RAM lets the service release GPU memory without discarding useful work. Across many interleaved agents, the state waiting for another call becomes a substantial resource in its own right.

Comparing OpenAI’s cached-input pricing with hardware rental costs from Verda, an affordable Finnish hyperscaler, gave us an estimate of about 2 MB of KV state per token, assuming a 30-second GPU lifetime and a two-minute RAM lifetime. Even with the longer lifetime, retaining the cache in RAM costs only about 30% as much as retaining it on the GPU.

Can AI become efficient enough to escape?

The obvious escape is to make remembering more efficient. Models can share KV across heads and layers, compress historical state, use fewer bits, or attend selectively. Shared prefixes let many requests reuse one physical copy of common instructions.

DeepSeek's V4.1 Flash combines compressed global state, cross-layer reuse, lower precision, and bounded reconstruction of selected state. Its technical report describes separate GPU-resident and persistent host/SSD caches. Better architecture can shrink both what must be stored and what must be moved.

But efficiency has a rival: how much work we ask the system to do. A tenfold reduction in memory per token is consumed by ten times as many retained tokens. Longer histories, more concurrent agents, and more delegated tasks can expand the working set as quickly as architecture shrinks each context.

New production expands supply. Better models stretch each gigabyte further. More ambitious workloads pull in the other direction. Whether RAM prices ease depends on which side moves faster.

AI's hunger for RAM is partly the price of avoiding an even greater hunger for computation.