Prefill: processing the prompt before the answer

Prefill is the stage in which a causal language model processes an input sequence and prepares the KV cache used for generation. The final input position produces scores for the first output token. Later decoding steps extend that cached sequence.

The prompt tokens are already known, so many of their operations can run in parallel. Causal attention still prevents a position from reading later positions. Parallel processing does not mean the model can see the future.

One wide pass, then repeated small steps

A long prompt gives matrix multiplications many token positions to work on at once. During ordinary autoregressive decoding, each individual sequence instead advances one position at a time. Servers batch different requests together to keep the hardware busy.

This difference helps explain why input and output tokens can have different prices. Both use the model's weights, but the opportunities for parallel work differ. A token price is still a service price, rather than a direct measurement of hardware work.

Why the first token takes time

Time to first token includes queuing, input processing and the first token's generation. A long uncached prompt can make prefill a substantial part of that wait. Reusing a matching cached prefix reduces the amount of new input processing.

Serving systems may split prefill into chunks or run prefill and decoding on different devices. That changes scheduling and data movement while preserving the underlying task: turn the known input into representations that generation can extend.