Autoregressive generation: feed the choice back in

Autoregressive generation builds a sequence by predicting its next element from the elements already present. In a language model, those elements are tokens. After selecting one token, the system adds it to the sequence and asks the model for another prediction.

The model supplies a distribution of possible next tokens. A decoding rule decides which one to use. Choosing the largest score is greedy decoding; sampling introduces controlled randomness. Either way, the selected token becomes part of the next step's input.

The loop behind a continuation

This single-sequence example uses PyTorch and Transformers. The first pass reads the whole prompt. Subsequent passes feed only the newly selected token, together with the accumulated cache. The weights are reused unchanged. The first run downloads GPT-2.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
ids = tokenizer("Far far away", return_tensors="pt").input_ids
sequence, pending, cache = ids, ids, None

with torch.inference_mode():
    for _ in range(40):
        result = model(pending, past_key_values=cache, use_cache=True)
        pending = result.logits[:, -1].argmax(-1, keepdim=True)
        sequence = torch.cat([sequence, pending], dim=1)
        cache = result.past_key_values
        if pending.item() == tokenizer.eos_token_id:
            break

continuation = tokenizer.decode(sequence[0], skip_special_tokens=True)

An end-of-sequence token or the chosen step limit stops this loop. Greedy selection keeps the dependency between successive choices visible.

Parallel training, sequential choices

During training, the target sequence is already available. A causal mask allows predictions for many positions to be evaluated in parallel without exposing their future tokens. During ordinary generation, those future choices do not exist yet.

KV caching avoids recomputing earlier keys and values. It does not remove the dependence of the next choice on the previous choice. That dependence explains why a long answer can take time even when the prompt was processed quickly.