Decoding: one model can produce different kinds of continuation

Decoding turns a model's output scores into a sequence. Greedy decoding selects the highest-scoring token at each step; sampling draws from a distribution; beam search keeps several partial sequences.

These procedures use the same learned model differently. A change in output quality can come from the selection rule rather than retraining.

Compare two policies on the same checkpoint

This PyTorch/Transformers example downloads a small causal model and tokenizer, then generates both continuations from the same prompt.

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "HuggingFaceTB/SmolLM2-135M"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name).eval()
inputs = tokenizer("Far away, the old observatory", return_tensors="pt")
greedy = model.generate(**inputs, max_new_tokens=40, do_sample=False)
sampled = model.generate(**inputs, max_new_tokens=40, do_sample=True,
                         temperature=0.8, top_p=0.9)
continuations = [tokenizer.decode(row, skip_special_tokens=True)
                 for row in (greedy[0], sampled[0])]

Greedy selection is locally deterministic for fixed scores. It does not guarantee the highest-probability complete sequence. Sampling introduces variation, while temperature and top-p change the distribution being sampled.

EOS tokens and length limits determine when generation ends. Compare methods with the same prompt and output budget; otherwise a longer answer can look better simply because it received more computation.