Temperature: reshaping the next-token choice

Temperature changes how strongly generation favours the model’s higher-scoring tokens. It rescales the logits before softmax converts them into probabilities. The model weights stay the same.

Same scores, different probabilities

For positive temperature T, each logit is divided by T. Values below one amplify the differences between scores, concentrating probability on the leading candidates. Values above one flatten the distribution and give lower-scoring candidates more chance.

The candidate ranking stays unchanged: dividing by a positive number preserves order. What changes is the probability assigned to each candidate. Temperature therefore affects sampling, while always selecting the maximum score gives the same choice at any positive temperature.

Sample from a model

With torch and transformers installed, this performs one generation step using temperature 0.7:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
inputs = tokenizer("A strange light appeared", return_tensors="pt")

with torch.inference_mode():
    logits = model(**inputs).logits[:, -1]
    probabilities = torch.softmax(logits / 0.7, dim=-1)
    next_id = torch.multinomial(probabilities, num_samples=1).item()

continuation = tokenizer.decode([*inputs.input_ids[0].tolist(), next_id])

torch.multinomial draws from the rescaled distribution. Repeating the operation can produce different continuations even with identical model scores. To choose greedily instead, use the largest score directly.

Temperature is one part of decoding. Top-k and top-p can also restrict which candidates remain available, so a temperature value alone does not specify the whole generation procedure. A zero-temperature setting in an API usually requests a greedy-style choice; the division above requires a positive number.

Changing this setting redistributes existing predictions. It does not add knowledge to the model or directly measure whether an answer is correct.

Related: tokens, logits, top-p sampling, our browser model.