Perplexity: how surprising was the real continuation?

Perplexity measures how well a language model predicts observed tokens. Take each actual next token's negative log-probability, average those values, then exponentiate. Lower perplexity means the model assigned higher probability to the observed continuation on that evaluation setup.

A model assigning every observed next token probability one quarter has perplexity four. Uniform predictions over a vocabulary of a thousand tokens give perplexity one thousand. These examples explain the scale; real distributions need not be uniform.

Score the next position, not the current one

This NumPy function accepts logits of shape tokens × vocabulary from a causal model and the corresponding token IDs for one unpadded sequence of at least two tokens. Position zero predicts token one, so the arrays are shifted before scoring. The final logits row has no observed next token here and is excluded.

import numpy as np

def perplexity(logits, token_ids):
    scores = logits[:-1]
    targets = token_ids[1:]
    shifted = scores - scores.max(axis=1, keepdims=True)
    log_normalizer = np.log(np.exp(shifted).sum(axis=1))
    losses = log_normalizer - shifted[np.arange(len(targets)), targets]
    return float(np.exp(losses.mean()))

The stable log-normalizer avoids explicitly forming tiny probabilities. This is a scoring calculation: it does not sample a continuation or update model weights.

Compare like with like

Tokenization changes what is being predicted and how many predictions count in the average. Dataset choice and the context available for each prediction also change the score. Comparing models requires compatible tokenization and evaluation conditions, or a metric normalized to a shared unit.

For text beyond the model's context window, overlapping evaluation windows can give predictions more preceding context than disjoint chunks. That can materially change the reported result.

Perplexity measures prediction on the chosen text. It does not directly score instruction following, factual accuracy or whether an answer helped a person.