Cross-entropy: make the observed token less surprising

Cross-entropy is a training objective that compares a predicted distribution with a target. With one correct class, it is the negative logarithm of the probability assigned to that class. A language model applies this to observed next tokens.

The same correct prediction can receive very different losses: probability 0.9 is rewarded more than probability 0.1. Confidently assigning almost no probability to the observed target incurs a large penalty.

The gradient reaches every candidate

This NumPy function accepts logits with shape examples × classes and one target ID per row. It returns mean loss and its derivative with respect to the logits.

import numpy as np

def cross_entropy(logits, targets):
    shifted = logits - logits.max(axis=1, keepdims=True)
    log_probs = shifted - np.log(np.exp(shifted).sum(axis=1, keepdims=True))
    rows = np.arange(len(targets))
    loss = -log_probs[rows, targets].mean()
    gradient = np.exp(log_probs)
    gradient[rows, targets] -= 1
    return loss, gradient / len(targets)

The gradient is the predicted probability minus the target indicator, divided by the number of examples. The correct class is pushed upward; competing classes are pushed downward in proportion to their current probabilities. An optimizer turns those derivatives into parameter updates through backpropagation.

Subtracting the maximum score makes the exponential calculation stable. Training libraries usually combine log-softmax and the loss internally; their cross-entropy interface commonly expects raw logits.

The target defines what improves

In next-token pretraining, the observed token supplies the target. In instruction tuning, the loss may include only selected response positions. The objective teaches whatever the chosen targets reward, including mistakes or biases present in them.

Perplexity exponentiates mean next-token cross-entropy. It changes the scale of the measurement, not the underlying predictions being evaluated.