Softmax: turn competing scores into probabilities

Softmax converts a vector of real-valued scores into positive values that sum to one. In a language model, the scores are logits over the vocabulary. Softmax makes them a distribution that sampling can use to choose the next token.

Each probability is the exponential of its score divided by the sum of all the exponentials. Scores compete through that shared denominator: raising one score reduces the probability assigned to the others.

Keep the calculation stable

This NumPy function accepts one logits vector. Subtracting its largest score before exponentiating preserves the result while avoiding overflow from large positive values.

import numpy as np

def softmax(logits):
    shifted = logits - np.max(logits)
    weights = np.exp(shifted)
    return weights / weights.sum()

Adding the same constant to every score leaves the probabilities unchanged. What matters is the difference between scores. For two candidates, a score gap of about 0.69 gives odds of roughly two to one; a gap of about 2.30 gives odds of ten to one.

Temperature divides the scores before softmax. Lower temperature magnifies relative score differences; higher temperature compresses them. It changes the distribution without training new weights.

A distribution is not a fact check

A large next-token probability means that this token is favoured under the model and current context. It does not establish that the resulting statement is true.

Training commonly combines logits with cross-entropy using a numerically stable library operation. When an API expects logits, applying softmax first changes its input and can give the wrong objective. The probability view is useful for generation; the training interface determines which representation to supply.