Decoder: the mask determines what a position can know

A decoder produces an output representation or sequence. In a causal language-model decoder, each position can use itself and earlier positions while predicting the next token.

The original encoder–decoder Transformer also gives the decoder cross-attention to encoder outputs. Decoder-only language models instead process the prompt and continuation in one causal sequence.

Enforce the information boundary

This NumPy example computes one attention head over three positions. Entries above the diagonal are blocked before softmax.

import numpy as np

rng = np.random.default_rng(7)
queries, keys, values = rng.normal(size=(3, 3, 4))
scores = queries @ keys.T / np.sqrt(4)
allowed = np.tril(np.ones((3, 3), dtype=bool))
scores = np.where(allowed, scores, -np.inf)
weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
weights /= weights.sum(axis=-1, keepdims=True)
output = weights @ values

The first output can read only the first value; the last can combine all three. Removing the mask would let early positions use later inputs during training, changing the prediction problem.

Known training tokens can still be processed in parallel because the mask enforces the dependency. During generation, an unknown next token must be selected before the following step can use it. The causal rule, not the word “decoder” alone, explains that difference.