Positional encoding: give tokens a sense of order

A token embedding identifies content. Positional encoding adds information about where that content occurs. Without some source of order information, attention's similarity calculation alone cannot distinguish all the arrangements a language model needs to understand.

The original Transformer added position-dependent sine and cosine vectors to embeddings. Other models learn position vectors. Rotary position embeddings (RoPE) instead rotate query and key components before attention compares them.

Rotate pairs, preserve their length

This NumPy function takes floating-point vectors of shape tokens × head_width, with an even width and one position per row. It rotates adjacent component pairs at different frequencies. Apply the same rule separately to queries and keys.

import numpy as np

def rope(vectors, positions, base=10_000.0):
    width = vectors.shape[1]
    frequencies = base ** (-np.arange(0, width, 2) / width)
    angles = np.asarray(positions)[:, None] * frequencies
    cosine, sine = np.cos(angles), np.sin(angles)
    even, odd = vectors[:, 0::2], vectors[:, 1::2]
    rotated = np.empty_like(vectors)
    rotated[:, 0::2] = even * cosine - odd * sine
    rotated[:, 1::2] = even * sine + odd * cosine
    return rotated

For one pair, position determines the rotation angle. When both query and key positions move by the same offset, their relative rotation stays unchanged. Their dot product therefore carries relative-position information while the rotation preserves each vector's length.

Multiple pairs rotate at different rates, giving the representation several positional scales. Actual models can choose different frequency bases, component layouts or scaling rules.

Position rules affect context extension

A model's trained position range and its behaviour beyond that range matter. Changing a RoPE scale can permit longer inputs, but a larger accepted sequence is not by itself evidence of reliable long-context performance.

Position information and a causal mask do different jobs. Position information represents order; the mask controls which positions may be read. Both matter in a causal Transformer.