Normalization: control the scale of a token vector

Normalization changes the scale, and sometimes the centre, of intermediate representations. In Transformers, it helps keep the numerical behaviour of a deep stack manageable as activations pass through many residual updates.

LayerNorm subtracts a vector's mean and scales by its standard deviation. RMSNorm scales by the root mean square without subtracting the mean. Both can include learned per-feature scale parameters.

Normalize features, not neighbouring tokens

For an array of token vectors, normalization usually works along the final feature axis. Each token gets its own normalization statistics. It does not average a sentence into one vector or mix information across positions.

import numpy as np

def rms_norm(x, gain, eps=1e-6):
    scale = np.sqrt(np.mean(x * x, axis=-1, keepdims=True) + eps)
    return x / scale * gain

Here x ends in a feature dimension, gain has one value per feature and eps keeps a zero vector from causing division by zero. The scale statistics are computed from the current activation; they are not a lookup of facts learned during training.

Multiplying a vector by a positive constant approximately disappears under RMS normalization, apart from the stabilizing epsilon. Adding a constant does not: RMSNorm deliberately keeps a different invariance from LayerNorm.

Placement changes the training path

Putting normalization before an operation or after a residual addition changes how activations and gradients travel through the stack. Names such as pre-norm and post-norm describe that placement, not a different way to tokenize text.

Normalization stabilizes the computation around attention and FFNs. It does not replace either operation's learned transformation.