Feed-forward networks: work inside each token

In a Transformer, the feed-forward network (FFN) usually transforms each token's vector independently. Attention mixes information between positions; the FFN applies learned nonlinear computation to the representation at each position.

The same FFN weights are reused for every token in that layer. Shared weights do not mean identical outputs: different token representations enter the computation.

Expand, gate, project back

A common gated variant, SwiGLU, sends the input through two projections into a wider space. SiLU transforms one branch, which multiplies the other element by element. A third matrix projects the result back to the original width.

This NumPy function accepts token rows of width d. The gate and up matrices have shape d × h; the down matrix has shape h × d. Biases are omitted.

import numpy as np

def swiglu(x, gate, up, down):
    projected = x @ gate
    activated = projected / (1 + np.exp(-projected))
    return (activated * (x @ up)) @ down

The intermediate width gives the block more nonlinear features to combine. A gated block has three weight matrices instead of the two in a simple expand–activate–contract FFN, so architectures often adjust its hidden width to keep the parameter budget comparable.

Repeated work is still substantial

Every active token uses this transformation at every applicable layer. FFNs therefore account for a substantial share of many language models' parameters and arithmetic, even though they do not compare token positions themselves.

A mixture of experts can replace a single shared FFN with a selection among expert FFNs. That changes which parameters run for a token while keeping the distinction between cross-position attention and per-position transformation.