Mixture of experts: many weights, a few active

A mixture of experts (MoE) model routes work to selected neural networks called experts. In a sparse Transformer MoE layer, a learned router chooses a small number of expert feed-forward networks for each token. Attention and other parts of the model can remain shared.

This separates total parameters from active parameters. The model can hold many experts while evaluating only a few for one token. A different token can take a different route. An expert is a learned computation; the name does not promise a human-readable speciality such as mathematics or history.

Route, evaluate, combine

This NumPy function implements one token's top-two routing. router maps its input vector to one score per expert. Only the selected expert functions run; their outputs are combined with weights normalized over that selected set. Each expert returns a vector of the same shape.

import numpy as np

def moe_token(x, router, experts, k=2):
    scores = x @ router
    selected = np.argpartition(scores, -k)[-k:]
    weights = np.exp(scores[selected] - scores[selected].max())
    weights /= weights.sum()
    return sum(weight * experts[index](x)
               for weight, index in zip(weights, selected))

Real implementations process batches of tokens and move them to the devices holding the selected experts. Routing formulas vary; this example makes selection and the weighted combination visible.

Sparse computation still needs memory

Inactive experts skip computation for this token, but their weights must remain available for other tokens. Total parameter count therefore matters for storage and serving memory even when active compute is much smaller.

Routing also has a hardware consequence: if many tokens choose the same expert, its device can become a bottleneck. Training encourages balanced use, and distributed systems must exchange token representations between devices. These costs explain why multiplying the number of experts does not translate into free model capacity.

MoE changes which feed-forward networks run. Attention decides how token positions exchange information. They are complementary operations inside a Transformer.