Batches: more examples behind one update

A batch groups examples for a computation or training update. During training, gradients from its targets are combined before the optimizer changes the parameters. Processing several examples together also gives matrix operations more parallel work.

A larger batch does not mean a longer context for one example. Batch size and sequence length are separate dimensions, although both affect memory and total token count.

Accumulate pieces of a larger batch

Gradient accumulation processes smaller microbatches, adds their gradients and performs one optimizer step afterward. This PyTorch example averages over valid target tokens across the whole effective batch. microbatches is a list of input/target tensor pairs; the model returns aligned logits, and target -100 means ignore that position.

import torch.nn.functional as F

def accumulated_step(model, optimizer, microbatches):
    total = sum((y != -100).sum().item() for _, y in microbatches)
    optimizer.zero_grad()
    for inputs, targets in microbatches:
        logits = model(inputs)
        loss = F.cross_entropy(
            logits.reshape(-1, logits.shape[-1]), targets.reshape(-1),
            reduction="sum", ignore_index=-100)
        (loss / total).backward()
    optimizer.step()

Using the sum of each microbatch's losses and the shared token denominator matters when microbatches contain different numbers of valid targets. Averaging their separate means equally would give short and long microbatches equal influence.

Same objective, different execution

The combined gradient represents the chosen mean-token objective. Batch-dependent layers and stochastic operations can still make execution differ from a single larger forward pass. Accumulation also requires multiple passes, so it saves peak activation memory rather than making the work disappear.

In serving, batching instead combines requests to use hardware efficiently. Their histories and outputs remain separate. A server can add new requests as others finish; the useful unit is active work, not a permanent group of users.