Inference: use the weights without building an update

Inference runs a trained model to produce predictions, representations or generated outputs. For a language model, it includes processing the prompt and repeatedly choosing new tokens.

Training additionally stores information for differentiation and updates the parameters. Inference can avoid that gradient bookkeeping, but still needs weights, activations and any retained context state.

Two switches with different jobs

This PyTorch helper takes a Transformers-style causal model and tokenized inputs and selects one next-token ID per batch item.

import torch

def next_token(model, inputs):
    model.eval()
    with torch.no_grad():
        logits = model(**inputs).logits[:, -1, :]
    return logits.argmax(dim=-1)

eval() changes the behaviour of layers such as dropout. no_grad() disables gradient recording. Neither operation substitutes for the other, and neither removes the model's stored weights.

A multi-step generation loop can reuse KV state instead of recomputing the whole prompt. Request batching, context length and cache reuse then affect both latency and memory. The same checkpoint can therefore have very different serving costs under different workloads.