Instruction tuning: teach which response to produce

Instruction tuning trains a model on requests paired with desired responses. It adapts a pretrained predictor toward following instructions, using examples of how to answer rather than relying only on a continuation of arbitrary text.

An example can contain a system message, a user question and an assistant response. A chat template turns those roles into a token sequence. The formatting tokens are part of what the model learns to interpret.

Read the prompt, score the response

A common objective trains only on assistant positions. User and system tokens still provide context, but their prediction losses are excluded. This makes the distinction between *using* a token as input and *scoring* it as a target explicit.

This function accepts a Transformers causal language model, token IDs for one unpadded conversation and a matching boolean mask from a conversation-aware data pipeline. The model shifts targets internally. The mask marks assistant target tokens, including any intended end-of-response marker.

def instruction_loss(model, token_ids, assistant_mask):
    labels = token_ids.masked_fill(~assistant_mask, -100)
    return model(input_ids=token_ids, labels=labels, use_cache=False).loss

Masking a target does not erase that token from the input. Later response positions can still attend to the prompt. An attention mask and a loss mask therefore solve different problems.

Examples specify behaviour

A dataset can teach concise answers, a structured format or how to handle particular requests. It can also teach undesired habits if examples are inconsistent. Evaluate on held-out instructions and compare with the starting model.

Instruction tuning is a form of fine-tuning. It differs from in-context learning, where examples guide a request without updating weights, and from preference training such as RLHF.