Pretraining: the text supplies its own targets

Pretraining builds a model's starting capabilities from a broad training objective before a narrower adaptation stage. For a causal language model, the input text itself supplies next-token targets. A person does not need to label the correct next word separately for every position.

That is self-supervised learning: the training signal is constructed from the data. It does not mean the model receives no target, loss or human choices about the dataset.

One sequence becomes many predictions

For tokens A B C D, inputs A B C predict targets B C D. A causal mask stops each prediction from reading the future target. Training can evaluate all those positions together because the observed sequence is already available.

This TensorFlow step assumes an existing causal Keras model that returns logits and unpadded token rows. The optimizer updates the model's trainable weights.

import tensorflow as tf

def pretrain_step(model, optimizer, tokens):
    inputs, targets = tokens[:, :-1], tokens[:, 1:]
    with tf.GradientTape() as tape:
        logits = model(inputs, training=True)
        losses = tf.keras.losses.sparse_categorical_crossentropy(
            targets, logits, from_logits=True)
        loss = tf.reduce_mean(losses)
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    return loss

The shift creates the targets; the model architecture enforces which context each target may use. Leaving out either part changes the task, potentially making it trivial by exposing the answer.

The objective shapes the starting point

Learning next-token prediction can develop reusable representations and behaviours because many kinds of structure help predict text. It also learns patterns from unreliable or biased examples. Prediction success is not a direct guarantee of factual or helpful answers.

Instruction tuning and other post-training methods then adapt that starting point. Our Small Language Model experiment makes the pretraining ingredients visible with our own Transformer and a streaming corpus.