Dataset: the examples and the way they reach the model

A dataset is a collection of examples together with a structure for accessing them. An example might be an image and label, or a token sequence whose shifted copy supplies the prediction targets.

A dataset abstraction need not mean a fully downloaded table. Map-style access can retrieve an item by index; iterable access produces a sequence and can stream data larger than RAM or disk.

Turn tokens into training pairs

This TensorFlow example uses a short illustrative token sequence. Replace it with a tokenized document to build causal language-model inputs and targets.

import tensorflow as tf

token_ids = tf.constant([11, 24, 8, 31, 7, 19, 6, 12, 4])
windows = tf.data.Dataset.from_tensor_slices(token_ids).window(
    5, shift=4, drop_remainder=True
)
training = (
    windows.flat_map(lambda window: window.batch(5))
    .map(lambda row: (row[:-1], row[1:]))
    .batch(8)
    .prefetch(tf.data.AUTOTUNE)
)

Each five-token window produces four input tokens and their four next-token targets. Advancing by four lets the boundary target become the next window's first input.

map, batch and prefetch describe the input pipeline rather than changing the model. Shuffling and filtering still matter: the order and mixture of examples determine what the optimizer sees. Keep document boundaries explicit when joining real text.