Tokenizer: the vocabulary is part of the model interface

A tokenizer converts text into the IDs a model consumes and decodes IDs back into text. Its pipeline can include normalization, pre-tokenization, subword segmentation and special-token processing.

The vocabulary defines what each ID means. Swapping tokenizers while keeping model weights fixed can make the embedding lookup interpret the same number as a different piece of text.

Train a tiny tokenizer, not a language model

Install tokenizers==0.22.2 for this example. It trains a byte-level BPE vocabulary on three in-memory strings; the complete byte alphabet lets it represent other text too.

from tokenizers import Tokenizer, decoders, pre_tokenizers, trainers
from tokenizers.models import BPE

texts = ["Small models learn patterns.", "Models learn from text.",
         "Small vocabularies split longer words."]
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tokenizer.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(
    vocab_size=300, special_tokens=["[UNK]"],
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
)
tokenizer.train_from_iterator(texts, trainer=trainer)
encoded = tokenizer.encode("Small models learn.")
restored = tokenizer.decode(encoded.ids)

The learned merges determine which byte sequences become single tokens. They do not create embeddings or teach a model to predict language; those weights require separate training.

This tokenizer has no text-normalization stage, so the decoder can reconstruct the encoded example exactly. Other pipelines may lowercase or normalize characters and intentionally lose the original spelling. Chat templates and model-specific special tokens add another layer to the interface.