Tokens and tokenization

A token is one unit in a language model’s input or output. A tokenizer converts text into a sequence of numerical token IDs. Each ID selects a row in the model’s learned embedding table. That row is the embedding vector the network works with.

Why a token is not a word

Many tokenizers keep common words or fragments together and split less familiar text into smaller pieces. Spaces, punctuation and spelling can affect the result. A token can represent a whole word, part of one, or even part of a character’s byte representation.

Byte-pair encoding (BPE) builds a vocabulary by repeatedly merging frequent neighbouring units. The resulting units are useful for representing text compactly; they do not need to match syllables or meanings. The model’s tokenizer and vocabulary determine the split, so the same sentence can have different token counts in different models.

From text to the next token

With torch and transformers installed, this example loads GPT-2 and its matching tokenizer. The first run downloads their files:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tokenizer("The capital of France is", return_tensors="pt")

with torch.inference_mode():
    scores = model(**inputs).logits[0, -1]

next_id = scores.argmax().item()
continuation = tokenizer.decode([*inputs.input_ids[0].tolist(), next_id])

logits[0, -1] selects the scores at the last input position for the first example in the batch: one score per possible next token. argmax() chooses the highest-scoring token. Decoding the extended sequence produces the prompt plus that token. This is one step of text generation.

Why the count matters

Models predict a distribution over their vocabulary and select an output token at each generation step. Longer token sequences use more context positions and require more generation steps for the same text. Token counts therefore matter for context limits, latency and API billing. Count with the target model’s tokenizer instead of treating a word or character count as interchangeable.

Related: context window, embedding, KV cache.