Encoder: let every visible token inform the representation

An encoder turns an input into representations used by another part of a system. A Transformer text encoder typically allows each token to attend to all unmasked input positions, so later words can change an earlier token's representation.

This differs from a causal decoder, whose position cannot inspect future tokens. BERT uses bidirectional context to learn representations from masked-token prediction.

Read the contextual states

This PyTorch/Transformers example downloads BERT and its tokenizer, then pools token states while excluding padding.

import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
encoder = AutoModel.from_pretrained("bert-base-uncased").eval()
inputs = tokenizer(["A bank by the river.", "A bank approved the loan."],
                   padding=True, return_tensors="pt")
with torch.no_grad():
    states = encoder(**inputs).last_hidden_state
mask = inputs.attention_mask.unsqueeze(-1)
pooled = (states * mask).sum(dim=1) / mask.sum(dim=1)

last_hidden_state contains a vector for each input position, not a generated answer. The attention mask controls which positions participate; pooling combines the visible states into one vector per input.

A retrieval system can use an encoder to represent documents and queries, but the training objective and pooling method determine how useful their distances are. The encoder architecture alone does not define a similarity metric.