Self-supervised learning: the data supplies the target

Self-supervised learning constructs a training signal from the data itself. Hide part of a sequence and predict it, predict the next token, or make two transformed views of an image produce compatible representations.

No separate human label is needed for each example, but the objective is still deliberately designed. It determines what information the representation must preserve.

Hide a token, keep its answer

This PyTorch function accepts a masked-language model, token IDs and a Boolean selection mask containing at least one selected position. Only selected positions contribute to the loss.

import torch
from torch.nn import functional as F

def masked_loss(model, token_ids, selected, mask_id):
    corrupted = token_ids.clone()
    corrupted[selected] = mask_id
    logits = model(input_ids=corrupted).logits
    return F.cross_entropy(logits[selected], token_ids[selected])

The original tokens supply the targets; the model sees replacements at those positions. This isolates the masking mechanism rather than implementing the full BERT training recipe.

Contrastive methods such as SimCLR use a different signal: two views of one image should agree relative to other examples. In both cases the supervision comes from a relationship constructed within the data, and learned representations can later support other tasks.