Parameters: shared weights can do two jobs

Parameters are adjustable values in a model, such as matrix entries and biases. Training fits them; inference normally reads them without changing them.

Parameter count counts stored learned values, not the number of times those values are used. A shared matrix can participate in several operations while occupying one parameter allocation.

Tie input and output embeddings

This PyTorch model uses one matrix both to look up token vectors and to project hidden vectors back to vocabulary scores.

import torch
from torch import nn
from torch.nn import functional as F

class TiedTokenModel(nn.Module):
    def __init__(self, vocabulary, width):
        super().__init__()
        self.tokens = nn.Embedding(vocabulary, width)

    def forward(self, token_ids):
        hidden = self.tokens(token_ids)
        return F.linear(hidden, self.tokens.weight)

For vocabulary size V and width d, the matrix holds V×d parameters. Reusing its transpose for the output projection avoids a separate d×V matrix. Gradients from both uses accumulate into the same weights.

The example has no contextual layers, so it is not a useful language model by itself. Its point is the real weight-tying mechanism used inside larger models. Parameter count still says little about context memory, temporary activations or the computation needed for one request.