Small language model: a different operating budget

A small language model (SLM) is a language model designed or described relative to a modest parameter and resource budget. There is no single accepted size boundary: the term can include models with millions or a few billion parameters.

Smaller weights make local execution, short startup times and repeated low-cost calls more practical. They do not automatically make a model accurate on a specialized task.

Count the runtime, not only the download

A quantized download may be expanded when loaded. Our Small Language Model experiment runs in the browser and lets readers try the result; its int8 download is decoded to float32 weights at runtime.

KV cache, activations and the runtime add memory beyond those weights. A small model with a long context can therefore have a much larger working set than its file suggests.

Phi-3 and SmolLM2 illustrate another important point: training quality and quantity matter at small scale too. Choose an SLM by measured task quality, latency and memory on the intended device, not by assuming “small” is one capability class.