# Small Language Model — complete source

To start training, save just training/train.py as train.py. It contains the
configuration, Transformer, data pipeline and training loop. No project modules
or configuration files need to be copied alongside it.

Install dependencies in your Python environment, then run:

    python -m pip install tensorflow==2.21.0 keras==3.15.1 datasets==5.1.0
    python train.py

For NVIDIA GPU dependencies on Linux, use tensorflow[and-cuda]==2.21.0 in
place of the plain TensorFlow package (quote the package name in your shell). GPU use is automatic when TensorFlow
sees a compatible GPU; otherwise it uses CPU. Training requires substantial
RAM and compute. The corpus streams into a RAM cache, not dataset files;
small Hub metadata can still be cached.

The default is the 19.4M model, saved under runs/small. To train the larger
model instead, or change the update target:

    python train.py --large
    python train.py --steps 1000 --run runs/short-experiment

The larger model uses runs/large by default. Choose a new run directory when
changing model dimensions.

A run directory contains configuration, bounded checkpoints, loss.csv and
optimizer state. Repeating a command resumes that run to its configured target and skips training
if the target is already reached. Targets need not be multiples of500.
The recipe sets a 20,000-update target. The published small model was trained
across resumed sessions to step 20,422; the published larger model is the
completed checkpoint at step 20,000. A fresh run is not a byte-identical reproduction.

Only for browser export, also save training/export.py beside train.py and
create the browser directory using the browser files shown in the viewer.
The commands below assume train.py and export.py are in a training directory,
with browser as a sibling directory. Run from the training directory:

    python export.py runs/small ../browser/model --quantize
    python export.py runs/large ../browser/model/large --quantize

The exporter writes the model manifest, uint8 weights, deterministic gzip
sibling and reference logits. The browser expands weights back to float32.
The page offers both models; export both before choosing the larger option.
Download sizes printed on the page describe our published models, not every
fresh training run. To use only the small model, leave the default selected.

From the browser directory:

    python fetch_runtime.py
    python -m http.server 8000 --bind 127.0.0.1

Open http://localhost:8000. The Python HTTP server serves static files only;
inference runs in the browser. Use HTTP locally, or HTTPS when hosting remotely.
For HTTP gzip downloads, configure your static server to serve each .bin.gz
sibling when Accept-Encoding includes gzip. The simple local server sends the
uncompressed uint8 binary; the model and computation are the same.

The browser page is the demo source before the site's optional Analytics
fragment is inserted. These files do not include an Analytics tag. No API key,
server inference, VM management or deployment scripts are needed.

TensorFlow.js is fetched from the pinned URLs in vendor.json with SHA-256
verification. Its Apache 2.0 license is downloaded as vendor/LICENSE. The
SimpleStories dataset revision is pinned in train.py; see the article for
its dataset and paper links.