Learning rate: control how far training moves

The learning rate sets the scale of parameter updates. A useful gradient can still produce a bad step if the update is too large. An extremely small rate can make progress unnecessarily slow.

For an adaptive optimizer, it is a global multiplier around additional per-parameter scaling. It does not mean every weight changes by the same amount.

Start gently, then reduce the step

Warmup raises the rate during an initial interval. A later decay reduces it as training progresses. Warmup followed by cosine decay is one common schedule, not a universal requirement.

This function defines the rate for 0 <= step <= total, with 0 < warmup < total. Steps here mean optimizer updates, rather than individual examples or gradient-accumulation microbatches.

from math import cos, pi

def learning_rate(step, total, warmup, peak):
    if step < warmup:
        return peak * step / warmup
    progress = (step - warmup) / (total - warmup)
    return peak * (1 + cos(pi * progress)) / 2

The schedule reaches peak at the warmup boundary and zero at the final step. It changes the update scale, not the dataset order or the model's output temperature.

Changing batch size changes the clock

If gradients from several microbatches accumulate into one update, the schedule should advance at the intended optimizer-update cadence. Advancing it on every microbatch silently runs through the schedule faster.

Batch size, optimizer choice, initialization and the task affect useful rates. A spike in loss may indicate unstable steps, but changing the learning rate cannot repair every data or implementation problem.

A checkpoint used to resume training should preserve the relevant schedule position and optimizer state. Restoring only the weights can start a different training trajectory even when the visible model is the same.