Epoch: one pass is not one update

An epoch usually means one pass over the training dataset. A batch is a portion of that pass; an optimizer update is a change to the weights. Gradient accumulation can combine several batches into one update.

With 10,000 examples and batches of 100, a complete epoch contains 100 batches. Accumulating five batches gives twenty optimizer updates, assuming no examples are dropped.

A stream needs its own clock

A repeated or effectively unbounded input stream may have no natural endpoint. Keras can define an epoch as a fixed number of batches through steps_per_epoch.

Then “three epochs” describes three reporting intervals, not necessarily three visits to every example. Shuffling buffers, resampling and mixing datasets can further change which examples are seen.

When comparing training runs, report processed examples or tokens and optimizer updates alongside epochs. Changing the dataset or accumulation factor can radically change the amount of work hidden behind the same epoch count.