Training data: repeated text changes what the model sees

Training data supplies the examples used to fit a model. Its influence depends on more than the number of files or tokens: filtering, duplication, mixture weights and sampling determine which patterns receive repeated updates.

A document copied a hundred times contributes a hundred training appearances unless the pipeline removes or reweights the copies. Token volume and information diversity are therefore different quantities.

The corpus is a design choice

RefinedWeb demonstrated that filtering and deduplicating web text can produce strong language-model training data. The useful lesson is about the pipeline, not that every task should use the same source mixture.

Document where examples came from, how they were selected and transformed, and which domains or languages are underrepresented. Dataset documentation makes later quality investigations possible.

Keep training and evaluation provenance separate. A held-out file is not genuinely new evidence if its near-duplicate already appeared in training. Better data can improve a fixed model, while more repetitions of weak data can simply increase the work.