Validation data: the feedback you are still learning from

Validation data is held out from parameter fitting but used to choose settings, checkpoints or models. That makes it part of the development process. A separate test set estimates the performance of the resulting choice.

Repeatedly selecting the best validation score can adapt the whole project to that set, even if no gradient ever touches its examples.

Split the right unit

If several rows come from the same document, a random row split can place near-duplicates on both sides. This scikit-learn example keeps document groups together. The arrays illustrate groups, not a measured corpus.

import numpy as np
from sklearn.model_selection import GroupShuffleSplit

features = np.arange(24).reshape(12, 2)
document_ids = np.repeat(["a", "b", "c", "d"], 3)
split = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=7)
train_ids, validation_ids = next(split.split(features, groups=document_ids))
train_documents = set(document_ids[train_ids])
validation_documents = set(document_ids[validation_ids])

Fit preprocessing, such as scaling or a learned vocabulary, using training data only. A pipeline helps apply the resulting transformation consistently to validation examples.

For forecasting, preserve time order rather than randomly mixing future observations into training. The useful split imitates how new data will arrive, not merely a convenient percentage.