Hyperparameters: choose how learning happens

A hyperparameter controls a model or its training procedure rather than being fitted by the ordinary parameter update. Learning rate, layer width and weight decay are examples. Some settings change during training according to a chosen schedule.

The distinction is procedural. An outer optimization can search hyperparameters while an inner optimization learns weights.

Search useful ranges

This scikit-learn example tunes ridge regularization on generated regression data. Random search samples a continuous logarithmic range rather than spending equal effort on every decimal interval.

from scipy.stats import loguniform
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import RandomizedSearchCV

features, targets = make_regression(n_samples=300, n_features=12, noise=10, random_state=7)
search = RandomizedSearchCV(
    Ridge(), {"alpha": loguniform(1e-4, 1e3)},
    n_iter=20, cv=5, scoring="neg_mean_squared_error", random_state=7,
)
search.fit(features, targets)
model = search.best_estimator_

search.best_estimator_ is refitted on all supplied training rows after cross-validation selects the setting. Evaluate it on separate test data; the winning cross-validation score was used to make the choice.

Random search is useful when only some settings strongly affect the result. More trials still cost computation, and comparing many candidates makes a lucky validation score easier to find.