Regularization: prefer a different solution

Regularization changes learning to discourage solutions that fit the training data too specifically. It can add a penalty, inject noise, restrict a model or stop training early.

A larger training error can be worthwhile if the resulting rule generalizes better. Regularization is therefore a choice about which solutions to prefer, not simply another way to minimize the original training loss.

Watch a weak direction shrink

This NumPy example solves ridge regression for several penalty strengths. The objective is squared residual sum plus alpha * sum(weights**2).

import numpy as np

features = np.array([[1., 1.01], [2., 1.99], [3., 3.01]])
targets = np.array([1., 2.1, 2.9])
solutions = {
    alpha: np.linalg.solve(
        features.T @ features + alpha * np.eye(2),
        features.T @ targets,
    )
    for alpha in (0., 0.1, 10.)
}

The two features are nearly redundant. Without a penalty, the model uses large opposing coefficients to explain a tiny difference. Ridge suppresses that fragile direction.

The example has no intercept; in ordinary regression, an intercept is usually treated separately from the weight penalty. Choose regularization strength on held-out data. More shrinkage can remove useful signal as well as unstable detail.