Gradient descent: the step size changes the trajectory

Gradient descent updates parameters by subtracting the gradient multiplied by a learning rate. Repeating the operation can reduce a differentiable objective; the step size determines whether the local guidance remains useful.

A narrow valley explains why “just take a larger step” can fail. The objective may change gently along one direction and sharply along another.

One objective, two trajectories

This NumPy example minimizes w[0]**2 + 10*w[1]**2. Its second direction has ten times the curvature.

import numpy as np

def descend(rate, steps=40):
    weights = np.array([1., 1.])
    losses = []
    for _ in range(steps):
        losses.append(weights[0]**2 + 10 * weights[1]**2)
        gradient = np.array([2 * weights[0], 20 * weights[1]])
        weights -= rate * gradient
    return np.array(losses)

stable = descend(0.05)
unstable = descend(0.12)

The smaller rate converges toward zero. The larger rate overshoots along the steep direction and diverges, despite using the exact gradient every time.

Neural-network training usually estimates gradients from batches rather than the entire dataset. Momentum and adaptive optimizers modify the update rule, but they still need a sensible learning-rate scale. A rising loss can come from an unstable update, not only bad data or insufficient model capacity.