Gradient: which change would reduce the loss?

A gradient collects a scalar function's partial derivatives with respect to its inputs. In training, those inputs are usually the model parameters and the scalar is the loss.

Each component measures local sensitivity: how much the loss would change for a small change to that parameter while holding the others fixed. The gradient points toward steepest increase under the usual Euclidean geometry; gradient descent steps in the opposite direction.

Differentiate the model's error

This TensorFlow example differentiates mean squared error through a two-feature linear model. Automatic differentiation follows the recorded operations.

import tensorflow as tf

features = tf.constant([[1., 2.], [3., 1.]])
targets = tf.constant([4., 5.])
weights = tf.Variable([0.2, -0.1])
with tf.GradientTape() as tape:
    predictions = tf.linalg.matvec(features, weights)
    loss = tf.reduce_mean(tf.square(predictions - targets))
gradient = tape.gradient(loss, weights)

gradient has the same shape as weights. Computing it does not update anything: an optimizer must apply a change afterward.

This is a local signal, not a guarantee about a large step or a globally best model. Backpropagation is the efficient chain-rule procedure used to compute these gradients through a network.