Optimizers: a gradient is not yet an update

An optimizer decides how to change parameters using their gradients. Plain stochastic gradient descent scales the current gradient by a learning rate. Adaptive optimizers also use state accumulated from previous steps.

Adam tracks an exponentially weighted mean of gradients and a mean of squared gradients. The first carries direction through noisy updates; the second adjusts the scale of each parameter's update. Bias correction compensates for starting those moving averages at zero.

AdamW keeps weight decay separate

This NumPy step updates a weight array and its two state arrays in place. Initialize moment and square to zeros, start step at 1 and increment it after each update. The defaults use Adam's familiar decay coefficients.

import numpy as np

def adamw(weight, gradient, moment, square, step, rate, decay=0.01):
    beta1, beta2 = 0.9, 0.999
    moment *= beta1
    moment += (1 - beta1) * gradient
    square *= beta2
    square += (1 - beta2) * gradient * gradient
    corrected_moment = moment / (1 - beta1 ** step)
    corrected_square = square / (1 - beta2 ** step)
    weight *= 1 - rate * decay
    weight -= rate * corrected_moment / (np.sqrt(corrected_square) + 1e-8)

The weight-decay line shrinks the parameter directly. It does not add a penalty into the gradient moments. This decoupling distinguishes AdamW from simply adding an L2 penalty to Adam's loss.

State has a memory cost

A parameter can need its weight, gradient and optimizer state during training. The two Adam moment arrays are one reason training memory exceeds the storage size of the final model. Mixed precision and sharding change where those arrays live and how they are represented.

The optimizer does not choose what counts as a good answer. The loss supplies that objective, backpropagation supplies derivatives, and the learning-rate schedule controls the scale of successive steps.