Reinforcement learning: learn from consequences

Reinforcement learning (RL) trains an agent or policy through rewards associated with its actions. A policy chooses what to do; interaction with an environment produces observations and rewards. The objective concerns expected accumulated reward, not merely matching a provided action label.

Actions can affect later situations. An action that looks good immediately may prevent a better outcome later, so assigning credit over a sequence is a central problem.

Give an action a stronger or weaker push

A neural policy can produce logits over actions. A policy-gradient loss weights the selected actions' log-probabilities by an advantage: an estimate of how much better their outcomes were than a baseline expectation.

This PyTorch function takes batch × actions logits, chosen action IDs and one advantage value per row. It is the basic policy-gradient objective, not a complete PPO trainer.

from torch.distributions import Categorical

def policy_loss(logits, actions, advantages):
    policy = Categorical(logits=logits)
    return -(policy.log_prob(actions) * advantages.detach()).mean()

Positive advantage encourages a chosen action; negative advantage discourages it. Detaching the advantages keeps this loss from differentiating through the procedure that estimated them. An optimizer then updates the policy parameters through the logits.

Reward specifies the training pressure

A reward can come from a game score, a verified solution, human feedback or a learned reward model. Each choice defines which behaviour receives pressure to improve. A convenient score can leave out things people actually care about.

A language model can be a policy over tokens, which connects RL to reasoning training and RLHF. Running an agent with tools is different: it can adapt to observations during a task without any weight update at all.