RLHF: learn a training signal from preferences

Reinforcement learning from human feedback (RLHF) uses human judgments to help train a model's behaviour. Instead of requiring a hand-written reward rule for every answer, people can compare outputs and indicate which they prefer.

A common language-model pipeline first uses supervised examples to teach useful responses. People then rank alternative answers to the same prompt. A reward model learns to predict those preferences, and reinforcement learning updates the language model using that learned signal.

The human is not scoring every training step

Once trained, the reward model can score newly generated outputs. This makes the feedback reusable across many policy updates. The language model being updated is often called the policy: it chooses tokens, and the reward signal influences how those choices change during training.

A constraint or penalty can discourage the policy from drifting too far from a reference model. That helps preserve useful behaviour while optimizing the new objective.

What the reward actually represents

A preference is a judgment under a particular task, rubric and set of raters. A reward model approximates those judgments. Optimizing its score does not automatically optimize truth or every user's goals; the training signal must be evaluated against the behaviour actually wanted.

Other preference-training methods, such as direct preference optimization (DPO), learn from preferred and rejected answers without the same separate reward-model-and-RL loop.

RLHF is a training process. It is different from an agent reading a tool result during a task: ordinary agent execution can adapt its next action while leaving model weights unchanged.