Alignment: turn intended behaviour into a learning signal

Alignment concerns making a model or system behave according to intended goals, instructions and constraints. The technical difficulty is turning those intentions into training signals and evaluation procedures that generalize.

Predicting likely text is not the same objective as answering helpfully. Post-training can change which responses a model prefers without replacing all of its underlying capabilities.

Where do the preferences come from?

InstructGPT used demonstrations and human comparisons to shape assistant behaviour. Constitutional AI explored model-generated critiques and preferences guided by written principles. These approaches differ in how the feedback is produced, while both translate desired behaviour into an optimization signal.

The feedback is still a proxy. A reward model can prefer a convincing answer that is wrong, and optimizing that preference more strongly need not improve the real outcome.

Evaluate behaviour on new situations, including conflicts between instructions and evidence. Application permissions and checks provide additional control, but are separate from what the model has learned to prefer.