Benchmark: the test also defines the contest

A benchmark is a specified collection of tasks, data and scoring rules used to compare systems. A score belongs to that procedure: prompts, tool access, attempts and computation budgets can all change what is being measured.

BIG-bench combines many different language tasks. MACHIAVELLI evaluates agents in interactive stories and measures harmful behaviour alongside reward. They ask different questions, so neither compresses naturally into a universal intelligence score.

Keep the tradeoff visible

Imagine one agent finishes 80% of tasks in one attempt; another finishes 90% after ten attempts. The second result may be more useful, but it does not establish better one-attempt performance or lower cost.

Record the model version, evaluation setup and budget with the result. Examine task-level failures as well as an aggregate: strong performance on common tasks can conceal a missing capability.

A benchmark is most informative when its tasks resemble the decisions you care about. Once the test becomes a training target, a fresh held-out evaluation becomes especially valuable.