Model evaluation: the score includes the experiment

Evaluation measures how a model behaves on chosen tasks. A benchmark supplies examples and a scoring procedure; the reported result also depends on prompts, decoding settings, available tools and the amount of computation allowed.

Two systems can use the same model weights and receive different scores because one gets more attempts or a better checking procedure. A leaderboard number therefore describes an experimental setup as well as a model.

One attempt or several?

For code generation, pass@k asks whether at least one of k sampled candidates passes the tests. Given n generated candidates and correct passing ones for a task, this function computes the standard estimator for 1 <= k <= n.

from math import comb

def pass_at_k(n, correct, k):
    return 1 - comb(n - correct, k) / comb(n, k)

If one of ten candidates passes, pass@1 is 0.1 but pass@5 is 0.5 under this estimator. The model has not changed. The opportunity to sample more candidates has.

A usable system also needs a way to select or verify a successful candidate; a benchmark that tests every candidate can use information the deployed user may not have.

The checker is part of the claim

Weak tests can accept incorrect programs. EvalPlus showed that adding tests can expose failures and even change model rankings. Similarly, a text judge's preference is a different measurement from independently verified factual correctness.

Keep evaluation data separate from training and repeated tuning decisions. Record the task set, prompt, sampling settings, budget and scoring rules so comparisons can be interpreted.

Perplexity measures prediction on observed text. Instruction following, useful reasoning and working code require evaluations that actually test those behaviours.