Benchmarks and leaderboards: compare evidence before choosing a model

A benchmark is a defined set of tasks and an evaluation protocol used to compare systems. A leaderboard orders submitted or measured results under a stated scoring method. A result describes a particular model or agent configuration on that test; it is not an unconditional measure of intelligence or production readiness.

The interview skill is to explain what was tested, how it was scored, what may be misleading, and which product evaluation follows. Public results help narrow candidates. Your workload, constraints and verified outcomes determine deployment.

Source review: September 24, 2026. This chapter names current versions where verified and avoids maintaining a table of rapidly changing model winners.

Read six parts of every claim

Harness means the machinery that runs tasks and grades outputs. Scaffold means the agent's surrounding orchestration, prompts, tools and control logic. They interact: sandbox resources and timeout behavior can affect an otherwise unchanged agent.

For model isolation, compare models under appropriately matched conditions. For a product decision, comparing complete stacks with different scaffolds can be valid if the workload, acceptance criteria and constraints are comparable. State which question the experiment answers; “never compare different harnesses” is too absolute.

Capability map: knowledge, reasoning and mathematics

Sources and definitions: MMLU-Pro project, GPQA paper, HLE project, ARC Prize, CritPt dataset, FrontierMath tracks. The LM Evaluation Harness task catalog helps identify exact configurations for many established text benchmarks.

HLE-Diamond was released on September 22, 2026 with 1,000 questions split into knowledge and reasoning partitions. Its release reports tool-free and tool-enabled conditions separately. A score on this subset cannot be substituted for a score on the earlier 2,500-question HLE set. Release and protocol.

AIME has a small annual problem set, so one item can move a score noticeably. Repeated sampling measures model variability but does not create more distinct mathematics problems. A high answer score does not by itself establish proof-writing ability.

Coding: identify the unit of engineering work

The official SWE-bench site distinguishes its subsets and a matched Bash Only view. EvalPlus strengthens function-level tests. LiveCodeBench organizes problems by release time. ProMax research concerns large multilingual refactoring. These are complementary workloads, not successive names for one identical exam.

Current Terminal-Bench: the maintainer documents 4.0. Its changes include environment resources, task repairs and removals, requiring fresh runs for the major version. It reports cost and tokens alongside resolution rate. Do not present a 3.0 result as a 4.0 result or say that public benchmarks never include cost. 4.0 release.

A benchmark audit needs the right denominator

OpenAI's February 2026 SWE-bench Verified audit examined 138 selected difficult cases, not a representative random sample of every task. It found material issues in 59.4% of that audited subset. That does not mean “60% of all 500 tasks are broken.” The same report describes contamination evidence on particular tasks. Audit and selection method.

The subsequent July 2026 SWE-bench Pro audit also reports substantial task-quality issues. Therefore “switch to Pro and validity is solved” is not a defensible conclusion. Inspect the current task release, contested cases and maintainer fixes; reproduce important comparisons on your own audited tickets.

Agents, tools, computers and knowledge work

BFCL V4 provides versioned tool evaluation. The τ-bench repository now describes τ³-bench, including voice and knowledge tasks; its July 2026 grading change affects banking-knowledge comparisons. The repository name still contains tau2-bench, so the URL alone does not identify the evaluated version. Its earlier τ² paper is from 2025, not a Q1 2026 debut.

The OSWorld project points to OSWorld 2.0, released June 26, 2026. Historical OSWorld-Verified scores remain tied to that older protocol. Check current primary documentation rather than using a historical human/model comparison as today's ceiling.

METR's time-horizon explanation is especially important: a 50%-success horizon is based on estimated human task duration for tasks in its suite. It is not the time the agent runs unattended, not a guarantee it can do every task of that duration, and not evidence of 50% reliability in every occupation.

pass@k and pass^k answer different questions

• pass@k: at least one of k attempts succeeds under the stated protocol.
• pass^k: all k repeated attempts succeed under the stated protocol.

For independent identical success probability p = 0.8 and k = 4, these are 1 − (1 − p)^4 = 99.84% and p^4 = 40.96%. Real tasks have different probabilities and attempts may correlate. Compute benchmark estimators from repeated per-task outcomes; do not exponentiate an aggregate observed success rate and call it the measured result. A production user may have only one attempt and no oracle to choose the correct candidate. See agent evaluation.

Long context, multimodal work and factuality

RULER is a useful reminder to measure performance by length and task. LongBench v2 describes lengths in words, while LongBench Pro reports its own token-based range and bilingual tasks. Words and tokens are not interchangeable. A maximum accepted context length is a capacity limit, not a guarantee of uniform answer quality.

Primary descriptions: MMMU-Pro, Video-MME-v2, SimpleQA Verified, FACTS Grounding. Vary length, evidence position, distractors and required reasoning when translating these ideas into a product test.

Preference scores and composite indices

The Text Arena's human-preference board estimates relative preference using a Bradley–Terry model, with uncertainty; calling every current score “Elo” obscures the methodology. Arena-Rank publishes the ranking approach. Style control adjusts for selected formatting/length features; it does not isolate perfect objective truth.

Arena now also documents an optional factuality signal that combines preference with automated claim assessment. State which board/toggle you mean. Ordinary preference votes alone are not factual correctness; a factuality-augmented score also inherits the verifier's limits. New agent-specific boards can use different methods, so inspect their methodology separately.

MT-Bench uses model grading of a compact multi-turn test. Arena-Hard is an automated preference-oriented evaluation, not identical to live human Arena votes. Do not transfer a proxy's measured correlation or bias estimate to every later model and product.

The Leaderboard Illusion raises concerns about selective variant disclosure and other ranking effects. Arena's response disputes several claims and their magnitude. The practical response is to inspect testing/disclosure policy, fresh votes, uncertainty and the actual workload, rather than declaring either all leaderboards fraudulent or every rank authoritative.

Sources: Artificial Analysis methodology, Epoch methodology explanation, HAL paper. Composite weights encode priorities. They can improve broad comparison but cannot create evidence for a task absent from their components.

Four threats to a useful comparison

A public benchmark is not automatically known to be in a particular model's training data. A claimed cutoff or absence of detected overlap does not establish the opposite either. Canary strings, overlap searches and membership tests have limits; a negative finding from a small audit may be underpowered.

Private cases reduce direct exposure but make independent reproduction harder. Live cases need stable snapshots and maintained graders. More inference effort can help, waste resources or hurt a workflow; measure it rather than assuming monotonic improvement.

Worked comparison: shortlist, test, then decide

Two illustrative systems each attempt 500 private repository tasks once under the same test contract:

The paired outcomes are 400 both-pass, ten A-only, 25 B-only and 65 both-fail. B has 15 net additional successes, not 425 wholly different successes. Its additional $150 costs $10 per net additional accepted patch in this run. That may be worthwhile or not, depending on value, human review and severity. These point estimates do not establish generalization or statistical certainty.

For one task in a 500-case test, the score changes by 0.2 percentage points. That is score granularity, not a confidence interval. Analyze paired uncertainty, language/repository slices and unsafe changes. Two separate confidence intervals overlapping does not by itself settle a paired significance question. Repeated runs add evidence about stochastic stability; they do not erase dataset bias.

flowchart LR
    P[Product tasks and constraints] --> B[Relevant public evidence]
    B --> S[Small candidate shortlist]
    S --> E[Matched private evaluation]
    E --> A[Failures, uncertainty, latency and total cost]
    A --> G{Meets release criteria}
    G -->|No| R[Repair or choose another candidate]
    G -->|Yes| C[Bounded canary]
    C --> M[Verify real task outcomes]

Interview questions and answer checks

Final notes

Remember task → system → budget → grader → uncertainty → product fit. A good interview answer does not memorize a winner. It uses public benchmarks for relevant evidence, audits the comparison, and justifies a system with measured outcomes under the actual constraints.