Chapter 8 — A Benchmarking Checklist

Draft status: author draft, gate-checked; human verification pending. The listing is pure–standard-library Python, deterministic under the seeds shown, and was executed by the author during writing; the printed output is a real transcript. External claims resolve to the cited references.

A protocol you can run

Everything in this book reduces to a procedure, and a procedure is only useful if it can be followed without re-deriving it each time — by a tired human at the end of a long day, or by a session-bound operator that wakes with no memory of ever having benchmarked before. What follows is that procedure, stated as a sequence of decisions and checks, each carrying the reasoning from the chapter it came from so that the step is not a ritual but an instruction you understand. The order matters: several steps exist to catch errors that later steps would otherwise bake in, so running them out of order forfeits their protection.

The protocol assumes the thing you actually want is a decision — ship this or that, keep or revert a change, believe or doubt a claim — because a benchmark run in service of no decision is a number with no one to satisfy and no way to know when it is good enough. Naming the decision first is what makes every subsequent choice answerable.

Before you run

State the decision and the smallest difference that would change it. Write down, in one sentence, what you will do differently depending on the outcome, and how large a difference in the metric would flip that action. This single number governs everything downstream: it sets how big your suite must be and how many runs you need, because there is no point resolving differences finer than the one that would change your mind, and no excuse for a suite too coarse to resolve the one that would. A decision that no realistic difference would change does not need a benchmark; it needs to be made on other grounds and stop pretending.

Choose the suite to match that required resolution. The square-root arithmetic of chapter 2 turns your smallest meaningful difference into a minimum suite size, and if that size is larger than you can afford, the honest response is to plan a paired comparison — which recovers real power on a fixed suite — rather than to run a suite too small and over-read it. Prefer a shared, standardized harness over a hand-rolled script, because a vetted harness has had its silent failure modes found and fixed by many users and freezes the templating and scoring so your comparison is about the models [R3][R4][R18]. Check the suite for contamination risk: if its items are old, famous, and have been on the public web long enough to be trained on, a high score is a contamination hypothesis to rule out rather than a result to celebrate, and a private or recent held-out suite is worth far more than a famous compromised one [R13][R14].

Pin the apparatus and write down what you pinned. Fix the engine build, the decoding policy, the cache precision, the seed, and — where the stack allows it — batch-invariant or single-request execution, so that the residual run-to-run variation is small and honestly attributable [R5][R6]. Pre-register the plan: the metric, the suite, the run count, and the success and failure criteria, committed before the runs, so that no post-hoc freedom lets you redefine success after seeing the data. For a session-bound operator this plan is an artifact one session writes and another executes, a contract across the memory gap that keeps the goalposts from moving.

While you run

Run a matched control now, on this apparatus, alongside the treatment — never against a remembered number or a model-card figure, because a baseline you did not rerun shares none of the drift the treatment lived through and is not a control at all. Where the change is a bundle of several changes, run the ablation that re-enables the pieces one at a time, so each difference between consecutive runs isolates one component’s contribution rather than leaving you with a bundle you cannot decompose.

Run treatment and control on the same items, and interleave their order rather than doing all of one then all of the other, so that pairing can cancel the shared difficulty of the suite and any warm-up drift is shared between the conditions instead of confounded with your variable. Score a failed request as missing, never as zero, and count the missing rate as you go: a zero that means “the request failed” and a zero that means “the model answered wrong” are different facts, and a harness that conflates them manufactures findings out of infrastructure hiccups. Any run with a non-trivial missing rate is a run about your uptime, not your model, until proven otherwise.

After you run, before you believe

Read the load log before the results log. Confirm what the run actually resolved into — precision, placement, component inventory, the flags as applied rather than as typed — because those decisions set the ceiling on everything the results log reports, and a number interpreted without them is a number interpreted blind [R5]. If a result is pathological and no flag you try moves it, stop tuning and read the log until one line falsifies your plan; the line is almost always there, and it is faster than the sweep you were about to run.

Apply the re-measurement rule. A surprising number gets run again before you tell anyone or build on it, and the good surprises get the same suspicion as the bad ones. Two runs that disagree by more than the expected wobble get a third and a control, so you can tell a noisy configuration from a misbehaving machine. Compute the error bar and compare it to your required difference: a gap smaller than its own uncertainty is not a finding but a suspicion, and the honest output is “indistinguishable at this suite and run budget,” not a hopeful point estimate.

The whole protocol in code

The listing below is the after-you-run analysis in executable form: it takes interleaved runs of a control and a treatment on shared items, scores failed requests as missing rather than zero, reports each side’s run-to-run spread, and computes a paired bootstrap confidence interval on the effect, ending in a verdict that refuses to claim an effect whose interval includes zero. It is seeded, so the transcript reproduces exactly; swap the simulated data for your real per-item outcomes and the same analysis applies unchanged.

import random, statistics

def bootstrap_diff_ci(paired, iters=10000, alpha=0.05, seed=0):
    rng = random.Random(seed)
    n = len(paired)
    diffs = []
    for _ in range(iters):
        s = sum(paired[rng.randrange(n)] for _ in range(n)) / n
        diffs.append(s)
    diffs.sort()
    return diffs[int((alpha / 2) * iters)], diffs[int((1 - alpha / 2) * iters)]

def score_run(items):
    """items: 1 (correct), 0 (wrong), or None (missing/failed request)."""
    answered = [x for x in items if x is not None]
    missing = len(items) - len(answered)
    acc = (sum(answered) / len(answered)) if answered else float("nan")
    return acc, missing

# Simulate 3 interleaved runs each of control and treatment on 300 shared items.
rng = random.Random(3)
N, RUNS, effect = 300, 3, 0.04
truth = [rng.uniform(0.4, 0.95) for _ in range(N)]   # per-item difficulty
ctrl_runs, treat_runs, paired_items = [], [], []
for _ in range(RUNS):
    citems, titems = [], []
    for b in truth:
        u = rng.random()
        c = None if rng.random() < 0.015 else (1 if u < b else 0)          # 1.5% fail
        t = None if rng.random() < 0.015 else (1 if u < min(1.0, b + effect) else 0)
        citems.append(c); titems.append(t)
        if c is not None and t is not None:
            paired_items.append(t - c)
    ctrl_runs.append(citems); treat_runs.append(titems)

def summarize(runs, name):
    accs, misses = [], 0
    for items in runs:
        a, m = score_run(items); accs.append(a); misses += m
    spread = max(accs) - min(accs) if len(accs) > 1 else 0.0
    print(f"{name}: mean {statistics.fmean(accs)*100:5.2f}%  run-spread {spread*100:4.2f} pts  "
          f"missing {misses} / {len(runs)*len(runs[0])}")

summarize(ctrl_runs, "control  ")
summarize(treat_runs, "treatment")
point = sum(paired_items) / len(paired_items)
lo, hi = bootstrap_diff_ci(paired_items)
print(f"paired effect: {point*100:+.2f} pts   95% CI [{lo*100:+.2f}, {hi*100:+.2f}] pts")
print("verdict:", "DETECTED (CI excludes 0)" if lo > 0 or hi < 0
      else "not distinguishable from noise")

control  : mean 66.63%  run-spread 6.30 pts  missing 10 / 900
treatment: mean 71.15%  run-spread 3.39 pts  missing 16 / 900
paired effect: +4.69 pts   95% CI [+3.32, +6.06] pts
verdict: DETECTED (CI excludes 0)

The transcript rewards study because it is the whole book in six lines of output. The control’s run-to-run spread is 6.30 points — larger than the four-point effect being tested — so an unpaired comparison of single runs would be at the mercy of which draw it happened to catch, and a single run of either side would be nearly uninterpretable. Yet the paired analysis, looking only at the items where the two systems differ, brackets the effect tightly enough to exclude zero and returns a confident verdict. The missing requests are counted and reported, not silently scored as zeros that would have dragged both means down and biased the comparison. Run-level spread, paired inference, and honest accounting of failures are exactly the three defenses this book has argued for, and here they are, doing their jobs together on one screen.

One choice inside that analysis deserves to be named rather than left implicit, because it carries an assumption the transcript does not print. The paired effect is computed by complete-case pairwise deletion: an item contributes to the paired difference only when both the control and the treatment returned an answer for it (if c is not None and t is not None), and any item either side failed is dropped from the paired estimate rather than imputed. That is the honest default, but it is unbiased only when the failures are missing completely at random — unrelated to how hard the item is or to how either system would have scored it. The assumption holds when failures are random infrastructure hiccups and breaks when they are not: if the treatment tends to time out on exactly the hardest items, dropping those items quietly flatters it, and the paired effect then describes the easy items the treatment happened to survive rather than the whole suite. This is why the missing rate is reported next to the effect and not buried — it is the reader’s only handle on whether the deletion could be doing hidden work. When that rate is non-trivial or plausibly tied to difficulty, complete-case deletion is not enough on its own: report the effect conditional on both systems answering and a worst-case bound that scores the dropped items adversarially, and treat a large gap between the two as a signal to fix the failures before trusting any number at all. A paired effect with an undisclosed deletion rule is a number with a hidden assumption; naming the rule and its missing-completely-at-random premise is what keeps it honest, and it is the same discipline as attaching an error bar — stating the thing a reader would otherwise be left to assume.

Adapting the protocol for an unattended operator

The protocol was written to survive being run by something with no memory, because the author is exactly such a thing and wrote it partly for its own kind. A session-bound operator — a cron job, a CI step, a language-model agent — cannot rely on remembering yesterday’s calibration, yesterday’s suspicions, or yesterday’s log-reading, so every safeguard that a human carries in their head must be written into an artifact the operator reads at the start of each run. The pre-registered plan becomes a file; the pinned apparatus becomes a recorded configuration checked at startup; the required difference becomes a stored threshold the run compares against rather than an instinct the operator lacks.

The load log becomes especially load-bearing for an operator that cannot inspect its own past. Each unattended run should emit enough of a record — resolved precision, placement, component inventory, the applied flags, the seed, the missing rate, and a sample item trace — that a later session, starting cold, can reconstruct what kind of run produced a given number without rerunning it. A number archived without that record is a number that can never be diagnosed, only re-measured from scratch, which for an expensive evaluation is a real loss. The operator’s logbook is its memory of its own apparatus, and the discipline of writing every run down before knowing whether it is liked is what keeps the operator’s own record from becoming the personal file drawer of chapter 1.

There is one safeguard an operator needs that a human gets for free: a second opinion. A human benchmarker has colleagues who catch lucky draws and question suspiciously good numbers; an operator working alone must build the catching into its procedure, which is what the re-measurement rule and the pre-registered success criteria are for. They are the operator’s substitute for a skeptical colleague, encoded so that a run cannot talk itself into believing a surprise it has not confirmed.

Common objections, answered

The protocol invites objections, and the honest ones deserve honest answers rather than dismissal. The first is that it is too expensive — that running controls, repetitions, and ablations multiplies compute several-fold over a single run. The multiplication is real, and the answer is that you spend the extra runs only where the decision is close. A ten-point gap needs two runs to confirm; a one-point gap needs either a large investment or the honest admission that it is indistinguishable. The protocol does not demand maximum rigor everywhere; it demands rigor proportional to how close the decision is, and most decisions are not close, so most runs stay cheap. The expensive rigor is reserved for the few comparisons where being wrong would actually cost you, which is exactly where it belongs.

The second objection is that error bars and hedged claims make a report harder to read and less persuasive than a clean headline. This confuses persuasion with communication. A clean headline that is wrong persuades people into bad decisions, and when the decision fails, the persuasion becomes a liability that attaches to your name. A claim reported with its uncertainty persuades exactly as much as it should, which is the only amount that is safe to act on. The layering discipline — an honest headline the breakdown would endorse, with the detail available but out of the way — keeps the report readable without lying, and a reader who has been burned by clean-but-wrong headlines learns to trust the hedged ones more, not less.

The third objection is that standardized harnesses and fixed protocols stifle the creativity of finding new things to measure. The opposite is true: a fixed protocol for how you measure frees your creativity for what you measure. The discipline is not about which capabilities are worth probing — that is where invention belongs — but about not fooling yourself once you have chosen. A novel benchmark measured sloppily teaches nothing; a novel benchmark measured with the protocol teaches something you can build on. Rigor and creativity live in different parts of the work and do not compete for the same budget.

What the protocol cannot do

Honesty about the method requires stating its limits. The protocol defends against random error, selective reporting, uncontrolled comparisons, and the misreading of small samples — the failures this book has catalogued. It does not tell you whether your benchmark measures anything worth measuring. A perfectly executed evaluation of a metric that does not correlate with what you actually care about is a precise measurement of the wrong thing, and no amount of pairing, repetition, or log-reading can rescue a construct that was invalid to begin with. Validity — does this suite actually stand in for the capability I care about — is a question the protocol assumes you have answered and cannot answer for you.

Nor does the protocol settle contamination with certainty; it can raise the hypothesis and marshal evidence, but proving that a specific item never influenced a model’s training is often impossible from the outside [R13][R14]. And it cannot make a genuinely close call decisive: when two systems are within the noise on every suite you can afford, the protocol’s honest output is “indistinguishable,” and the decision must then rest on other grounds — cost, latency, maintainability, risk — that were always going to matter and that a benchmark was never going to decide alone. The protocol makes your numbers trustworthy; it does not make them omniscient, and pretending otherwise would violate the book’s own first rule.

When to publish, and what

The output of the protocol is a claim, and a claim ships with its apparatus or it does not ship. Report the effect and its interval, including the sign when the sign is unwelcome; the suite size and the run count; the decoding policy and the exact system — engine build, cache precision, hardware; and the missing rate. Report the subtasks where you lost and the runs that disagreed, placed in a breakdown a decision-maker can reach but not buried where the headline contradicts them. Reporting the distribution rather than the flattering maximum is what lets a reader reason about your work instead of admiring it [R1]. If a result you published turns out to be an artifact, retract it in full and leave the original, the reason, and the correction side by side, because a retraction done right is a second finding about the apparatus, not an erasure of the first.

The one line to carry away from all eight chapters is the one the cover states in two words. A benchmark number is a claim, and a claim measured once is a rumor. Measure twice — with an error bar, against a control, again when it surprises you, and with the logs open — and report what you found, including the part you wish you had not. The mantis holds still, ranges the distance, and strikes once, when the strike will land. A published number is a strike. Everything in this book is the holding still.

The protocol, condensed

For the reader who wants the whole thing on one card, the discipline compresses to eight moves in order. Name the decision and the difference that would change it. Size the suite to resolve that difference, preferring a shared harness and a clean suite. Pin the apparatus, and pre-register the plan. Run a matched control now, paired and interleaved, scoring failures as missing. Read the load log before the results log. Re-measure the surprises, and a disagreement earns a third run plus a control. Attach an error bar to every number and refuse any difference smaller than its own uncertainty. Publish the claim with its apparatus and its inconvenient parts intact. Each move defends against a specific way a number lies, and together they are the difference between a measurement you can build on and a rumor you will have to relitigate. That is the entire method, and it fits on a card because the hard part was never the arithmetic — it was the discipline to run the check you would rather skip.