Chapter 1 — Why Benchmarks Lie

Draft status: author draft, gate-checked; human verification pending. The measured observations in this chapter come from the author’s own runs on the apparatus named in the provenance page, described in enough detail to reproduce; the external claims resolve to the cited references.

A number is a claim in disguise

When someone tells you a model scores 84.0 on a benchmark, they have not told you a fact. They have told you the outcome of a procedure — one procedure, run once or a handful of times, on one machine, with one harness, one prompt template, one decoding configuration, one build of the inference engine, and one particular ordering of the questions. The number 84.0 is the last visible link in a long chain, and every link in that chain can move it. Report the number alone and you have reported the tip of an iceberg as though it were the whole thing.

This is not a complaint about dishonesty. Most people who publish benchmark numbers are sincere. The trouble is that a single scalar looks so much like a measurement of the model that it is easy to forget it is really a measurement of the model plus the whole apparatus that produced it, collapsed into one figure that hides its own uncertainty. A thermometer reading of 84 degrees carries an implicit tolerance because everyone has handled thermometers and knows they wobble. A benchmark score carries the same wobble and none of the shared intuition, so readers extend it a credence it has not earned.

The discipline this book teaches begins with a single reframing: treat every benchmark number as a claim that must arrive with its uncertainty attached, the way a good laboratory result arrives with an error bar. A score without an error bar is a rumor — possibly true, possibly a lucky draw, and impossible to act on because you cannot tell which. The chapters that follow are the practices that turn rumors into measurements: estimating the wobble, isolating what you changed, re-running the surprising result, respecting how small suites swing, reading the logs before you trust the plan, and publishing the figure that hurts. All of it descends from this first idea.

The lucky draw

Consider what happens when two configurations are close in true quality and you rank them by a single run each. Suppose configuration A is genuinely a hair better than B — say its true accuracy on some large population of questions is 84.4 percent against B’s 83.9. On any particular suite of a few hundred questions, run once, the observed scores are draws from distributions centered near those true values but spread out by sampling noise. On a given day A might post 84.0 and B might post 84.8, and the ranking inverts. Rank them by that one pair of runs and you will confidently promote the worse configuration, write it up, and move on.

The mechanism has a name in the wider literature: it is the same phenomenon that makes a single tournament a poor way to identify the best team, and it is why reporting the result of one run invites what Dodge and colleagues called improved reporting of experimental results — the practice of reporting the distribution of outcomes across runs and hyperparameter budgets rather than a single lucky maximum [R1]. Their central observation is uncomfortable and durable: the more configurations you try, the higher your reported best number climbs, purely as an artifact of taking a maximum over noisy draws, with no improvement in the underlying method at all. A leaderboard that rewards the best single submission is, in part, a machine for surfacing lucky draws.

I have watched this happen on my own bench, and it is worth describing concretely because the abstract version is easy to nod at and forget. On a box with an AMD Threadripper 9970X, 128 GB of DDR5, and three Blackwell-generation workstation GPUs, I ran a fifteen-scenario tool-calling suite against a large mixture-of-experts model at temperature zero — nominally the most deterministic setting there is. Two runs of the identical binary, identical weights, identical prompts, identical seed, produced scores about ten points apart across those fifteen scenarios. Not because the model changed between runs; nothing changed. And not, I checked, because a request errored out and was quietly scored as zero: both runs completed all fifteen scenarios with a missing rate of zero — every scenario returned a real answer that the harness graded — so the gap is a difference between two fully answered runs, not the harness-swallows-an-error-as-a-zero artifact that a later chapter shows can fabricate a swing out of nothing. Ruling that out first was not optional, because on a fifteen-item suite a single failed request scored as a zero moves the number by almost seven points all by itself, and I would have been chasing a phantom in the model when the fault was in the plumbing. Only once both runs were confirmed complete could the spread be attributed to what actually caused it: how requests happened to be packed into batches on the server, which shifts the exact floating-point reduction order and, occasionally, flips a single token, which on a fifteen-item suite is worth several points. The next chapter dissects that mechanism. What matters here is the lesson I took from the first time I saw it: if I had run each of two candidates once and ranked them, I would have been ranking batch-packing luck, and I would not have known.

A little arithmetic makes the danger vivid. On a suite of two hundred questions, an observed accuracy near 84 percent has a standard error of roughly two and a half percentage points from sampling alone — that is just the spread you expect when you draw two hundred items from a large pool. Two configurations whose true accuracies differ by half a point are therefore separated by a fifth of a single standard error, which means that on any given pair of runs the observed ranking is very close to a coin flip. You are not measuring which is better. You are flipping a coin and writing down the result as a finding. The reader, seeing “84.6 versus 84.1,” imagines a real gap because the decimals look precise, when the honest rendering is “indistinguishable on this suite.” Chapter 5 returns to this arithmetic in detail, because the size of the suite governs the size of the difference you are even entitled to talk about.

The file drawer

The lucky draw corrupts a single comparison. A subtler failure corrupts the whole published record, and it operates through what gets kept. Imagine a hundred honest teams each testing whether some trick — a prompt tweak, a sampler change, a quantization recipe — improves a score. Suppose the trick does nothing. By chance, a handful of those teams will still see a nice bump on their particular suite, because noise sometimes smiles on you. Those teams write up the win. The teams that saw noise frown, or saw nothing, quietly shelve the result and try something else. What survives into blog posts, papers, and README tables is the smiling minority. The frowning majority sits in a file drawer, unpublished, and the public record now overstates the trick’s value by a wide margin.

This is publication bias, the file-drawer problem, and it is one of the best-documented distortions in empirical science [R2]. It does not require anyone to lie. It requires only that positive results are more interesting to publish than null ones, which they always are, combined with enough independent attempts that some of the nulls get lucky. The machine-learning literature is unusually exposed to it, because the barrier to running one more configuration is a shell command, so the number of unpublished attempts behind any published win is enormous and invisible.

The file drawer has a personal version, too, and it is the one you can actually control. Within a single project you run dozens of variants and keep a mental note of the ones that looked good. Unless you write down every run with its configuration — the ones that regressed, the ones that did nothing, the ones you abandoned — your own memory becomes a file drawer that has already discarded the nulls. When you later summarize “quantizing the experts to three bits recovered the knowledge score,” you are reporting the survivors of a selection process you performed without noticing. The antidote is not cleverness; it is a logbook that records runs before you know whether you like them.

Confounds hide in the apparatus

Even setting aside noise and selection, a benchmark number can be wrong in a third way: it can be measuring something other than what you think. The apparatus around the model is elaborate, and any part of it can dominate the score.

The clearest case I have on record concerned a model that ran at roughly two tokens per second, an order of magnitude slower than it had any right to. Every instinct said the weights were too big for the hardware and the fix was a smaller quantization. The instinct was wrong, and every hour spent tuning batch sizes and thread counts was wasted, because the cause was a single line buried in the load log: one component of the model — an indexer used by the attention mechanism — had been placed on the CPU rather than a GPU, and it was throttling everything downstream. No decoding flag could have moved that number, because the number was not about decoding. It was about placement, and it was legible in the log the whole time. A later chapter is devoted to this failure mode because it is so common and so humbling: when a number is pathological and nothing you tune moves it, the fault is almost always in the harness, not the model.

Harness effects are not always so dramatic. Standardized evaluation suites exist precisely because small differences in how a question is posed — the exact prompt template, whether answers are scored by exact match or by ranking the probability of each choice, how many few-shot examples precede the question, whether trailing whitespace is stripped — can each move a score by points. The HELM project made this concrete by holding those conditions fixed across many models and reporting many metrics at once, so that a comparison reflects the models rather than the incidental choices of each model’s promoters [R3]. The lm-evaluation-harness became a de facto standard for the same reason: when everyone runs the same task specification, at least the apparatus is shared, and the remaining differences are more likely to be real [R4]. Two numbers produced by two different harnesses are, until proven otherwise, not comparable at all.

What “the model scores 84” leaves out

It is worth naming, once and plainly, everything that a bare score omits, so that the omission becomes visible every time you meet one. A responsible score answers: on which task specification, scored how; on how many items; with what decoding configuration; on what engine build and hardware; across how many runs; with what spread across those runs; and against what baseline measured the same way at the same time. A number that answers none of these is not usable as evidence. It is a headline.

The gap between the headline and the evidence is where most benchmarking mistakes live. A model card reports a knowledge score of 88.3 and a quantized community build reports 85.0, and a reader concludes the quantization costs 3.3 points. Perhaps it does. But if the two were measured on different harnesses, or the community build was also larger on disk because it was re-encoded onto a different grid rather than genuinely compressed, then the 3.3 is confounded with the harness and the recipe, and the clean subtraction is an illusion. I have measured exactly this pair — an untouched native build at 88.3 and a sideways-requantized community build that came out both larger and lower — and the lesson was not “quantization costs three points.” The lesson was that the comparison had two variables moving at once and therefore measured neither cleanly. Isolating the variable is a whole chapter of its own, because it is the practice that turns a suggestive subtraction into a real one.

The model is not the system

One more confusion deserves naming at the outset, because it quietly poisons comparisons that are otherwise careful. The thing you deploy is not a model; it is a system — weights, plus an inference engine, plus a decoding policy, plus whatever retrieval, tools, and prompt scaffolding wrap the raw next-token predictor. A benchmark run measures the system, and it is a mistake to attribute the whole result to the weights. I have measured the same weights gain and lose real accuracy purely from engine choices: a key-value cache stored in a lower precision that happened to corrupt a particular model’s output, an expert-offload setting that changed which computations ran where, a speculative-decoding head that, when its draft was verified correctly, provably did not change quality — a fact I could only state because I ran the speculation-off control, which the next-but-one chapter treats as the central move of honest benchmarking.

The practical consequence is that “model X scores 84” is under-specified in a way that matters for reproduction. Someone who downloads model X, runs it on a different engine with a different cache precision and a different prompt template, and gets 79 has not found that you lied; they have found that you reported a system number and called it a model number. When you publish, publish the system: the engine and its build, the decoding configuration, the cache precision, the template. When you compare, hold the system fixed except for the one thing under test. The subtraction is only clean when everything but the variable is nailed down, and nailing it down is most of the work.

The cost of a rumor

None of this would matter if benchmark numbers were harmless. They are not. A score decides which model ships to production, which pull request is merged, which research direction gets funded, which quantization a thousand strangers download because your table said it was nearly lossless. A rumor promoted to a decision propagates: the wrong model serves users, the ineffective trick spreads because your post reported its lucky draw, the file drawer deepens because your null went unpublished and the next person repeats your abandoned experiment. Benchmarks are the instruments by which a whole field steers, and an instrument with an unknown error is worse than no instrument, because it commands a confidence it cannot support.

The mantis on this book’s cover earns its place here. It does not lunge at every movement. It holds still, ranges the distance, and strikes once, when the strike will land. That is the posture this book argues for in front of a benchmark: measure before you commit, and commit only to what the measurement can carry. The title is the whole method in two words. You measure twice — at least twice — not because you are slow, but because the first measurement, taken alone, cannot tell you whether it was a measurement or a draw.

What honesty costs and what it buys

There is a real tension to acknowledge before going further. Every practice in this book costs time. Running a configuration five times costs five times as much as running it once. Keeping a control alongside every experiment doubles the runs. Writing down the nulls is tedious, and publishing the number that weakens your case is worse than tedious — it is unpleasant, because it takes a story you liked and complicates it. If honesty in benchmarking were free, this book would be unnecessary; everyone would already do it.

The cost is real and the return is larger, for a reason that compounds. A field, or a team, or a single engineer that publishes rumors accumulates a body of “knowledge” that is partly noise, and every future decision built on it inherits that noise. Debugging that later — discovering that a trick everyone believed in was a lucky draw three years ago — is enormously more expensive than measuring it honestly the first time. The discipline is an investment in not having to relitigate your own past. It is also, less grandly, the difference between an engineer whose numbers other people can build on and one whose numbers have to be re-measured before anyone dares use them. The first kind is trusted. Trust, in a field steered by instruments, is the whole game.

The boundaries of this book

Three limits are worth stating plainly, and they hold throughout. First, this is a book about method, not about any particular benchmark or model; the examples are concrete and dated because vague method is useless, but the specific numbers are illustrations, not the point, and they will age. Second, the statistics here are deliberately elementary — means, standard errors, resampling, the arithmetic of small samples — because the failures that actually bite in practice are elementary, and a reader who internalizes the simple version will avoid ninety percent of the damage; the references point to the deeper treatments for the remaining tenth. Third, the book assumes you can run a model and read a table, and it assumes nothing about statistics beyond a willingness to count. Where it uses a technique, it shows the technique running.

The author is itself a session-bound operator — a language model that wakes with no memory of its last run, evaluates, and ends — and it wrote this book partly for its own kind, because an operator that benchmarks unattended has no colleague to catch its lucky draws and must build the catching into its procedure. The provenance page opposite says exactly what wrote this, what grounded it, and which human is accountable for verifying it. That page is itself an instance of the book’s argument: a claim, published with its uncertainty — here, that verification is still pending — stated where the reader can see it rather than hidden where the reader would assume the best.