Chapter 4 — Re-measure

Draft status: author draft, gate-checked; human verification pending. The measured observations are the author’s own, on the apparatus named in the provenance page; the external claims resolve to the cited references.

The first result is a hypothesis

A single benchmark run does not produce a fact. It produces a hypothesis about what the fact might be, and the strength of that hypothesis depends entirely on things the run itself does not report: how much this measurement wobbles, whether the apparatus was in its usual state, whether the number is even plausible. Treating the first result as final is the error that all the earlier chapters converge on from different directions — the lucky draw, the missing error bar, the uncontrolled comparison all reduce to trusting one run too much. Re-measurement is the general antidote, and it deserves its own chapter because it has a structure: a small decision procedure that tells you, given what you have seen, whether to believe a number, rerun it, or reach for a control.

The procedure is short enough to state in a sentence and consequential enough to spend a chapter on. One surprising number means run it again. Two runs that disagree means run a third and a control. A number that is not surprising and agrees with its rerun can be believed at the precision its error bar allows. Everything else in this chapter is the reasoning behind those three moves and the traps that lurk in each.

The surprising number

Surprise is information. When a run produces a number far from what you expected — a jump, a collapse, a suspiciously round figure, a result that would be a breakthrough if true — the surprise is telling you that either your expectation was wrong or the measurement was. Both are worth knowing, and you cannot tell which from the single run that surprised you. The first move is always the same: run it again, unchanged, before you do anything else — before you tell anyone, before you build on it, before you start explaining it. The rerun costs one unit of compute and buys you the single most useful piece of evidence available, which is whether the surprise survives contact with a second draw.

Most surprises do not survive, and that is the point. The ten-point swing on the tool suite from chapter 2 was a surprise the first time I saw it; had I run only once, I would have recorded whichever draw I happened to get and reasoned about it as though it meant something. The rerun is what revealed the number as a random variable rather than a measurement. When a surprise evaporates on rerun, you have not wasted a run; you have avoided building a story on noise, which is far more expensive to unwind later than a single extra run is to perform now.

The discipline is hardest to follow when the surprise is good. A number that confirms your hope — the new recipe jumped four points, the optimization doubled throughput with no quality loss — is the one you least want to rerun, because rerunning it risks taking the good news away. That reluctance is precisely the file-drawer instinct from chapter 1 operating in real time: you are tempted to keep the lucky draw and skip the confirmation. The rule has to be symmetric or it is worthless. Rerun the good surprises with exactly the same suspicion you bring to the bad ones, because a lucky good draw that you publish does more damage than a lucky bad draw that you quietly investigate.

When two runs disagree

Two runs that agree, within the wobble you expect, let you proceed. Two runs that disagree by more than that wobble pose a sharper question, because now you have two candidate facts and no way to choose between them. The move here is not to average them and move on, and it is not to pick the one you like. It is to run a third — and, crucially, to run a control alongside it, an unchanged baseline measured in the same session on the same apparatus.

The third run and the control do different jobs, and you need both. The third run tells you about the distribution: with three points instead of two, you begin to see whether the disagreement is two clustered values and one outlier, or genuine wide scatter, or a bimodal pattern that hints at two distinct machine states. The control tells you about the apparatus: if the unchanged baseline is also scattering widely in this session, the problem is the measurement environment, not your variable, and no amount of rerunning the treatment will produce a clean number until the environment is fixed. Disagreement between two runs is ambiguous between “this configuration is noisy” and “the machine is misbehaving right now,” and the control is what disambiguates them.

I lean on this pairing whenever a comparison refuses to settle. If treatment and control both scatter, I stop benchmarking and start diagnosing the machine — checking load, thermal state, what else is resident, whether a background job is stealing the accelerators — because a noisy apparatus makes every number meaningless and there is no point collecting more of them. If the control is tight and only the treatment scatters, the instability is a real property of the treatment, and that is itself a finding worth reporting: a configuration whose score depends on the phase of the moon is worse, for production, than a slightly lower one that is stable.

Regression or noise

The most consequential re-measurement question in day-to-day work is whether a drop is a regression or noise. You change something — update the engine, adjust a setting, merge a branch — and the score falls. Did your change break something, or did you catch an unlucky draw? The stakes are asymmetric and both errors are costly: chase a phantom regression and you burn days debugging noise; wave off a real regression as noise and you ship a defect. The decision procedure is the same as before, sharpened by the fact that you have a natural control: the state before your change.

Rerun both sides, now, on the same apparatus — the new state and the old state — several times each, and compare their distributions rather than their single scores. A real regression shows up as a consistent gap: the new state’s runs cluster below the old state’s runs, and the gap exceeds the run-to-run spread of either. Noise shows up as overlapping clouds: the two sets of runs interleave, and the “drop” you saw was one low draw of the new state against one high draw of the old. Testing whether the gap is real is exactly the significance question, and the established treatments of significance testing in natural-language evaluation lay out which tests suit which metrics and how easily an underpowered comparison declares noise to be signal or signal to be noise [R10]. The practical version is humble: if the clouds overlap, you do not have a regression, you have a suspicion, and the correct next step is more runs or a larger suite, not a bisect through your commit history.

A worked regression scare

A concrete case makes the procedure less abstract. Suppose you update the inference engine and your knowledge suite, previously sitting around 88, comes back at 85 on the run you happen to do right after the update. Three points is alarming; three points is also, on a two-hundred-item suite, roughly one standard error, which the arithmetic of chapter 2 already warned you about. The wrong responses are equally available and equally tempting: panic and start bisecting the engine’s changelog, or shrug and assume it is noise because you would prefer it to be. Both skip the measurement.

The right response is to rerun both engine builds now, interleaved, several times each, on the same machine in the same session. Imagine the new build posts 85.4, 87.9, 86.1, 88.2, 86.8 and the old build posts 88.1, 86.9, 88.4, 87.2, 88.0. The new build’s mean is about 86.9 and the old build’s about 87.7, a gap under a point, and the two sets of runs plainly interleave — an 85.4 from the new build sits below an 86.9 from the old, but an 88.2 from the new sits above it. Overlapping clouds, gap smaller than the spread: this is noise, and the first alarming 85 was a low draw of a build that is fine. You have spent ten runs to avoid a multi-day bisect through a regression that does not exist, which is one of the best trades in the whole discipline.

Now imagine instead the new build posts 84.9, 85.3, 84.6, 85.1, 84.8 while the old build posts 88.0, 87.8, 88.3, 87.9, 88.1. The new build’s runs cluster tightly around 85 and the old build’s around 88, and no run of the new build reaches any run of the old. Non-overlapping clouds, gap several times the spread: this is a real regression, and now the bisect is justified because you have established there is something to find. The same ten runs that dismissed the phantom confirm the real defect. The procedure did not tell you the answer in advance; it told you which of the two situations you were actually in, which is the only thing that distinguishes a wise investigation from a wasted one.

Write down every run

Re-measurement only compounds into knowledge if the runs are recorded, and recorded before you know whether you like them. A logbook that captures every run — its configuration, its score, the session it belonged to, and one line of context — is what turns a scatter of measurements into a distribution you can reason about later. Without it, you are back to the personal file drawer of chapter 1, where memory has already discarded the runs that did not fit the story you now tell. The discipline is to append a row the moment a run finishes, not to curate a highlights reel at the end.

The payoff is largest for a session-bound operator that will not remember today tomorrow. When the author reruns a suite, each run’s number and apparatus go into a durable record at once, so that a later session — a different instantiation with no memory of this one — can see the full spread rather than a single remembered figure and can tell a stable configuration from a lucky one. The logbook is also what makes optional-stopping discipline enforceable after the fact: if every run is recorded, a reviewer can see whether you reported all of them or cherry-picked, and that visibility is itself a deterrent against the cherry-picking. A number you can defend is a number whose siblings are all written down beside it.

The traps of re-measurement

Re-measurement has its own failure modes, and the first is re-measuring the wrong thing. Rerunning a number does not validate it if the second run shares the same systematic error as the first. A harness bug that misparses one answer format will misparse it identically on every rerun; contamination that leaks test answers into training will inflate every run by the same amount. Repetition characterizes random error and is blind to systematic error, so a number can be perfectly reproducible and perfectly wrong. Re-measurement earns its keep against luck and instability; it earns nothing against a bug that fires the same way every time, which is why the later chapters on reading the logs and isolating the variable are not optional extras but the other half of the method.

The second trap is re-measuring until you like the answer. Running a configuration repeatedly and stopping when it finally posts a good number is the lucky-draw fallacy wearing the costume of diligence. If you run five times and report the best, you have taken a maximum over noise, and the more you rerun the higher that maximum climbs regardless of the truth — the same upward bias that inflates leaderboard bests [R1]. The rule that keeps re-measurement honest is to decide the number of runs before you look at them, or to report all of them, and never to let the results you have already seen decide how many more to collect. Optional stopping — peeking, then deciding whether to keep going based on the peek — quietly invalidates the statistics you are about to compute, because you have let the data choose the sample size.

The third trap is subtler and specific to systems that adapt over a session. Some apparatus-level state warms up: caches fill, memory fragments, thermal throttling engages after sustained load, a server’s batching behavior shifts as its queue depth changes. Runs performed back to back are not always independent draws from a stationary process; the tenth run of an hour-long session can systematically differ from the first. When this is a risk, randomize or interleave the order of treatment and control runs rather than doing all of one then all of the other, so that any warm-up drift is shared between the conditions instead of confounded with them. Interleaving is the re-measurement analogue of the matched control: it keeps the drift from masquerading as your variable.

A fixed seed hides variance; it does not remove it

A tempting shortcut promises to make re-measurement unnecessary: fix every seed, and the run becomes reproducible, so why run it twice? The shortcut misunderstands what a seed does. Fixing the seed pins one particular path through the randomness, so that path repeats exactly — but it does not shrink the variance of the underlying process; it merely hides it behind a single frozen draw. You have not removed the wobble; you have stopped looking at it, and the number you now report so reproducibly is one arbitrary sample from a distribution you can no longer see.

This matters because the frozen draw might be lucky or unlucky, and you have thrown away the ability to tell which. Two configurations each run at a fixed seed will differ partly because of the variable you care about and partly because their frozen draws landed at different points in their respective distributions, and the fixed seed makes that second contribution invisible rather than absent. A seed is genuinely useful — it makes a specific run reproducible for debugging, and it lets a reader re-execute your exact computation — but it is a tool for reproducibility, not a substitute for measuring variance. The honest practice is to vary the seed across your repeated runs precisely so the distribution shows itself, and to fix it only when you want one specific run to be repeatable for inspection. Reproducing a lucky draw a thousand times does not make it any less a lucky draw; it only makes you more confident in it, which is exactly the wrong direction. And on a served, batch-nondeterministic system the fixed seed does not even deliver reproducibility, because the numerics still shift with the batch — so the shortcut fails on its own terms in the very setting where it is most often reached for.

How much re-measurement is enough

There is no universal number of runs, and asking for one misunderstands the goal. The goal is to reduce the uncertainty in the decision you are about to make to below the point where it would change the decision. If you are choosing between two configurations that are ten points apart, two runs each that agree is plenty — the decision is not in doubt. If they are one point apart, no realistic number of runs on a small suite will separate them, and the honest decision is “indistinguishable, choose on other grounds” rather than an ever-larger pile of runs chasing a difference smaller than the noise. Re-measurement is instrumental, not ritual: you rerun until the answer to your actual question stops depending on which draw you happened to get, and then you stop.

The instinct to internalize is a reflex, not a formula. A surprising number should feel unfinished until it has been run again — incomplete in the way a sentence without a verb is incomplete — so that reaching for the rerun becomes automatic rather than something you do when you remember to be rigorous. The mantis does not strike on the first flicker of movement; it confirms the target is real, and where it is, before it commits, because a strike is expensive and a miss teaches the prey to flee. A published number is a strike. Confirm the target first.