Chapter 3 — Run the Control That Isolates the Variable

Draft status: author draft, gate-checked; human verification pending. The measured observations are the author’s own, on the apparatus named in the provenance page; the external claims resolve to the cited references.

The subtraction that isn’t

The most seductive sentence in benchmarking is “we changed X and the score went up by three points, so X is worth three points.” It is seductive because it has the grammatical shape of a controlled experiment while frequently being nothing of the kind. The score went up by three points; that part is often true. Whether X caused it depends entirely on what else moved between the two measurements, and in a modern inference stack a great many things move whenever you touch one of them. The subtraction is only valid when X is the sole difference between the two runs. When it is not, you have measured the sum of several changes and attributed the whole sum to the one you were thinking about.

The remedy is the oldest idea in experimental science and it does not get less important for being old: change one thing at a time, and run the baseline that differs from your treatment in exactly that one thing. The baseline is the control. Its entire job is to absorb everything you did not mean to test, so that the difference between control and treatment is attributable to the variable and nothing else. A benchmark result without a matched control is not an experiment; it is an observation, and observations cannot support causal claims no matter how many decimal places they carry.

Why one change at a time is hard here

The advice sounds trivial until you try to obey it on a real system, where “changing X” routinely drags several other things along by necessity. Switch a model from full precision to a quantized build and you have changed not only the weights’ precision but very possibly the file size, the memory footprint, the layer placement across devices, the kernels that get selected, and the amount of computation that spills to the CPU. Any of those can move a score. If the quantized build also happens to run more layers on the GPU because it now fits, and it scores higher, you cannot tell whether the quantization helped or whether simply keeping more of the model off the CPU helped. Two variables moved; you get one number.

The deeper reason this is hard is that inference stacks are built for performance, not for experimental hygiene, and performance systems adapt. Change the size of the model and the engine re-decides how to shard it; change the batch and the kernels re-decide how to tile the math; change the available memory and the runtime re-decides what to keep resident and what to recompute. These adaptations are the whole reason the system is fast, and they are exactly what make a clean one-variable comparison difficult, because the system will helpfully change five other things to accommodate the one thing you changed. Obeying “one variable at a time” often means fighting the runtime — pinning the shard layout, forcing a fixed batch shape, disabling an optimization — so that the thing you did not mean to vary stays put. The extra effort is not fussiness; it is the difference between measuring your variable and measuring the runtime’s reaction to your variable.

I have this confound on record in an unusually clean form. A widely shared community quantization of a large model was, by its published table, both larger on disk and lower on a knowledge benchmark than the untouched original — roughly 175 gigabytes at 85.0 against the original’s 160 gigabytes at 88.3. The naive reading, “quantization costs three points,” is wrong twice over. It is wrong because the “quantized” build was in fact larger, which already falsifies the premise that it was a compression; and it is wrong because at least two things differed between the two artifacts — the encoding of the weights and the total size, hence the memory behavior — so even the direction of the knowledge drop cannot be cleanly assigned to precision. The honest statement is narrow: this particular artifact, produced by this particular recipe, measured worse on this harness. To learn what precision alone costs, you would have to hold size, placement, and harness fixed and vary only the bit-width, which is a different and more careful experiment than downloading two files and subtracting their headline numbers.

The off-run

The purest control is the one that turns the variable off. If you are testing whether a feature helps, the matched baseline is the identical system with that feature disabled and nothing else touched. The value of the off-run is that it converts a theory into a measured fact, because it removes the last excuse: whatever difference remains cannot be attributed to any of the machinery both runs share, since they share all of it but the switch.

My clearest example concerns speculative decoding. A speculative decoder uses a small, fast draft mechanism to propose several tokens, which the full model then verifies in parallel; accepted tokens are emitted without a full forward pass each, so throughput rises. The obvious worry is that speculation might change the output, and therefore the quality, and a benchmark run of the speculative system alone cannot lay that worry to rest — because if the score differs from what you remember, you cannot tell whether speculation changed the answer or whether the difference is the ordinary run-to-run wobble from the previous chapter. The off-run settles it. Run the identical system with speculation disabled, on the identical suite, and compare. When I did this, the quality was indistinguishable between speculation-on and speculation-off within the run-to-run spread, which let me state as a measured fact — not a hopeful theory — that correct speculative decoding does not change output quality. That statement is only licensed by the control. Without it, “speculation doesn’t hurt quality” would have been a guess dressed as a result.

The reason the off-run is trustworthy for speculation specifically is worth understanding, because it generalizes. A correctly implemented speculative decoder is designed so that a verified token is exactly the token the full model would have produced on its own; the draft only proposes, and the full model’s distribution decides what is accepted. So the output should be identical up to the same floating-point non-determinism that afflicts any run, and the off-run is what confirms the implementation actually honors that design rather than merely claiming to. A control does not only isolate a variable; it audits whether your system behaves the way its design promises.

The historical baseline is not a control

A specific and common way to lose the control is to compare against a number from the past. You measure your treatment today and compare it against a score you recorded last month, or against the figure printed on a model card, or against a leaderboard entry someone else produced. This feels like a comparison, and it is — but it is not a controlled one, because between then and now the apparatus almost certainly drifted. The engine was rebuilt, a kernel was updated, the harness was revised, the machine ran under different load, or the earlier number was itself produced on hardware and software you cannot fully reconstruct. The difference you compute is the effect of your change plus the effect of every drift since the baseline was taken, and you cannot separate them.

The discipline is inconvenient but simple: a control must be run now, on this apparatus, alongside the treatment. A remembered number is a hypothesis about what the control would say, not the control itself. When I want to know what a change does, I rerun the baseline in the same session, on the same build, on the same machine, immediately before or after the treatment, so that whatever drifted, drifted for both. This costs a full extra run every time, and it is the single most common corner cut in practice, precisely because it feels redundant — you have a baseline number, why measure it again? Because the number you have was measured by a different apparatus, and the whole point of a control is that it shares the apparatus with the treatment. A baseline you did not rerun is a baseline you are only pretending to have.

Confounds have a taxonomy

It helps to know the shapes confounds take, because once you can name them you start seeing them before they ruin a comparison. The first is the bundled change: you meant to vary one knob, but turning it necessarily turned others, as with the quantization that also changed size and placement. The second is the drifting apparatus: you varied the knob cleanly, but something in the environment changed between the two runs — a different engine build, a different concurrent load on the server, a different time of day when the machine was hotter and throttled. The third is the selection confound: the two runs are not on the same items, because the harness sampled a different subset, or excluded errored items differently, so you are comparing scores on two different suites. The fourth is the measurement confound: the metric itself changed, because a scoring script was edited, or a template was updated, or a tie-break rule differs between the two runs.

Each has the same antidote — hold it fixed — but they hide in different places, so a checklist that only guards against bundled changes will be blindsided by a drifting apparatus. The standardized-harness projects exist precisely to freeze the measurement and selection confounds across everyone who uses them: when two people run the same task specification in the lm-evaluation-harness, the items, the templating, and the scoring are shared, so the remaining differences are more likely to be about the models [R4]. HELM makes the same move at the level of a whole comparison, fixing the conditions and reporting many metrics so that one promoter’s favorable choice of template cannot masquerade as model quality [R3]. Using a shared harness is not a bureaucratic nicety; it is how you eliminate two of the four confounds for free.

Ablations, or one variable at a time taken seriously

When a change is really several changes bundled together — a new recipe that adjusts precision, placement, and a prompt template at once — a single before-and-after tells you only that the bundle helped or hurt, not which part did the work. The tool for taking apart a bundle is the ablation: a series of runs in which you re-enable the pieces one at a time, starting from the baseline, so that each run differs from the previous by exactly one component. The difference between consecutive runs is the isolated contribution of the piece you just added, and the run that finally matches the full treatment confirms you have accounted for all of it.

Ablations are how you convert a lucky recipe into understanding. My own work on expert precision in mixture-of-experts models proceeded this way: rather than declaring “this quant recipe is good,” I varied the precision of the routed experts alone, holding the attention layers, the norms, and the harness fixed, and watched two different capabilities respond differently to the same knob. Knowledge-style accuracy degraded gently and recovered once the experts had enough bits; tool-calling ability fell off a cliff at low precision and needed markedly more bits to come back. Neither fact is visible from a single headline score of a single recipe. They are visible only because each run changed one thing, so each drop or recovery could be assigned to the one thing that changed. An ablation is more expensive than a single comparison by exactly the number of components you are separating, and it is the price of being able to say why rather than merely whether.

The discipline has a natural stopping point, which is worth respecting so ablations do not sprawl. Separate the components that plausibly carry the effect and that you might set independently in practice; do not ablate combinations that never occur together or knobs you would never move alone. The goal is an explanation you can act on, not a full factorial of every switch in the system.

Pairing beats averaging

An unpaired comparison — the average of the treatment runs minus the average of the control runs — makes each side see through the full variability of the items, and the shared difficulty of the suite is baked into both averages as noise. A paired design cancels that shared difficulty. Run control and treatment on the same items, compare them item by item, and the intrinsic hardness of each question drops out of the per-item difference, leaving only the effect of the variable. The listing below simulates paired evaluation — the same items, the same “luck” seen by both systems, with the treatment shifting each item’s success threshold by a real five points — and reports the standard error of the effect computed both ways.

import random, math

rng = random.Random(30)
n = 500
effect_true = 0.05
control, treat, diffs = [], [], []
for _ in range(n):
    b = rng.uniform(0.35, 0.90)   # this item's shared difficulty
    u = rng.random()              # this item's shared luck, seen by both systems
    c = 1 if u < b else 0
    t = 1 if u < b + effect_true else 0
    control.append(c); treat.append(t); diffs.append(t - c)

pc, pt = sum(control) / n, sum(treat) / n
effect = pt - pc
se_unpaired = math.sqrt(pc * (1 - pc) / n + pt * (1 - pt) / n)
md = sum(diffs) / n
var = sum((d - md) ** 2 for d in diffs) / (n - 1)
se_paired = math.sqrt(var / n)
print(f"effect (both views): {effect*100:+.2f} pts")
print(f"unpaired SE: {se_unpaired*100:.2f} pts  ->  z = {effect/se_unpaired:.2f}")
print(f"paired   SE: {se_paired*100:.2f} pts  ->  z = {effect/se_paired:.2f}")
print(f"items that disagreed: {sum(1 for d in diffs if d)} of {n}")

effect (both views): +5.00 pts
unpaired SE: 3.00 pts  ->  z = 1.67
paired   SE: 0.98 pts  ->  z = 5.12
items that disagreed: 25 of 500

The realized effect on this seed is +5.00 points — twenty-five of five hundred items flipped, which is exactly the five-point shift the code injected; a different seed would land a little above or below that, since the number of flips is itself a draw. The point estimate is the same whether you compute it paired or unpaired — the effect is the effect — but the paired standard error is a third of the unpaired one, and the signal-to-noise ratio triples. The reason is visible in the last line: of five hundred items, only twenty-five ever disagreed between the systems, and the entire effect lives in those twenty-five. The unpaired view forces you to detect that concentrated signal through the variance of all five hundred items on both sides; the paired view looks only where the systems actually differ. When you can run control and treatment on identical items — and with a fixed suite you almost always can — pairing is close to free statistical power, and abandoning it throws away sensitivity you have already paid for. This run shows exactly that trade in action: the unpaired z of 1.67 falls short of the conventional 1.96 threshold, so the unpaired test would report “no significant difference,” while the paired z of 5.12 clears it decisively on the same data. A difference that clears significance paired but not unpaired is not a contradiction; it is the paired test correctly using information the unpaired test discarded.

Negative controls catch a broken harness

There is a second kind of control worth running, aimed not at your variable but at your apparatus: the negative control, a condition whose result you already know, run to confirm the harness is behaving. The idea is borrowed from wet-lab science, where a well that should show nothing is run alongside the experiment precisely so that a contaminated reagent announces itself. In benchmarking, a negative control is an input the system should score at a known, uninteresting value, and a departure from that value is a bug in the measurement rather than a property of the model.

Several are cheap and catch real problems. A model given a multiple-choice suite with the answer choices shuffled but the scoring key not updated should score at chance; if it scores far above chance, your scorer and your data have drifted out of alignment. A deliberately empty or nonsense prompt should not produce a passing answer; if it does, your scorer is matching something other than correctness — a stray substring, a default value, a lenient parse. A random-guessing baseline on a four-choice task should land near twenty-five percent; a harness that reports it at zero or at fifty is mis-scoring, and every real number it produces is suspect. I reach for these whenever a result is surprisingly good, because a suspiciously high score is as often a scorer that has been fooled as a model that has excelled, and the negative control tells the two apart in one run. A positive result that survives its negative controls is worth far more than one that was never checked against a value you already knew.

The control tells you when to stop

A well-chosen control does more than validate a positive result; it tells you when a measurement is not worth trusting yet. If your treatment and your control differ by less than the control’s own run-to-run spread, you have not measured an effect — you have measured noise, and the correct output is “no detectable difference at this suite size and run budget,” not a hopeful point estimate. This is the same discipline as the previous chapter’s error bar, applied to a comparison rather than a single number: the difference between two configurations gets its own uncertainty, and a difference smaller than its uncertainty is not a finding. Running the control is what generates that uncertainty for free, because the control’s spread is the yardstick.

There is a failure I have committed and want to warn against by name: reasoning about a control instead of running it. It is easy, and tempting, to argue that a change cannot affect quality — the tokens are identical, the design guarantees it, the math says so — and to skip the off-run on the strength of the argument. Sometimes the argument is right. But the whole thesis of this book is that arguments are not measurements, and the times the argument is wrong are exactly the times you most need to know. The off-run is cheap insurance against a confident theory, and confident theories are precisely what benchmarks exist to check. Run the control even when — especially when — you are sure you know what it will say.