Chapter 2 — Error Bars Before Claims

Draft status: author draft, gate-checked; human verification pending. The listings in this chapter are pure–standard-library Python, deterministic under the seeds shown, and were executed by the author during writing; the printed outputs are real transcripts.

Where the wobble comes from

A benchmark score has at least three independent sources of variation, and confusing them is the root of most misreported results. The first is sampling variation: your suite is a finite draw from the larger space of questions you actually care about, and a different draw would score differently. The second is decoding variation: if you generate with any randomness — a temperature above zero, top-p sampling, a non-fixed seed — the same model on the same question can produce different answers. The third, and the one that ambushes careful people, is execution variation: the same model, same input, same seed, run twice, can still produce different outputs because the arithmetic underneath is not bit-for-bit reproducible across runs.

Each source calls for a different response, so it pays to keep them separate. Sampling variation you quantify with the size of your suite and the arithmetic of proportions. Decoding variation you either eliminate, by generating greedily, or you embrace and average over, by running many samples. Execution variation you first have to believe in, because it is genuinely surprising, and then either suppress or report. A single “error bar” that lumps all three together is better than none, but you will make sharper decisions if you know which source dominates your particular measurement.

Determinism is a promise the stack does not keep

The most common way to convince yourself a measurement is exact is to set the temperature to zero and take greedy decoding, so that each next token is the arg-max of the model’s distribution. With no sampling, the reasoning goes, the output is a deterministic function of the input, and repeated runs must agree. On a single request, on a single device, with a fixed software stack, this reasoning very nearly holds. It stops holding the moment the request shares a batch with other requests, which is the normal condition of any served model.

The reason is floating-point arithmetic. Addition of floating-point numbers is not associative: (a + b) + c can differ from a + (b + c) in the last bits, because each intermediate result is rounded. The large reductions inside a transformer — summing across the hidden dimension, across the sequence, across experts — are computed in an order that depends on how work was tiled onto the hardware, and that tiling depends on the shape of the batch. When your request is packed next to a short request, the batch is one shape; next to a long one, another; and the reduction order shifts. Most of the time the last-bit differences vanish under the arg-max. Occasionally they land on a near-tie between two candidate tokens and flip the choice, and from that point the two generations diverge. The PyTorch project documents the underlying non-reproducibility candidly: results are not guaranteed to be bit-for-bit reproducible across different hardware, different software versions, or even different batch sizes, and some operations have no deterministic implementation at all [R5].

A detailed 2025 analysis from Thinking Machines traced this precise mechanism in served LLM inference and named the culprit: the lack of batch invariance in the kernels. The numerics a request sees depend on what else is in its batch, which depends on concurrent load, which is not under the requester’s control — so the “deterministic” temperature-zero endpoint is only deterministic for a batch of one, and a production server almost never serves a batch of one [R6]. Their write-up also shows that the effect is fixable, with batch-invariant kernels, at some throughput cost — which matters, because it means execution variation is a property of your serving configuration, not a law of nature, and you can choose to pay to remove it when a measurement demands it.

I met this the hard way, and the encounter is the origin of this book’s title. Running a fifteen-scenario tool-calling suite at temperature zero against a large mixture-of-experts model, on a Threadripper workstation with three Blackwell GPUs, I recorded scores about ten points apart on two runs of an unchanged binary with a fixed seed. My first assumption was a bug in the harness; my second was a bug in the model; both were wrong. The variation was batch-packing nondeterminism doing exactly what the analysis above describes, amplified by a short suite where one flipped tool call is worth several points. The number that a single run would have reported was, in the strict sense, a random variable, and I had been treating its draws as facts.

The size of a sampling error bar

Before touching the harder sources, it is worth being fluent in the easy one, because it sets a floor on how precise any score can possibly be. When a benchmark reports accuracy — the fraction of items answered correctly — the result is a proportion, and the sampling standard error of a proportion has a closed form. If the true accuracy is p and the suite has n independent items, the standard error of the observed accuracy is the square root of p times one-minus-p, divided by the square root of n. The first listing computes it and prints the resulting rough 95 percent interval for a few suite sizes at a realistic accuracy.

import math

def stderr_proportion(p, n):
    return math.sqrt(p * (1.0 - p) / n)

p = 0.84
for n in (50, 200, 1000, 5000):
    se = stderr_proportion(p, n)
    half = 1.96 * se
    print(f"n={n:5d}  se={se*100:5.2f} pts   95% ~= {p*100:.1f} +/- {half*100:.1f} pts")

The transcript makes the point that no amount of care about the model can rescue you from a small suite:

n=   50  se= 5.18 pts   95% ~= 84.0 +/- 10.2 pts
n=  200  se= 2.59 pts   95% ~= 84.0 +/- 5.1 pts
n= 1000  se= 1.16 pts   95% ~= 84.0 +/- 2.3 pts
n= 5000  se= 0.52 pts   95% ~= 84.0 +/- 1.0 pts

Fifty questions cannot distinguish an 84 from a 74; even a thousand questions leaves a two-point interval. This is a hard floor set by the arithmetic, before you add decoding or execution noise on top. Any claim of a one-point improvement measured on a few hundred items is, on sampling grounds alone, a claim about noise. The formula assumes independent items, which real suites violate — clustered topics, repeated templates, and contamination all correlate the errors — and dependence almost always makes the true interval wider than this floor, never narrower. Treat the closed form as the most optimistic error bar you are entitled to, and reach for resampling when you want a number that respects your actual data.

The bootstrap: an error bar for anything

The closed form works for a plain proportion, but real evaluations report messier quantities — a mean score with partial credit, a pass@k on code, a weighted average across task groups — and deriving a formula for each is tedious and error-prone. The bootstrap sidesteps the algebra entirely. Its idea is disarmingly simple: your suite is a sample from a population, so treat the suite itself as the population and draw new samples from it, with replacement, thousands of times. The spread of the statistic across those resamples estimates the spread you would have seen across fresh suites. The technique is standard and well described in the statistics literature [R7]; the second listing implements it in the standard library, seeded so the transcript reproduces exactly.

import random, statistics

def bootstrap_ci(scores, iters=10000, alpha=0.05, seed=0):
    rng = random.Random(seed)
    n = len(scores)
    means = []
    for _ in range(iters):
        resample = [scores[rng.randrange(n)] for _ in range(n)]
        means.append(sum(resample) / n)
    means.sort()
    lo = means[int((alpha / 2) * iters)]
    hi = means[int((1 - alpha / 2) * iters)]
    return statistics.fmean(scores), lo, hi

# 200 graded items: 168 correct (1.0), 32 wrong (0.0) -> 84.0% observed
scores = [1.0] * 168 + [0.0] * 32
point, lo, hi = bootstrap_ci(scores)
print(f"observed  {point*100:5.2f}%")
print(f"95% CI    [{lo*100:5.2f}%, {hi*100:5.2f}%]  width {round((hi-lo)*100,2)} pts")

observed  84.00%
95% CI    [78.50%, 89.00%]  width 10.5 pts

The bootstrap interval on two hundred items brackets roughly plus-or-minus five points, landing close to the closed-form proportion above — reassuring, but not independent confirmation, because both are drawn from the same two hundred items and both lean on the same large-sample approximation. That shared assumption is worth a caveat, because for a plain proportion neither of these two intervals is the textbook-correct one. The normal ±1.96·se interval and the naive percentile bootstrap both under-cover when the accuracy sits near zero or one or when the suite is small: the true 95% interval catches the real value less than 95% of the time, and it can even run past 100% at the top of the scale. The defensible default for a bare proportion is the Wilson score interval or the Clopper–Pearson exact interval, both of which are built for binomial data, stay inside [0, 1], and hold their coverage where the normal approximation frays; a statistics library gives you either in one call. Use the bootstrap not because it is the best interval for a proportion — it is not — but because it is the one tool that keeps working when your metric stops being a plain proportion. The payoff is generality: change scores to per-item partial credit, or to a list of per-run whole-suite scores, and the same seven lines give you an interval with no new algebra, where a closed form would need re-deriving. When you report a benchmark result, reach for Wilson or Clopper–Pearson if the metric is a simple accuracy, and for the bootstrap once it is anything messier; either way the interval is the error bar that turns the number into a claim you can defend.

Reporting a range, not a point

Knowing the wobble is useless until it changes what you write down. The rule is to report the interval and let the point sit inside it, rather than reporting the point and mentioning the interval as an afterthought. “84.0 (95% CI 78.5–89.0, n=200, greedy, single run)” communicates honestly in one line: the best estimate, its uncertainty, the suite size, the decoding policy, and the run count. A reader can immediately see that this result cannot adjudicate a two-point difference, and will not embarrass themselves by trying.

For execution and decoding variation, the interval you want is across runs, not across items, and you get it by running the whole suite several times and treating the per-run scores as your sample. Feed those run-level scores into the same bootstrap, or simply report their mean and standard deviation. The distinction matters: an item-level interval tells you how much the suite limits you; a run-level interval tells you how much the stack limits you. On a batch-nondeterministic server, the run-level spread can dwarf the item-level one, which is the whole reason a single greedy run is not the safe measurement it appears to be.

Two practices make run-level intervals affordable. First, pin everything you can — a fixed seed, a fixed engine build, and, where the stack offers it, batch-invariant or single-request execution — so that the residual spread is small and honestly attributable. Second, when you cannot pin execution, budget for repetition: three to five full runs is usually enough to see whether the spread is a fraction of a point or a chasm, and the answer decides how many decimal places you are allowed to print. Dodge and colleagues make the broader case that reporting the distribution of results, and the budget that produced them, is what lets a reader reason about your numbers at all [R1]; the run-level error bar is the smallest honest version of that report.

How many runs is enough

The question every practitioner asks next is how many times to run the suite, and the honest answer is that the data tells you rather than a rule of thumb. Start with two full runs of the configuration you care about. If they agree to within a small fraction of the precision you need — the two tool-suite runs that landed ten points apart did not — then a third run is mostly confirmation and you can report a mean with a note that the spread was negligible. If they disagree by more than the difference you are trying to detect, you have learned something more valuable than a score: you have learned that this measurement is execution-dominated, and no single run of it means anything. Now you run enough repetitions to characterize the spread, typically five, and you report the mean with its run-level interval rather than any individual figure.

There is a temptation to economize by running a smaller suite more times, or a larger suite fewer times, and the two are not interchangeable. Repetitions of the whole suite characterize execution and decoding variation; a larger suite shrinks sampling variation. If your spread is dominated by batch nondeterminism, adding items does nothing and adding runs is the cure; if your spread is dominated by a small suite, adding runs of the same short suite just measures the same small sample repeatedly. Diagnose which source dominates — two runs versus a back-of-envelope proportion error bar usually reveals it — and spend your compute on the source that is actually hurting you. Spending it on the other is a common and expensive mistake.

Combining the sources without fooling yourself

The three sources of variation compound, and a subtle error is to measure one and quietly present it as the whole. An item-level bootstrap on a single run gives a tight, honest-looking interval that accounts only for sampling — it is blind, by construction, to the fact that a second run of the same suite might have scored three points lower from batch nondeterminism. Publishing that tight interval while execution variation is large is a way of being precisely wrong: the number carries an error bar, the error bar is real, and it is answering a question you were not asking. The reader will assume the interval covers run-to-run reproducibility, because that is what they care about, and it does not.

The clean approach keeps the two intervals distinct and reports whichever is larger, or both. Run the suite several times; for each run you have a whole-suite score. The spread of those run-level scores captures execution and decoding variation directly, with no modeling assumptions at all — it is simply what happened when you ran it again. Within any single run, an item-level bootstrap captures sampling variation. In a well-behaved, pinned configuration the item-level interval dominates and the run-level spread is a rounding error, and you can say so. In a batch-nondeterministic served configuration the run-level spread dominates, and it is the number that governs how many digits you may honestly print. Either way, the reader is owed the larger of the two, because a decision made on the smaller one is a decision made on a fiction.

A reporting template you can reuse

It helps to fix a written form so that reporting the full claim becomes automatic rather than a thing you remember to do when you have time. The author’s own logbook records every headline number as a single line with a fixed shape: the score, the interval and what kind it is, the suite size, the number of runs, the decoding policy, and the exact system that produced it — engine build, cache precision, and hardware. “84.0, run-level 82.1–85.6 over 5 runs, n=200, greedy, engine b1234 + f16 KV, 3×Blackwell” is long, and its length is the point: every field is a lever that can move the score, so every field must travel with it or the number cannot be reproduced or trusted. A table of such lines is auditable at a glance; a table of bare scores is a table of rumors with good posture. The discipline costs one line of typing per result and saves the reader — often a future version of yourself with no memory of today — from reconstructing the apparatus from nothing.

Decoding variation and the temptation of the maximum

When a task is scored by sampling multiple generations — pass@k on code, self-consistency on reasoning — decoding variation is not a nuisance to suppress but a quantity to estimate carefully, and it hides a trap. The pass@k metric introduced with the HumanEval code benchmark is itself an estimator: it estimates the probability that at least one of k samples passes, from a larger number of samples, precisely because the naive “generate k, check if any pass” is a high-variance draw [R8]. Reporting a single lucky pass@1 from one generation per problem is the decoding-variation version of the lucky draw from chapter 1, and the fix is the same in spirit: estimate the expectation, and report its spread.

The temptation, always, is to take a maximum — the best of several sampled runs, the best of several temperatures, the best checkpoint — and report it as the result. A maximum over noisy draws is biased upward by construction, and the more draws you take, the more it inflates. Nucleus sampling was introduced partly because greedy and pure-sampling decoding sit at opposite failure modes, and the choice of decoding policy is itself a lever on the distribution of outputs you are measuring [R9]. The honest move when decoding is random is to fix the policy, state it, sample enough to estimate the quantity you claim, and report that quantity with its interval — never the smiling maximum.

Decimals are a claim about precision

A small typographic habit leaks dishonesty into otherwise careful reports: printing more decimal places than the error bar can support. A score written as 84.37 announces, by its three significant figures, that you can distinguish it from 84.28 — a claim of hundredth-of-a-point resolution. On a two-hundred-item suite whose sampling error alone is two and a half points, that claim is absurd, and the extra digits are not precision but decoration that misleads the reader into a confidence the measurement cannot bear. The number of digits you print is itself an assertion about your uncertainty, and it should agree with the error bar sitting next to it.

The rule is to round the point estimate to the resolution its interval justifies, and to let the interval carry the story. If the interval is plus-or-minus five points, “84” is the honest rendering and “84.37” is a small lie of overstated precision. If you have driven the interval down to a few tenths through a large suite and many runs, then a decimal place is earned and should be shown. Matching your digits to your uncertainty costs nothing and inoculates the reader against the most common visual trick in benchmark reporting, which is a wall of decimals implying a precision that no suite of that size could ever deliver. When in doubt, print fewer digits and show the interval; a reader can always compute a finer number from your data, but they cannot un-see a false one.

What an error bar is not

An error bar quantifies variation; it does not certify correctness. A tight interval around a wrong number is still wrong. If your harness scores a correct answer as incorrect because of a whitespace mismatch, running it a thousand times gives you a beautifully tight interval around a systematically depressed score. Sampling error and execution error are random errors, and repetition characterizes them; harness bugs and contamination are systematic errors, and repetition only makes you more confident in the wrong value. The remaining chapters attack the systematic errors directly — by isolating the variable, by re-measuring against a control, by reading the logs, and by publishing the result that does not fit — because the error bar, essential as it is, is only ever half of the truth. It tells you how much a number would move if you ran it again. It cannot tell you whether the number was ever measuring the right thing.