Chapter 5 — Small Suites Swing Hard

Draft status: author draft, gate-checked; human verification pending. The listing is pure–standard-library Python, deterministic under the seed shown, and was executed by the author during writing; the printed output is a real transcript. External claims resolve to the cited references.

The tyranny of the denominator

The single most under-appreciated number in a benchmark report is the one that usually goes unmentioned: how many items the score was computed over. That denominator governs everything. A score is a fraction, and the smaller its denominator, the more it lurches with each item that flips. On a ten-item suite, one item is ten points. On a fifty-item suite, one item is two points, and a suite that small cannot resolve any difference finer than that no matter how carefully you run it. The precision you are entitled to claim is bounded from below by the size of your suite, and that bound is often far coarser than the differences people confidently report.

The arithmetic from chapter 2 gave the bound in closed form — the standard error of a proportion falls only as fast as the square root of the sample size — which has a discouraging consequence: halving your error bar costs four times the items. There is no cheap route to precision through cleverness; precision on a proportion is bought by the item, in quadratically increasing quantities. A ten-point swing does not mean your harness is broken or your model is unstable. On a short suite it can mean nothing more than that the suite is short, and the swing is the denominator doing exactly what a small denominator does.

Ranking is even harder than scoring

Most benchmarking is not really about a single score; it is about a comparison — is A better than B, did my change help, which model should ship. Comparisons on small suites are harder than they look, because the difference between two noisy numbers is noisier than either number alone. To see how bad it gets, it is worth simulating directly, and it is worth being explicit about which comparison you are simulating, because the answer depends on it. The listing runs two regimes side by side for a realistic case where A is truly better than B but only by half a percentage point. The first regime is unpaired: A and B are each run once on their own suite of size n, with their own independent luck, and you rank them by those two single numbers — the situation you are in when you compare a score you measured against a score someone else published. The second regime is paired in the sense of chapter 3: both systems are graded on the same items with the same luck, so the shared difficulty cancels. Counting how often the worse system B is nonetheless ranked ahead of A shows how much that one design choice matters.

import random

def ranking_accuracy(true_a, true_b, n, trials=2000, seed=0):
    rng = random.Random(seed)
    up_wrong = pr_wrong = pr_tie = 0
    for _ in range(trials):
        # UNPAIRED: two independent single runs, each with its own luck.
        sa = sum(1 for _ in range(n) if rng.random() < true_a)
        sb = sum(1 for _ in range(n) if rng.random() < true_b)
        if sb > sa: up_wrong += 1
        # PAIRED: both systems graded on the SAME per-item luck (chapter 3).
        pa = pb = 0
        for _ in range(n):
            u = rng.random()
            if u < true_a: pa += 1
            if u < true_b: pb += 1
        if pb > pa: pr_wrong += 1
        elif pa == pb: pr_tie += 1
    return up_wrong / trials, pr_wrong / trials, pr_tie / trials

print("true gap 0.5 pt (A=84.5%, B=84.0%): rate the WORSE system B is ranked first")
print(f"{'n':>7}  {'unpaired B-first':>16}  {'paired B-first':>14}  {'paired tie':>11}")
for n in (100, 500, 2000, 10000):
    up, prw, prt = ranking_accuracy(0.845, 0.840, n)
    print(f"{n:7d}  {up*100:15.1f}%  {prw*100:13.1f}%  {prt*100:10.1f}%")

true gap 0.5 pt (A=84.5%, B=84.0%): rate the WORSE system B is ranked first
      n  unpaired B-first  paired B-first   paired tie
    100             41.0%            0.0%        59.4%
    500             39.8%            0.0%         7.3%
   2000             34.2%            0.0%         0.0%
  10000             15.7%            0.0%         0.0%

Read the unpaired column first, and read it as one seed’s draw — the percentages jitter by a point or two if you change the seed, but the shape is stable. Compared on a hundred independent items, the genuinely-better configuration is ranked behind the worse one more than four times in ten — a coin flip with a rounding error. Even at ten thousand items, a half-point true difference is called backwards roughly one time in six. This is not a defect of the simulation; it is the reality of comparing close systems by independent single runs, and it is why a leaderboard ordering of configurations that sit within a point of each other — each an independently produced number — is largely a ranking of luck. When you see two models a fraction of a point apart on a suite of a few hundred items, the correct reading is not “the top one is better” but “these are indistinguishable, and the order will likely reverse next week.”

The paired column tells the other half of the story, and it is the reason chapter 3 pressed so hard on pairing. In the idealized shared-luck model — where the better system succeeds on every item the worse one does — pairing never ranks B ahead of A at any suite size, because B can never beat A on an item it can only tie or lose. What pairing cannot do on a small suite is break the tie: at a hundred items the two systems return the identical score almost sixty percent of the time, because the half-point gap is so small that on most draws no item happens to separate them. So pairing does not manufacture a rank out of nothing — it refuses to call a tie a win, which is exactly the honesty a single unpaired number lacks. Real evaluations are not quite this idealized, because two systems’ luck is only partly shared rather than identical, so a real paired comparison sits between the two columns; but it sits far closer to the paired one, and it never suffers the four-in-ten reversal rate that independent single runs do. The lesson is not that pairing is magic but that the unpaired column is the one you are usually reading, and it is much worse than it looks.

Why leaderboards mislead

A leaderboard concentrates every failure mode in this book into one table and adds a new one. It ranks by a single number, so it surfaces lucky draws (chapter 1). It rarely publishes error bars, so the ranking looks more certain than it is (chapter 2). It compares numbers produced by different submitters on possibly different apparatus, so the comparisons are uncontrolled (chapter 3). And it adds a distinctly leaderboard-shaped pathology: the test set is public and reused by everyone, over and over, which quietly destroys its ability to measure generalization.

The mechanism is adaptive overfitting, and it is well understood. Each time someone consults a holdout set to decide which of their models to keep or publish, they leak a little information about that specific holdout into their choices. Do this thousands of times across a whole community and the collective process overfits the public test set, so that scores climb without the underlying capability improving — progress against the leaderboard rather than against the world it was meant to stand in for. The theory of preserving statistical validity under adaptive reuse shows both why naive holdout reuse fails and that the damage is bounded only if the number of adaptive queries is controlled or the holdout is protected [R11]. A public leaderboard is the opposite of a protected holdout: it invites unlimited adaptive queries by construction.

The people who run serious leaderboards know this and fight it, which is itself instructive. Well-run efforts fix the evaluation conditions so that submissions are at least comparable, document their exact task specifications and scoring so results can be reproduced, and normalize across tasks so a single easy benchmark cannot dominate — the Open LLM Leaderboard’s methodology notes are an example of this kind of care made explicit [R12]. Standardized harnesses and holistic, multi-metric evaluations exist partly to make leaderboard-style comparisons less misleading by freezing the shared apparatus [R3][R4]. None of this repeals the arithmetic: even a perfectly run leaderboard, ranking configurations that sit within a point of each other on a finite public suite, is reporting an order that is mostly noise near the top, and reading it as a strict ranking is a mistake the leaderboard’s own methodology page will often warn you against.

The more you test, the more you fool yourself

There is a distinct small-suite hazard that grows with your own diligence, which makes it especially treacherous: the more comparisons you run against a suite, the more likely you are to find a “significant” difference that is pure chance. If you test twenty independent tweaks against a benchmark and use the usual one-in-twenty threshold for calling a result significant, then on average one of the twenty will clear the bar even if every tweak is worthless — that is what a one-in-twenty false-positive rate means. Run enough experiments against the same suite and you are guaranteed to harvest some winners that are nothing but noise, and because you ran them yourself, one at a time, each felt like an honest individual test. This is the multiple comparisons problem, and it is one of the oldest traps in applied statistics [R16].

The defenses are not exotic. When you make many comparisons, raise the bar for each in proportion to how many you made — the Bonferroni correction, dividing your significance threshold by the number of tests, is the blunt and safe version [R17]. Better still, separate exploration from confirmation: use one suite, or one split, to generate hypotheses freely, and a second, untouched suite to confirm the survivors, so that the confirmation set has seen none of your adaptive choices. The instinct to internalize is that a significant-looking result found after many attempts is weaker evidence than the same result found on the first try, and how much weaker depends on how many attempts preceded it — a count you must therefore keep. A win you cannot say how hard you searched for is a win you cannot calibrate.

The composite-score illusion

Many headline benchmarks are averages over many subtasks — dozens of subjects, several skill categories, a basket of datasets rolled into one figure. A composite feels more stable than any single subtask, and in one sense it is: averaging does reduce variance. But the composite hides where its own uncertainty lives, and two failures hide inside it. The first is that a composite can move because one small, noisy subtask swung, while the reader attributes the movement to the whole capability. The second is that a composite can stay flat while large, offsetting changes happen underneath — a gain on one subtask cancelling a loss on another — so that the single number reports “no change” over a system that changed a great deal in ways that matter.

The honest treatment of a composite is to report its components, or at least their spread, alongside the roll-up. A knowledge benchmark spanning many subjects should travel with the range across subjects, not only the mean, because a model that is uniformly mediocre and a model that is excellent at half the subjects and poor at the other half can post the same average while being wildly different in use. My own experience with quantization recipes drove this home: a single averaged score moved little as I lowered precision, while underneath it a knowledge component held up and a tool-calling component collapsed — two behaviors with opposite responses to the same knob, invisible in the average and obvious the moment the components were reported separately. Averaging is a form of compression, and like any compression it discards information; a composite score is only as honest as the breakdown you are willing to publish beside it.

Contamination: when the suite is not a sample at all

Small suites have a second, sharper problem that no amount of care about sample size can fix: if the test items leaked into the model’s training data, the score is not measuring the capability you think it is. A model that has seen the exact questions and answers can recite them, and its score reflects memorization rather than the skill the benchmark was built to probe. On a large, diverse suite a little contamination inflates the score modestly; on a small suite, a handful of leaked items can swing the whole result, because each item is worth so much.

Contamination is not hypothetical, and it is measurable. Work on tracing data contamination in large language models demonstrates that models often perform suspiciously well on the specific splits and phrasings that plausibly appeared in their training corpora, and offers methods to detect when a benchmark instance was likely seen during training [R13]. The broader study of memorization shows that models reproduce training data more as they grow, as examples repeat in the corpus, and as more context is supplied — so the larger and more capable the model, the more seriously contamination must be taken, not less [R14]. A benchmark built years ago, widely copied across the web, and scraped into every subsequent training run has a real chance of being partly memorized, and its scores drift upward over time for reasons that have nothing to do with improving reasoning.

The defenses are practical even if none is complete. Prefer suites whose items are recent enough to postdate a model’s training cut-off, or held-out privately and never posted; probe for contamination by checking whether a model completes a benchmark item from a partial prompt with suspicious fluency; and treat a suspiciously high score on an old, famous, public benchmark as a contamination hypothesis to rule out rather than a triumph to announce. Above all, distrust the premise that a public test set is a random sample from the population you care about. Once it has been on the web long enough to be trained on, it is no longer a sample; it is a memorized answer key of unknown coverage, and its denominator has stopped protecting you.

Reading a leaderboard responsibly

Since leaderboards are not going away, it is worth having a way to read one that respects the arithmetic. Treat the ordering as bands, not ranks. Entries whose scores sit within roughly a standard error of one another belong to the same band and are, on the evidence shown, indistinguishable; the fact that one printed a higher decimal is not information you can act on. Look for the size of the evaluation suite and reconstruct the approximate error bar yourself if the board does not print one — the square-root arithmetic takes ten seconds and immediately tells you how wide each band is. Weight your attention toward gaps that exceed a band and away from the jostling at the very top, which is usually the part most contaminated by luck, adaptive overfitting, and undisclosed apparatus differences. A leaderboard read as a rough sorting into a few tiers is useful; a leaderboard read as a strict order down to the decimal is a way to be confidently wrong on a schedule.

The same caution applies to your own internal leaderboards, the running tables of configurations a team keeps. They accumulate the same adaptive-overfitting debt, because every decision to keep or discard a configuration based on the table leaks a little information about the table’s specific items into your choices. A table consulted a thousand times to pick winners has quietly become a holdout you have overfit, and its numbers have drifted from what a fresh suite would say. The remedy is the same one serious public boards reach for: hold a portion of your evaluation data in reserve, never consulted during development, and spend it only to confirm a decision you have already made on the working set.

Small suites are not useless

None of this means small suites should be thrown away, and it would be a misreading to conclude that only enormous benchmarks are worth running. Small suites are cheap, fast, and invaluable for catching gross failures — a configuration that scores twenty points below the field has a problem you can see on fifty items, and you do not need ten thousand to know a system is broken. The error is not using small suites; it is over-reading them, treating a fifty-item score as if it carried the precision of a five-thousand-item one and adjudicating fine differences it cannot resolve.

The matched-control and pairing techniques from chapter 3 also recover real power on small suites, because they change what you are measuring. A paired comparison on a hundred shared items can detect an effect that an unpaired comparison of two independent hundred-item runs cannot, since pairing removes the shared difficulty that dominates a small sample’s variance. The statistical-power literature in natural-language evaluation lays out how small the detectable effect really is for a given suite size and how to design a comparison that has a fighting chance of finding a true effect rather than merely failing to reject the null [R15]. The takeaway is not “never use small suites” but “know what a suite of this size can and cannot decide, and never let a small denominator write a check the arithmetic cannot cash.”

Estimate the detectable effect before you run

The simulation earlier in this chapter answers a question you can and should ask before committing to a suite: given this many items, how small an effect can I realistically detect? That question has an answer in advance, and computing it turns suite sizing from hope into arithmetic. The statistical-power view frames it precisely — for a given suite size, significance threshold, and true effect, there is a computable probability that your comparison will actually detect the effect, and running a study whose power is low is buying a high chance of a null result that means nothing [R15]. An underpowered comparison does not just risk missing a real effect; it also makes any positive result it does produce less trustworthy, because among low-powered studies a larger fraction of the “wins” are flukes.

The practice is to work backward from the difference that would change your decision. If a two-point improvement would make you ship, ask what suite size gives a good chance of detecting two points against your measured run-to-run noise, and if the answer is larger than you can afford, you have learned something crucial before wasting any compute: this comparison, at this budget, cannot be made cleanly, and you should either enlarge the suite, adopt a paired design that recovers power, or accept that you will decide on other grounds. Discovering a study was underpowered after running it is a common and demoralizing waste; the power calculation is cheap, and it is the difference between a suite chosen to answer your question and a suite chosen by whatever was convenient. A benchmark you could never have won is a benchmark you should not have run.

Matching the suite to the question

Every benchmarking decision implies a smallest difference you need to detect, and that difference should set the suite size rather than the other way around. If a one-point improvement would change what you ship, you need a suite and a run budget capable of resolving one point, which the square-root arithmetic says is thousands of items or a tightly paired design, and probably both. If only a five-point difference would change your decision, a few hundred items may suffice, and spending compute to resolve finer differences you will not act on is waste. Deciding the required resolution before running — the smallest difference that would change your mind — turns suite sizing from guesswork into arithmetic and inoculates you against the most common small-suite mistake, which is discovering after the fact that your suite could never have answered your question. The mantis measures the distance before it commits to the strike; the benchmarker measures the resolution before committing to the suite.