A leaderboard orders systems by a point estimate. That ordering is a convenience, not a result. The question worth asking of any published gap is narrower and more answerable: if we resampled the evaluation set, how often would this gap keep its sign, and how wide is the interval around it? This article sets out a paired bootstrap protocol for answering that question, states what the resulting interval does and does not cover, and gives a reporting template that makes the answer reproducible.
Why the rank is not the report
Rodriguez et al. note that leaderboards are a straightforward ranking of models, and that this simplicity can mask nuances in evaluation items and in the models being compared (ACL 2021). Bestgen makes the complementary point: simple ranking has been criticized because one system’s superiority over another may result from chance alone rather than from a real difference, and he argues for the widest possible use of bootstrap confidence intervals for comparing system performance instead of state-of-the-art status and significance testing (LREC 2022).
Both claims point at the same reporting failure. A rank collapses three quantities into one integer: the observed difference, the uncertainty around it, and the direction of the comparison. The bootstrap interval keeps all three visible.
Define the estimand before resampling anything
The quantity to estimate is the paired per-example difference in a chosen metric between two systems evaluated on the same rows. Write di for the difference on row i, and the observed gap as the mean of di over the N evaluation rows. The estimand is the mean of that difference over the population the evaluation set is meant to represent.
Two commitments follow from this definition, and both are easy to get wrong.
The resampling unit is the evaluation row, not the token. A benchmark score is an estimate, not a property of a model, and the unit of evidence is usually the task, conversation, or user query rather than the token (hypothesis-testing notes). Resampling tokens when the claim concerns tasks produces intervals that are too narrow, because tokens within a row are not independent draws from the population of tasks.
The comparison is paired. Both systems answer the same rows. Most rows are easy or hard for both, so the per-row differences carry less noise than two independently estimated accuracies. An unpaired interval throws that structure away and overstates uncertainty; a paired interval uses it.
The paired bootstrap, step by step
The procedure below follows the description in Bestgen (2022).
- Compute the observed difference in performance between the two systems on the whole evaluation set. This is the best available estimate of the true difference.
- Repeat the following three steps a large number of times:
- Draw a bootstrap sample by sampling N rows from the evaluation set with replacement.
- Compute each system’s performance on that bootstrap sample.
- Compute the difference between the two performances.
- Use the resulting bootstrap distribution of the difference to form the interval.
Bestgen reports using the bias-corrected and accelerated (BCa) interval with 10,000 samples in his case studies, and notes that 1,000 to 5,000 bootstrap samples are generally considered enough for a 95% interval. The percentile method is simpler; BCa is the safer default when the bootstrap distribution is skewed or the evaluation set is small.
A minimal implementation, with the seed and draw count recorded as parameters rather than defaults buried in the code:
import numpy as np
def paired_bootstrap_delta(a_correct, b_correct, draws=10000, level=0.95, seed=0):
"""Percentile bootstrap interval for metric(B) - metric(A) on paired rows."""
delta = np.asarray(b_correct, dtype=np.float64) - np.asarray(a_correct, dtype=np.float64)
rng = np.random.default_rng(seed)
n = len(delta)
estimates = np.empty(draws, dtype=np.float64)
for k in range(draws):
idx = rng.integers(0, n, size=n)
estimates[k] = delta[idx].mean()
alpha = (1.0 - level) / 2.0
lo, hi = np.quantile(estimates, [alpha, 1.0 - alpha])
return delta.mean(), lo, hi
The function assumes binary correctness per row. For a metric that decomposes over rows, such as F1 or BLEU, record per-row sufficient statistics and resample those alongside the rows; the bootstrap applies to any performance metric, which is part of its appeal for NLP comparisons.
A worked example, clearly labeled as hypothetical
The following numbers are constructed to show how the same observed gap reads differently under different interval widths. They are not results from any model, dataset, or run.
Suppose two systems are evaluated on the same 400 rows and the observed accuracy gap is 1.5 points in favor of system B. Three reporting scenarios:
| Scenario | Observed gap | 95% interval | Reading |
|---|---|---|---|
| Wide interval | +1.5 pts | [−0.8, +3.8] | The sign of the gap is not stable under resampling of these rows. |
| Narrow interval, near zero | +1.5 pts | [+0.2, +2.8] | The sign is stable, but the lower bound is close enough to zero that the practical size of the advantage is unresolved. |
| Narrow interval, away from zero | +1.5 pts | [+1.1, +1.9] | The sign is stable and the interval is tight around a small difference. Whether 1.5 points matters is a separate question the interval cannot answer. |
The first and third rows are different failures. An interval that includes zero says the evaluation set does not pin down the direction of the difference. An interval that excludes zero but sits tightly around a trivial difference says the direction is pinned down and the magnitude is small. Collapsing both into “significant” or “not significant” hides which situation the reader is in.
What the interval covers, and what it does not
The bootstrap conditions on the observed evaluation rows. It quantifies how much the score would move if the finite set were perturbed, under the assumption that the observed rows are representative enough that drawing from them mimics drawing from the population (hypothesis-testing notes). That assumption is doing real work, and it fails in identifiable ways.
The interval does not correct for benchmark contamination, label errors, or distribution shift. It cannot fix a benchmark whose rows all test the wrong skill. Those are separate audits, and a narrow interval on a contaminated benchmark is a precise estimate of the wrong quantity. Berg-Kirkpatrick et al. found that significance levels computed on a test set were reasonably well calibrated when new data was drawn from the same distribution, but that the predictive ability of the significance level dropped off as the domain of the new data diverged from the test set (EMNLP-CoNLL 2012). An interval on a leaderboard gap inherits the same limitation: it is a statement about the sampled evaluation set and the population it represents, not about every input the system will see.
Dror et al. add a related caution: most significance tests assume independent observations, while many NLP evaluation sets consist of sentences from the same document, source, or author, and the nature of those dependencies is hard to characterize (ACL 2018). When rows are clustered, resampling rows understates uncertainty. The resampling unit should match the unit of generalization the claim is about.
A reporting template
A gap that survives resampling is only reproducible if the record is complete. The following fields are the minimum for a comparison to be re-run and checked:
- Model and version for each system, with the exact checkpoint identifier.
- Seed and decoding parameters (temperature, top-p, max tokens, stop conditions) for each run.
- Evaluation set: the exact rows, a hash of the row set, and the pairing structure (which rows each system answered).
- Metric definition, including how ties, refusals, and unparseable outputs are scored.
- Bootstrap configuration: number of draws, interval method (percentile or BCa), confidence level, and the seed used for resampling.
- Paired or unpaired, stated explicitly.
- Observed gap and both interval bounds, reported together.
The evaluation-set hash and the pairing structure are not optional bookkeeping. The interval is only reproducible if the rows and their pairing are fixed; a hash makes silent row changes visible, and the pairing structure determines whether the paired bootstrap is even the right procedure.
Interpretation discipline
Three rules keep the report honest.
An interval that includes zero is not evidence of no difference. It says the evaluation set does not resolve the direction. A larger or differently sampled evaluation set might.
An interval that excludes zero is not evidence of importance. Bestgen makes this point directly: a statistical test dichotomizes results into significant versus not significant, while statistically significant does not mean important, and with enough data any difference, however small, can be declared significant. The interval’s job is to show the size of the difference alongside its uncertainty, not to replace one binary with another.
Keep the observed difference and the bounds side by side. The observed gap is the best point estimate; the bounds say how much it could move. Reporting only the bounds, or only the gap, loses information the reader needs.
What would change the conclusion
A gap’s survival under resampling is conditional on three choices, and changing any of them can change the answer.
A different evaluation set. The interval is computed on the rows at hand. A set drawn from a different domain, or with a different difficulty distribution, can move both the observed gap and its width. Berg-Kirkpatrick et al.’s domain-shift result is the relevant warning here.
A different metric. Accuracy, F1, and BLEU weight rows differently. A gap that is stable under one metric may not be stable under another, and the metric definition should be reported precisely enough that a reader can recompute it.
A different pairing structure. If the two systems did not answer the same rows, the paired bootstrap does not apply. If rows are clustered by document or conversation, the resampling unit should be the cluster, not the row.
For readers who want to check a published gap against these rules, the useful next step is to obtain per-example predictions, fix the row set and its hash, and run the paired bootstrap with the parameters recorded. A gap that survives that procedure is a stronger claim than a rank. A gap that does not is a claim that should not have been made in the first place.
Frequently asked questions
How many bootstrap draws are enough?
Bestgen cites 1,000 to 5,000 samples as generally sufficient for a 95% interval, and reports using 10,000 in his case studies. More draws reduce Monte Carlo noise in the interval endpoints; they do not fix a biased evaluation set.
Should I use the percentile method or BCa?
BCa is the safer default when the bootstrap distribution is skewed or the evaluation set is small. The percentile method is simpler to implement and adequate when the distribution is roughly symmetric and the set is large. Report which one you used.
Can I compare more than two systems at once?
Yes, but the intervals are pairwise and the comparisons multiply. Bestgen notes that confidence intervals are interpreted as uncertainty bounds on an estimate rather than as significance tests, so a Bonferroni-type correction is not automatically required; the reader should still be told how many comparisons were made.
Does a narrow interval mean the benchmark is good?
No. A narrow interval means the score is precisely estimated on the rows at hand. It says nothing about whether those rows test the skill the claim is about. Contamination, label error, and distribution shift are separate audits that the bootstrap cannot perform.
