The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.

IBM's Benchpress pins the seeds that build its circuits and passes no seed to the code that compiles them. On one circuit, across 400 seeds per version, the real change between two Qiskit releases is +5.37%. The suite's own three-run protocol reports it as a ≥10% regression 24.4% of the time. Twenty runs per version — on the order of 80 compute hours — still leaves 3.74%.

A benchmark suite is an instrument. You point it at two versions of a compiler and it tells you whether the new one got worse. Somebody then merges or reverts a pull request on the strength of that answer.

So the question I care about is not "is this compiler slower". It is how often does the instrument give an answer that disagrees with its own long-run behaviour. I call that finite-sample decision risk: the probability that a verdict computed from k runs per version disagrees with the verdict implied by the mean of the same measurement.

Where the randomness comes from

Qiskit's transpiler is stochastic. Its routing pass uses randomness, and generate_preset_pass_manager takes a seed_transpiler argument to pin it.

At the pinned revision I measured, Benchpress's Qiskit gym does not pass it. Circuit construction seeds are fixed — seed=12345, fourteen occurrences. The compilation seed is not. So every gate count the suite reports is one draw from a distribution nobody measured.

One detail matters for honesty here, and I got it wrong in an earlier draft: this is not universal across the suite. The BQSKit gym in the same repository does seed its compiler, with seed=0. seed_transpiler is a Qiskit-specific API that the other gyms cannot call. The claim is about the Qiskit gym, and only about it.

What one circuit does

bv_n140 is a Bernstein–Vazirani circuit that Qiskit issue #14402 names directly. Mapped to a heavy-hex lattice, going from Qiskit 1.4.3 to 2.0.0, measured at 400 seeds per version across 21 operating-system processes. This is a separate measurement from the pre-registered experiment below, and bv_n140 is not part of the pre-registered endpoint:

The true change is nowhere near the threshold. The verdict crosses it a quarter of the time anyway.

I did not trust that number when I first got it, because all 200 of the original seeds came from one contiguous block inside one process — so any per-process state, hash randomisation, allocator layout, a cached RNG, was held constant. I drew a disjoint set of 200 seeds spanning 5.9 million to 1.08 billion, ran them across ten fresh processes with differing PYTHONHASHSEED, and got 22.6% [16.4, 30.6] against 26.3% [18.5, 34.2] from the contiguous set. Each point falls inside the other's interval.

Running it more does not fix it

The obvious response is "then run it more times". I measured exactly how much that buys, pooling all 400 seeds:

Twenty runs per version is 20 × 2 versions × roughly two hours per suite run — the issue's own figure, which I could not independently confirm — so on the order of 80 compute hours. It still leaves 3.74%. An earlier version of this page said forty hours, which halves the arithmetic: forty hours is wall-clock on two machines running concurrently, not the compute total.

Fixing seed_transpiler removes the variance attributable to the transpiler seed at a single run. It does not make one run correct, and an earlier version of this page came close to saying it did. A fixed seed makes the answer repeatable, including when that answer is wrong: on bv_n280, recorded seed 663193 gives A = 1040 against B = 1157 — a +11.25% call against a measured θ of +4.89%. Frozen, reproducible, and the wrong side of the threshold.

Taking the minimum of three runs instead of the mean — a common instinct — makes it worse, not better. All eight circuits I tested still err, and every rate roughly doubles: bv_n280 goes from 17.3% to 28.4%.

The number in the issue

Issue #14402 reports +46.1% for that circuit. A single three-run comparison of the same circuit, under the same protocol, returns anywhere from −10.5% to +100.0%, with a 95% range of [+9.3%, +57.2%].

+46.1% sits at the 88th percentile of that distribution. The same range extends below the +10% threshold. The reported figure is not wrong — it is one draw, reported as though it were a measurement.

The part I pre-registered

One circuit is an anecdote. So I wrote down the selection rule, the analysis, the endpoint and the labels, and committed them before the first tracked commit of the 39-circuit raw dataset. The commit ordering is checkable:

a36a34a  2026-09-03 23:51:08  pre-registration fbc573d  2026-09-03 23:52:38  analysis code 30224a0  2026-09-04 09:13:07  raw data

What that ordering supports is a frozen follow-up design after exploration, not blindness to earlier outcomes — and the difference matters enough that the paper corrects it twice. A census and exploratory results preceded the pre-registration: an earlier commit the same day already identifies heavy-hex as the worst case, and the ≤10 s selection rule is computed from that pre-existing census. Git records when files were committed. It does not record when their outputs were first looked at, and I should not have written as though it did.

The rule selected every qasmbench-large circuit whose twelve-seed heavy-hex runtime in that census was under ten seconds — 39 circuits, 200 recorded seeds per arm, 15,600 primary transpilations.

The pre-registered endpoint: 12 of 26 eligible circuits have a decision-error rate whose interval excludes zero. That is 46.2%, Wilson 95% [28.8%, 64.5%]. Seven of 26 sit at 5% or worse, four at 10% or worse.

What I withdrew

That 46.2% is the number I trust least in this work, and I want to say why rather than let someone find it.

The 26 circuits are not 26 independent observations. They come from eleven algorithm families, and within a family the results are nearly all-hit or all-miss. A cluster bootstrap over families gives [17.4%, 81.0%] — against the Wilson interval's [28.8%, 64.5%]. The effective sample size is far below 26, and no interval I can compute is informative about a population of circuits.

So the suite-level rate is withdrawn as a headline. The 12/26 stands as a descriptive count of these 26 circuits and nothing more.

Three other claims went the same way, and all four withdrawals are still in the repository with the evidence that defeated them. One of them was mine: I had reported a correlation of ρ = +0.876 between "compilation is stochastic" and risk, presented as a discovered mechanism. Circuits whose two arms are constant have zero risk by arithmetic, so a correlation computed over a sample containing them is measuring a definition, not a finding. Simulated data containing no compiler at all reproduces the statistic.

What went wrong in my own instrument

The paper's whole argument is that measurement apparatus should be checked rather than trusted, so it would be poor form not to turn that on myself.

After three rounds of adversarial review and a sixteen-phase red team — all of which attacked the claims — I had an independent audit read the ~4,600 lines of Python that nobody had ever read. It found six defects. None changed a published number. The worst one was this:

My numeric inventory recomputed every reported figure and compared it to a recorded value. But the recorded value was itself a function call evaluated when the registry was built, and the recomputation called the same function on the same file. It compared f() with f(). It could not fail. I proved it by falsifying every value in a summary file — tripling them and adding 40 percentage points — and the checker still reported "41/41 reproduce".

The values are frozen literals now, twenty-nine of forty-one re-derive from raw measurements rather than from a summary, and re-running that same falsification produces twelve failures. A checker that cannot fail is worse than no checker, because it buys confidence it has not earned.

That was the first audit of the apparatus, and it was not the last. The paper's own account of the review of its claims is four rounds of hostile review plus two first-principles audits, which removed five claims from the work. Every audit round after that attacked the apparatus instead, and the pattern held every time: each one ended with a green verifier that the next adversarial reader then broke. Not one of them changed a scientific number. What they broke was the checking — a keyword that bought an exemption, a substring that satisfied a presence test, a claim scan that could not see Seven of the twenty-six because that is not how it expected a fraction to be spelled. The verifier now runs thirteen stages and a suite of forty-four deliberate corruptions, each of which must be rejected by a named stage for a named reason. That is evidence it can turn red. It is not a proof that nothing else gets through, and the paper says so.

What this does and does not say

It does not say Qiskit's transpiler got worse, or that Benchpress is a bad suite. It measures one property of one benchmarking protocol: that an unseeded stochastic compiler, sampled three times, produces regression verdicts that disagree with its own long-run behaviour at rates that are not small.

Limits, stated plainly: one SDK, one version pair, one machine. The +10% threshold is mine — Benchpress defines none and the issue states no formal cut. The estimate of the true change is a plug-in from 200 seeds, not an external criterion. And seed_transpiler is not a universal remedy: it removes false positives on the circuits I tested, but was worse on three of the four circuits showing false negatives.

Everything is in the record

The paper, the complete analysis code, and the 15,600 pre-registered per-seed transpilations — 39 circuits, 200 seeds, two arms — are archived with a DOI, along with the exploratory, replication and cross-machine runs. verify.py re-runs thirteen read-only stages: the toolchain pin, the test suite, the numeric inventory, the replication artifact, a tautology proof, the paper's claims, the endpoint rebuilt from raw, raw integrity, the mutation suite, the evidence anchor, the saved-results binding, the rendered manuscript and the published PDF. It takes tens of minutes, not the minute an earlier version of this page claimed.

Version 4 is the audited release, and it is the one to cite:

• Version 4 record — 10.5281/zenodo.22848938
• All versions (concept DOI) — 10.5281/zenodo.22310059
• Tagged release — github.com/Mormolykos/qvalidation @ v4.0.0
• Repository — github.com/Mormolykos/qvalidation

The reproducibility archive is the exact frozen source tree at commit e663912; every file in it was hashed against its Git object before release.

If you maintain a benchmark that drives accept/reject decisions on a stochastic system, the cheap version of this check is: run the same comparison twenty times without changing anything, and look at the spread of verdicts rather than the spread of values. If the verdict moves, the number of runs is part of your instrument, and it belongs in the write-up.