Fourteen Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.

Eight speakers each recorded the same 1,360 sentences in six phonation states, so the words are held constant and only the delivery changes. On one frozen list of 11,935 trials, equal error rate ran from 0.047 to 0.233 — produced by nothing but the choice of encoder. The encoder that came last is the one shipping inside a widely used open-source TTS system.

A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors is treated as an answer to "are these the same person." That number gets used for authentication, for voice-cloning pipelines, and increasingly as the yardstick in research papers that ask whether a synthetic voice preserved someone's identity.

The number is not neutral. It is a reading from an instrument, and I wanted to know how much the instrument itself was contributing.

So I ran fourteen of them over an identical, frozen list of 11,935 trials and compared what they said about the same audio.

The corpus is the reason the question is answerable

Most emotional-speech corpora use different sentences for different emotions. That is a serious problem for this question: a model can learn angry vocabulary instead of angry delivery, and you cannot tell the two apart afterwards.

This corpus cannot make that mistake. Eight speakers each recorded the same 1,360 sentences in six phonation states — neutral, happy, angry, scared, shouting, whisper. Lexical content is held fixed while phonation varies, which separates what is said from how it is said.

Enrolment is neutral speech. The test is whether the encoder still recognises the same person while they shout, whisper, or are frightened.

The trial list was frozen in writing before a single model was loaded, and every encoder scored the identical list. That matters more than it sounds: it makes every comparison paired, and a paired comparison is a much sharper instrument than comparing two marginal confidence intervals.

The headline: a five-fold spread on identical audio

Equal error rate spans 0.047 to 0.233 across the panel — same trials, same speakers, same recordings. Nothing varies but the encoder. 29 of 91 pairwise comparisons survive Holm–Bonferroni correction across the full pairwise family.

If you are choosing an encoder for expressive or emotional audio, that spread is your decision. It is larger than most of the architectural differences people argue about.

The most robust finding: CAMPPlus is last, without exception

CAMPPlus has the highest error of the fourteen, and all 13 of its comparisons survive family-wise correction. It is highest for every one of the eight speakers and every one of the six conditions. There is no pooling artifact here — it does not win a single cell.

Margins run from +0.087 [+0.064, +0.127] against ECAPA-TDNN to +0.185 [+0.150, +0.215] against the lowest-error encoder in the panel.

The mechanism is the useful part

CAMPPlus's genuine scores are unremarkable. The problem is its impostor scores.

It places different people at cosine 0.28–0.34, where ECAPA-TDNN and ReDimNet place the same pairs near 0.034 on the same audio. That is a compressed embedding space: different speakers land close together, so the threshold separating "same person" from "different person" has almost nowhere to sit.

And it is visible at neutral speech — 0.031 EER where several ReDimNet checkpoints reach 0.000. So expressive speech does not cause the problem. Expressive speech makes it visible.

This matters practically because a widely used open-source TTS system conditions on CAMPPlus. I want to be precise about what that does and does not mean, because it is the easiest sentence in this article to over-read. It is a verified fact about that system's source code. It is not a measurement of its audio. I measured encoders on human recordings, in isolation. Whether that choice degrades the system's output is a separate experiment, and I have not run it.

Parameter count does not order the panel

Spearman ρ between parameter count and expressive EER is +0.135 over fourteen encoders. No ordering, and if anything the wrong sign.

The 20.8 M model ranks 13th of 14. A 4.81 M model ranks 1st. Published VoxCeleb1-O position does not transfer either — the leaderboard you would pick from is computed on calm read speech, and that is not the condition failing you.

I have to be careful about the winner, and this is one of the claims I retracted. The 4.81 M model survives as an argmin — selected in 89.4% of speaker resamples, and leading in all eight leave-one-speaker-out refits — but after correction it is separated from only 7 of 13. The defensible wording is lowest observed error, not "beats the others."

A clean pre-registered negative

The obvious objection is that preprocessing produced this. It did not. Spectral denoising does not materially change the measurement: |ΔEER| ≤ 0.039 over 10,765 twinned trials, and whispered speech — the condition where removing aspiration noise was the stated worry — is among the least affected.

Ten retracted claims, and why they are in the paper

The published record ships docs/DECISIONS.md rather than a tidied summary, because several of the things I had to withdraw were the headline at the time.

• An 11.7 dB spectral effect, asserted with an instrument whose F0 artifact budget is 12.09 dB. Sweeping fundamental frequency with the spectral envelope held exactly fixed moved the descriptor further than the effect I was claiming. It was never a measurement. The fix, gate_f0.py, now runs before analysis and refuses a descriptor until its artifact budget is known — and it proved its own sensitivity by rejecting two of the replacement measures I wrote to satisfy it.
• "Shouting does not transfer." Judged on RMS after the analyser had peak-normalised every clip, so the figure was crest factor, not loudness. I caught that by ear before the measurement caught it.
• A tie-break defect in the EER estimator. argmin returns the first minimiser — the lower threshold rather than the balanced operating point. Invisible at unit weights, and common under the integer speaker multiplicities a bootstrap actually runs at. Fixing it moved one comparison across the Holm threshold, 28 → 29 of 91. The correction favoured the paper, which is exactly why it is published rather than quietly kept: a report about what happens when an instrument is wrong cannot itself carry a number produced by a wrong instrument.

The rule that came out of this, and I have now paid for it three separate times: validate the instrument before measuring with it.

The statistical procedure

The trial list is a directed graph on 8 speakers: genuine trials are the self-loops, impostor trials the 56 ordered edges. The independent unit is the speaker, who sits on both sides of the comparison. So the resample is a vertex bootstrap — draw the 8 vertices with replacement, take the induced sub-multigraph, weight the impostor edge by the product of the two multiplicities. A speaker who is not drawn contributes to neither side.

An earlier scheme resampled enrolment speakers only, which dropped a speaker as an enrollee while leaving them in as an impostor. Correcting it widened the marginal intervals (width ratio median 1.11, max 1.58) and cost two already-written claims. Paired differences were almost unaffected — both encoders in a pair score the same shared impostor clips, so the dependence cancels in the difference. That is a second, independent argument for the paired design.

B = 20,000 draws, seed fixed, families declared before testing, Holm–Bonferroni step-down at family-wise α = 0.05. The bootstrap p-value floor at that B is 1×10⁻⁴, below Holm's strictest threshold here (5.5×10⁻⁴), so no comparison is limited by resolution.

The companion experiment: enrolment composition

A second pre-registered experiment asks whether enrolling a speaker on expressive material, instead of neutral-only, recovers the loss. It does, on all 14 encoders, every one surviving Holm and clearing its own null band.

The mechanism is the opposite of the intuitive one. Enrolling on a single expressive state helps that state and slightly harms the others — transfer-matrix diagonal median −0.082, off-diagonal median +0.010. Guessing which state a user will be in, and guessing wrong, is worse than enrolling on calm speech.

A mixed set carrying a third as much of each state improves all five conditions, because it holds matched material for each. A pre-committed control ruled out the competing explanation: acoustic diversity alone delivers +0.002 where composition delivers −0.065, against a bar of 0.50 recovery share. It is coverage, not expressiveness.

If you run voice authentication, that is the cheapest thing in this article: spread enrolment thinly across states rather than guessing one.

What this paper does not claim

These are in the abstract rather than buried, deliberately.

• Nothing about any TTS system's output quality. Encoders were measured in isolation on human recordings.
• Eight speakers, one studio, one recording chain.
• Enrolment and test share a recording session, so absolute error rates are optimistic and are not comparable to VoxCeleb figures. Only relative comparisons between encoders on identical trials are licensed.
• Architecture is fully confounded with training corpus. No released checkpoint crosses the two families, so "the larger-speaker-count corpus helps" and "that architecture family is better" are equally consistent with every number here. Separating them requires training, which I did not do.

Reproducibility

The audio is a private, rights-cleared corpus and is not released. Everything downstream of it is — score tables, trial manifest, analysis code, decision record.

Verified in a clean virtual environment with numpy, spkproof and matplotlib only — no GPU, no torch, no audio: the results files regenerate line for line and all four figures rebuild. The analysis opens by checking its own weighted EER estimator against spkproof.panel.equal_error_rate and agreeing to 0.00e+00 across all fourteen encoders.

The repository also ships the redaction script that produced the release, because "anonymised" is a claim a reader should be able to audit rather than trust. Corpus paths were replaced rather than deleted — each filename embeds the sentence spoken, so publishing 11,935 paths would publish a private 1,360-phrase script — by an opaque clip identifier proven 1:1 across all 10,964 recordings, so the enrolment-leakage audit still runs and still reports zero.

The part that generalises

An established metric is not automatically a neutral one. An encoder, an evaluator, a classifier, or an LLM judge can become part of the experiment rather than a window onto it.

Before asking whether a system preserves speaker identity under some transformation, it is worth asking whether the instrument you are measuring with has been validated under that same transformation. Otherwise you will attribute the encoder's behaviour to the speaker — and the number will look perfectly reasonable while you do it.

Paper: Speaker Encoders Disagree About Who You Are When You Shout: A Content-Matched Benchmark of Fourteen Speaker Encoders Under Expressive Phonation — Zenodo, 29 August 2026, CC-BY-4.0.

DOI: 10.5281/zenodo.22158030 (concept DOI — always resolves to the newest version).

Code, data and the full decision record: github.com/Mormolykos/speaker-encoders-disagree — MIT for code, CC-BY-4.0 for the data tables.

The evaluation library the analysis checks itself against is spkproof (pip install spkproof), which ships the trial-design checks and the descriptor gate described above.