Three of Twenty Decoders Actually Stream. My Quality Metric Was Beaten by an Algorithm from 1984.

Twenty neural audio decoders, five pre-registered gates, identical audio. Only three reproduce full-context output from partial input with load-bearing state — fifteen are stateless chunking. The sole pre-registered cross-arm quality metric was topped by Griffin-Lim, which has zero trained parameters, in six states of six. A second measurement sharing no implementation found a near-zero median additional change for the same three arms when streamed. No listening test was run, and nothing here says which decoder sounds best.

If you are building a streaming text-to-speech system, you have to choose a decoder. The published record makes that harder than it should be: real-time factors quoted without hardware, "streaming" claimed for architectures that need the whole utterance, quality figures from different corpora at different sample rates through different measurement paths. None of it composes into a decision.

So I measured twenty of them under one frozen protocol on identical audio — representation to waveform only, because generator timing is a different question and a decode-only number read as end-to-end TTS performance is a failed report.

The reframing that changed the candidate set

The survey nearly excluded several candidates for the wrong reason. For a TTS build, only the decoder needs to be causal. The encoder runs at training time on complete utterances; at inference the generator emits tokens directly and no encoder sits in the serving path. NVIDIA's NanoCodec makes it concrete — non-causal encoder, causal HiFi-GAN decoder. "Is this codec causal?" is the wrong question.

Gate 2: "streaming" is a property almost nothing has

An arm counts as truly incremental only if both hold: stateful chunked decode reproduces full-context output to max|err| ≤ 1e-3, and state is load-bearing — the stateless error is at least 10× the stateful one. The second conjunct exists because the first can be satisfied by an arm that ignores state entirely.

The three causal configurations pass the state test with ratios of 3,530 to 36,740 against a threshold of 10. State is not marginally load-bearing; it is decisive.

The negative controls are what make the positives mean anything. The non-causal FocalCodec configurations are driven through the identical stateful code path. Their ratio is exactly 1 — carrying state changes nothing. Had a negative control passed, the experiment would have been void rather than the arm promoted.

Several of the fifteen never claimed to stream. The classification measures what they do; it is not a charge against what they advertised.

Gate 3: speed is not validity

First-audio latency spans 1.37 ms to 27.64 ms across the arms that have a first-audio number at all. MelFlow does not: it emits spectrogram frames, so producing playable audio from a partial stream needs an overlap-add stage this study records as not established, and its first-audio time is NOT ESTABLISHED. Its 375.91 ms is a steady-state median — a different quantity, and the two are never mixed. Real-time margin spans 64× down to 0.2×.

But two arms produce streamed output that fails the study's own validity gate at every chunk size tested. A third, focalcodec_12_5hz, fails at the 80 ms anchor but passes 2 of 5 tested sizes — its smallest viable configuration is 640.8 ms, not 80 ms.

The sharpest case: vocos_mel24 has the fastest first-audio in the study and no achievable streaming configuration at all. Its timing numbers are real; what they timed is not a valid streaming configuration.

One environment finding governs how any of these numbers may be read. A controlled probe — byte-identical tokens, same GPU, one arm run in every environment — found a 50.2% difference attributable purely to the transformers version, device-side, at identical torch. So the incumbent decoder may not be compared numerically against the other arms at all. That is not a statement about whether it is fast; it is a statement that the comparison does not exist.

Gate 4: the metric failed its own validity instrument

The pre-registration installed Griffin-Lim — a phase-retrieval algorithm from 1984, zero trained parameters — as a floor, and declared in advance what it would mean if it scored close to a trained neural codec: a finding about the metric, not about the decoders.

It did not score close. It scored best, in six states of six.

The obvious explanation was tested and refuted. The hypothesis that a low-level frame floor favours a magnitude-matching algorithm failed: splitting each cell at its own median energy, Griffin-Lim leads in both halves and by more in the loud half.

What remains is an explanation consistent with the result rather than a demonstrated mechanism, and the difference matters. Mel-cepstral distance as implemented here is derived from the magnitude spectrum and is blind to phase by construction. Griffin-Lim iterates toward consistency of the linear STFT magnitude and optimises nothing else, accepting whatever phase error that leaves. The two objectives are aligned in domain but are not the same quantity, and nothing in this study demonstrates that Griffin-Lim minimises the released cepstral distance exactly.

What is established does not depend on the mechanism: a metric whose ranking puts an untrained phase-retrieval algorithm first is not, on its own, a sufficient authority for cross-decoder quality. That is the pre-registered consequence, and it fired on the evidence rather than on the explanation.

Two limits travel with this. The instrument exists only in the mel route, so the same check is not established — not passed — in the codec route. And no new metric was introduced afterwards, because choosing a ruler once you know who won under the old one is precisely the failure the check exists to catch.

Q5: does the speaker survive, and does he stay one person?

A single similarity score conflates two different things, so this measures them separately: retention, how close a reconstruction sits to its own source, and dispersion, how tightly different reconstructions of the same speaker cluster together. A decoder can move a voice consistently — a stable new identity — or make the speaker wander. Those are not the same failure.

Both are read against references measured on source audio alone, before any reconstruction was opened: the human within-speaker ceiling and the between-speaker floor. Three speaker encoders, chosen mechanically from a previously frozen fourteen-encoder study, each calibrated independently. Cosine values are encoder-specific and never averaged.

Retention runs from 0.9952 down to 0.4316 (values quoted for ECAPA). The three encoders show broadly similar ordering, with some pairwise reversals — encodec24_q8 sits above dualcodec_12hz_v1 under ECAPA and below it under both ReDimNet encoders. Per-encoder spread reaches 0.1193, which is why a bare cosine without its encoder name is not a result, and why neighbouring arms should not be read as ranked against each other at all.

And here the two instruments agree on something uncomfortable: Griffin-Lim also has the highest speaker-embedding retention of the eighteen arms. That sentence is the whole claim — not that it is the best decoder, and not a perceptual claim. Two measurements, a spectral distance and three speaker encoders, both rank a phase-blind algorithm first. Both are blind to the same thing.

The streaming result, from a second direction

The identity measurement compares each arm against its own offline decode on the same recording, so no ceiling or floor enters and no cross-arm ranking is implied.

Two distinct quantities, and they are never quoted as one range: the offline ↔ streamed cosine, and the paired retention delta. All values below are ECAPA.

The three causal configurations show a near-zero median additional change under the tested encoders, supported recordings and imposed chunking regime. Their retention deltas are small but consistently negative. That is the whole claim — this study establishes no minimum detectable streaming change and no equivalence threshold, so "near-zero median" must not be read as "below what the instrument can detect". Nothing is claimed to be lost, and nothing is claimed to be preserved.

Thirteen others lose between −0.078 and −0.742 on the retention delta, an amount nowhere near the resolution limit. For focalcodec_25hz under one encoder the offline-versus-streamed cosine itself reaches −0.013: the streamed output is essentially orthogonal to that arm's own offline decode in this embedding space. That is a statement about speaker-embedding similarity only — it does not establish the absence of waveform or linguistic relationships, which were not measured.

This agrees with the quality gate's separate streaming detector — an aligned maximum-error detector on the waveform — which found the same three arms at 0.69–0.74% error where every other arm sat at 105–225%. Two measurements sharing no implementation, the same three survivors. They do share their input, the same reconstructions of the same corpus. The agreement is corroborating evidence: it makes an independent implementation error in both less likely. It cannot rule out an error in either — agreement between two implementations never can — and it says nothing about an artefact of the material both were computed on.

What the audit changed

Six rounds of independent adversarial review ran against this study — four before publication, one against the published package, and one against the correction itself. Three of the defects they found changed conclusions. They are in the paper because a methods section that hides them would misrepresent how the numbers were obtained.

• The cross-state reference used pairs the measurement forbade. Dispersion compared different sentences; its human ceiling did not, admitting the same sentence spoken in two emotions — in a parallel corpus, the most similar cross-state pair that exists. Eight labels changed when it was repaired.
• A ceiling measured on the wrong duration. Calibration used full-length recordings where the measurement compares 14-second prefixes. Counterfactual: 73 of 108 strata would have carried a different label, 67 of them false drift verdicts.
• A pair-grouping convention that was order-dependent. Permuting row order moved estimates by up to 0.089 — enough to flip a label. Replaced with a symmetric rule, verified invariant to machine precision.

The fifth round ran after publication and found the same pattern a fourth time, in the release itself: the scripts I advertised as the public reproduction commands resolved their data relative to where they sat in my private tree, and the package ships them flat. None of the three ran. The verifier I pointed readers at was the manifest of the private tree, which cannot verify a redacted package and reported 26 of 32 artifacts as drift.

Then a sixth round attacked the correction, before any of it went out, and found five more — including two claims in the repair that outran their evidence, a privacy scanner still listing the paths it knew about instead of declaring what counts as private, and a package verifier that printed its own integrity fingerprint without ever comparing it to anything.

The pattern is worth naming because it has now recurred in six disguises: verification that checks what is present rather than what is required. The estimator without its reference; the instrument without its evidence; the manifest without its inventory; a package that verified its own private origin rather than what it shipped; a scanner that listed roots instead of declaring a domain; and a verifier that printed a value instead of comparing it.

What this does not establish

• Perceptual quality was never measured. No listening test was run. Which decoder sounds best is not established and no sentence here implies it.
• The identity result is about speaker-encoder representations, not about how audio sounds and not about cloning quality.
• A large majority of cells fall outside the quality metric's validated support domain. Support is carried as a covariate, never as an exclusion.
• For one decoder family, identity dispersion and band-limiting cannot be separated in this design, so no causal claim is made about it.
• Seven speakers, one studio, one recording chain, English. Absolute values are not comparable to VoxCeleb-scale benchmarks.
• Thirteen arms were driven with chunked context they were never designed for. The result is the cost of streaming a decoder that was not built to stream.

What comes next

Human listening and adversarial detectability are declared later phases, neither started. The reconstructions are retained for exactly that reason — 35.65 GB kept rather than deleted, because a listening test is the only thing that can settle what two phase-blind metrics agreed to ignore.

The part that generalises

Two metrics ranked a zero-parameter algorithm first. Neither was wrong about what it measured; each was computed correctly. What neither establishes is perceptual quality — and the pre-registration that named that outcome in advance is the only reason it reads as a finding rather than an embarrassment.

If you are choosing a decoder on a published quality number, the question worth asking first is whether that metric has ever been shown a floor it should beat.

Paper: What a Streaming Decoder Costs: A Five-Gate Benchmark of Twenty Neural Audio Decoders, and a Metric That Failed Its Own Validity Check — Zenodo, September 2026, CC-BY-4.0.

DOI (v1.0.1, this version): 10.5281/zenodo.22811349 · all versions: 10.5281/zenodo.22798415 · superseded v1.0.0: 10.5281/zenodo.22798416

Release: v1.0.1 — the current corrected release.

Code, specifications, machine-readable results and the full audit trail: github.com/Mormolykos/decoder-vocoder-benchmark

Corrections since v1.0.0: CORRECTIONS_v1.0.1.md. Three commands regenerate every table and re-trace every headline claim from this package alone. That is a consistency check, not independent reconstruction — rebuilding the aggregates and their confidence intervals needs the withheld per-recording rows, and one published claim is marked WITHHELD in the public audit for exactly that reason.

The source audio, the derived speaker embeddings and the reconstructed audio are not released: the corpus is private, rights-cleared recordings of seven consented speakers, and an embedding of a named human is biometric data. Speaker names are pseudonymised throughout and every substitution is recorded with its source hash.