Three of Twenty Decoders Actually Stream. My Quality Metric Was Beaten by an Algorithm from 1984.

Twenty neural audio decoders, five pre-registered gates, identical audio. Only three reproduce full-context output from partial input with load-bearing state — fifteen are stateless chunking. The sole pre-registered cross-arm quality metric was topped by Griffin-Lim, which has zero trained parameters, in six states of six. A second measurement sharing no implementation found the same three arms lose nothing to streaming. No listening test was run, and nothing here says which decoder sounds best.

If you are building a streaming text-to-speech system, you have to choose a decoder. The published record makes that harder than it should be: real-time factors quoted without hardware, "streaming" claimed for architectures that need the whole utterance, quality figures from different corpora at different sample rates through different measurement paths. None of it composes into a decision.

So I measured twenty of them under one frozen protocol on identical audio — representation to waveform only, because generator timing is a different question and a decode-only number read as end-to-end TTS performance is a failed report.

The reframing that changed the candidate set

The survey nearly excluded several candidates for the wrong reason. For a TTS build, only the decoder needs to be causal. The encoder runs at training time on complete utterances; at inference the generator emits tokens directly and no encoder sits in the serving path. NVIDIA's NanoCodec makes it concrete — non-causal encoder, causal HiFi-GAN decoder. "Is this codec causal?" is the wrong question.

Gate 2: "streaming" is a property almost nothing has

An arm counts as truly incremental only if both hold: stateful chunked decode reproduces full-context output to max|err| ≤ 1e-3, and state is load-bearing — the stateless error is at least 10× the stateful one. The second conjunct exists because the first can be satisfied by an arm that ignores state entirely.

The three causal configurations pass the state test with ratios of 3,530 to 36,740 against a threshold of 10. State is not marginally load-bearing; it is decisive.

The negative controls are what make the positives mean anything. The non-causal FocalCodec configurations are driven through the identical stateful code path. Their ratio is exactly 1 — carrying state changes nothing. Had a negative control passed, the experiment would have been void rather than the arm promoted.

Several of the fifteen never claimed to stream. The classification measures what they do; it is not a charge against what they advertised.

Gate 3: speed is not validity

First-audio latency spans 1.37 ms to 375.91 ms steady-state, and real-time margin spans 64× down to 0.2×. But three arms produce streamed output that fails the study's own validity gate at every chunk size tested.

The sharpest case: vocos_mel24 has the fastest first-audio in the study and no achievable streaming configuration at all. Its timing numbers are real; what they timed is not a valid streaming configuration.

One environment finding governs how any of these numbers may be read. A controlled probe — byte-identical tokens, same GPU, one arm run in every environment — found a 50.2% difference attributable purely to the transformers version, device-side, at identical torch. So the incumbent decoder may not be compared numerically against the other arms at all. That is not a statement about whether it is fast; it is a statement that the comparison does not exist.

Gate 4: the metric failed its own validity instrument

The pre-registration installed Griffin-Lim — a phase-retrieval algorithm from 1984, zero trained parameters — as a floor, and declared in advance what it would mean if it scored close to a trained neural codec: a finding about the metric, not about the decoders.

It did not score close. It scored best, in six states of six.

The obvious explanation was tested and refuted. The hypothesis that a low-level frame floor favours a magnitude-matching algorithm failed: splitting each cell at its own median energy, Griffin-Lim leads in both halves and by more in the loud half.

What remains is structural and was foreseeable. Mel-cepstral distance is a distance on the mel magnitude spectrum; Griffin-Lim iteratively minimises exactly that quantity and has no other objective, while carrying phase error the metric cannot see. A metric cannot rank decoders on a quantity one of them is an optimiser for.

Two limits travel with this. The instrument exists only in the mel route, so the same check is not established — not passed — in the codec route. And no new metric was introduced afterwards, because choosing a ruler once you know who won under the old one is precisely the failure the check exists to catch.

Q5: does the speaker survive, and does he stay one person?

A single similarity score conflates two different things, so this measures them separately: retention, how close a reconstruction sits to its own source, and dispersion, how tightly different reconstructions of the same speaker cluster together. A decoder can move a voice consistently — a stable new identity — or make the speaker wander. Those are not the same failure.

Both are read against references measured on source audio alone, before any reconstruction was opened: the human within-speaker ceiling and the between-speaker floor. Three speaker encoders, chosen mechanically from a previously frozen fourteen-encoder study, each calibrated independently. Cosine values are encoder-specific and never averaged.

Retention orders the arms consistently across all three encoders, from 0.9952 down to 0.4316 (values quoted for ECAPA; per-encoder spread reaches 0.1193, which is why a bare cosine without its encoder name is not a result).

And here the two instruments agree on something uncomfortable: Griffin-Lim also has the highest speaker-embedding retention of the eighteen arms. That sentence is the whole claim — not that it is the best decoder, and not a perceptual claim. Two measurements, a spectral distance and three speaker encoders, both rank a phase-blind algorithm first. Both are blind to the same thing.

The streaming result, from a second direction

The identity measurement compares each arm against its own offline decode on the same recording, so no ceiling or floor enters and no cross-arm ranking is implied.

The three causal configurations lose nothing. Thirteen others lose between −0.078 and −0.742, and for one arm the offline-versus-streamed similarity reaches −0.013 — the streamed output bears no relation to that arm's own offline decode.

This reproduces the quality gate's separate streaming detector, which found the same three arms at 0.69–0.74% error where every other arm sat at 105–225%. Two measurements sharing no implementation, the same three survivors. They do share their input — the same reconstructions of the same corpus — so the agreement rules out an implementation error in either, not an artefact of the material both were computed on.

What the audit changed

Four rounds of independent adversarial review ran against this study, and three of the defects they found changed conclusions. They are in the paper because a methods section that hides them would misrepresent how the numbers were obtained.

• The cross-state reference used pairs the measurement forbade. Dispersion compared different sentences; its human ceiling did not, admitting the same sentence spoken in two emotions — in a parallel corpus, the most similar cross-state pair that exists. Eight labels changed when it was repaired.
• A ceiling measured on the wrong duration. Calibration used full-length recordings where the measurement compares 14-second prefixes. Counterfactual: 73 of 108 strata would have carried a different label, 67 of them false drift verdicts.
• A pair-grouping convention that was order-dependent. Permuting row order moved estimates by up to 0.089 — enough to flip a label. Replaced with a symmetric rule, verified invariant to machine precision.

The pattern is worth naming because it recurred in three disguises: verification that checks what is present rather than what is required. The estimator without its reference; the instrument without its evidence; the manifest without its inventory.

What this does not establish

• Perceptual quality was never measured. No listening test was run. Which decoder sounds best is not established and no sentence here implies it.
• The identity result is about speaker-encoder representations, not about how audio sounds and not about cloning quality.
• A large majority of cells fall outside the quality metric's validated support domain. Support is carried as a covariate, never as an exclusion.
• For one decoder family, identity dispersion and band-limiting cannot be separated in this design, so no causal claim is made about it.
• Seven speakers, one studio, one recording chain, English. Absolute values are not comparable to VoxCeleb-scale benchmarks.
• Thirteen arms were driven with chunked context they were never designed for. The result is the cost of streaming a decoder that was not built to stream.

What comes next

Human listening and adversarial detectability are declared later phases, neither started. The reconstructions are retained for exactly that reason — 35.65 GB kept rather than deleted, because a listening test is the only thing that can settle what two phase-blind metrics agreed to ignore.

The part that generalises

Two metrics ranked a zero-parameter algorithm first. Neither was wrong about what it measured; each was computed correctly. What neither establishes is perceptual quality — and the pre-registration that named that outcome in advance is the only reason it reads as a finding rather than an embarrassment.

If you are choosing a decoder on a published quality number, the question worth asking first is whether that metric has ever been shown a floor it should beat.

Paper: What a Streaming Decoder Costs: A Five-Gate Benchmark of Twenty Neural Audio Decoders, and a Metric That Failed Its Own Validity Check — Zenodo, September 2026, CC-BY-4.0.

DOI: 10.5281/zenodo.22798416

Code, specifications, machine-readable results and the full audit trail: github.com/Mormolykos/decoder-vocoder-benchmark

The source audio, the derived speaker embeddings and the reconstructed audio are not released: the corpus is private, rights-cleared recordings of seven consented speakers, and an embedding of a named human is biometric data. Speaker names are pseudonymised throughout and every substitution is recorded with its source hash.