Your TTS Model Sounds Great — Until It Says "GPUB"

ttsproof: automated failure-mode QA for text-to-speech, backed by a published 390-sample study.

I built a text-to-speech product, and I kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym, or a name — and quietly mangle it. Worse, the standard metric everyone reaches for, Word Error Rate, was lying to me in both directions: it flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em," and it missed real failures on short tokens where the speech recognizer was as unreliable as the TTS.

So I wrote the QA framework I wished I'd had, packaged it as ttsproof, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.

The two failures WER can't see

A TTS pipeline breaks in two different ways, and a single WER number blurs both:

1. Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click/pop at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, from the waveform.
2. Pronunciation / content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names.

ttsproof splits them apart and handles each honestly:

• Structural checks (no model needed) — empty/truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. Just numpy + soundfile.
• Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalized to spoken form (numbers, decimals, dates, clock times, acronyms, letters) before scoring, so 3:30 PM vs "three thirty" stops counting as an error.
• ASR-uncertainty quarantine — when the audio is structurally clean but the recognizer disagrees on a very short utterance (a letter, an acronym), the sample is set aside for a human instead of being auto-failed — because at that length, the ASR is as likely to be wrong as the TTS.

The study: 390 samples, and a blind human check

I evaluated the method against a production neural TTS service — 130 edge cases × 3 voices = 390 samples — and published it as a citable technical report (DOI 10.5281/zenodo.20757553, CC-BY-4.0).

• Zero structural audio-integrity defects across all 390 clips — the audio was always structurally clean, which matters, because it means the failures that did exist were all pronunciation, exactly the kind WER mislabels.
• Exact-match rate 0.769.
• Then the honest part: a blind human review of the ASR-uncertain "quarantine" zone. Of 42 uncertain clips, 23 (55%) were ASR false-negatives (the TTS said it right, the recognizer misheard) and 19 (45%) were genuine TTS mispronunciations. Fifteen control clips: 15/15 rated correctly, so the rater was reliable.

That 45/55 split is the entire argument for the quarantine verdict. Auto-passing that zone would ship 19 real mispronunciations; naive ASR-WER auto-failing it would wrongly kill 23 correct clips. Neither is acceptable, so ttsproof refuses to guess there.

What the real failures looked like

All 19 genuine failures were short, isolated letters and acronyms — and the pattern is oddly specific:

Note that the structural tail/too-short detectors did not fire on these — "GPUB" is intelligible speech, not a click. Structural checks and ASR-quarantine are complementary; neither alone catches everything.

Benchmark any engine in one command

Beyond the study, ttsproof ships a built-in corpus of 817 curated edge cases across 39 categories — numbers, currencies, dates, ISO timestamps, phone numbers, URLs, file paths, pronunciation-torture words (Worcestershire, synecdoche…), proper names (Reykjavík, Nguyễn…), Greek, Norwegian, and more. The corpus is versioned independently of the software, so published scores stay comparable across tool updates.

It's engine-agnostic — point it at any TTS via a command template or a folder of audio you already generated:

ttsproof benchmark --cmd "mytts --text {text} --wav {out}"

You get a category scoreboard, a self-contained report.html (waveforms, an audio player, and what the ASR actually heard), and a CI regression gate. You can even benchmark closed-source engines (ElevenLabs, OpenAI) through a SpeechSDK wrapper — an integration a user suggested after the first release.

Try it

pip install ttsproof            # structural checks + metrics + corpus pip install "ttsproof[asr]"     # + faster-whisper for pronunciation gating

• Repo: https://github.com/Mormolykos/ttsproof (MIT)
• The study: https://doi.org/10.5281/zenodo.20757553

It's already had its first outside contribution — a community fix for a real number-formatting bug — which is exactly the kind of thing I hoped for. If your TTS breaks on something, open an issue with the case; the corpus grows from real failures.