We Chased an Audio Bug for Months. The Instrument Was Broken.

A stray syllable at the start of generated speech. One ASR pass both invented it and missed it — prepending 15ms of silence changed the transcript.

Summary

Our synthesized speech sometimes began with a stray syllable — a leading "is" or "this" that wasn't in the text. We'd been detecting it with a single ASR pass for months. That detector was worthless: prepending 15 milliseconds of digital silence to the same audio changed the transcript. Once we built an instrument that actually worked, the defect turned out to be deterministic rather than stochastic, entirely determined by which reference clip was used, and caused by a documented requirement we'd violated in all 25 of our reference files.

Part 5 of a series on a consented voice clone. No names, no audio.

The instrument was broken

ASR does not produce a stable first token on a short clip.

Prepending 15, 25, 40, 55, 80, or 110 milliseconds of digital silence to the same audio file moved the transcript between three different readings — with and without the stray word. Using dither instead of zeros doesn't stabilize it either.

Our detector compared the first token of the hypothesis against the first token of the reference text and flagged a mismatch. On an unstable first token, that test both invents defects and misses real ones, at a rate determined by how much silence happens to precede the speech.

Everything measured with a single ASR pass was suspect, including a confident note in our own codebase reading "the comma form bleeds 14 of 14 takes." That number meant nothing.

The fix is a majority vote across padding offsets: synthesize once, transcribe at five different lead-silence values, take the majority, exit early on three unanimous. Slower, and it's the difference between a measurement and a coin flip.

If a detector's output depends on a transformation that shouldn't matter, it isn't a detector. Padding with silence is the audio equivalent of adding whitespace. It was the whole test.

Judge the artifact in the form it ships

Two traps compounded the broken instrument, and both are about measuring the wrong object.

Our pipeline prepends a 55 ms lead before playback. The same take measured clean unpadded and failed 5 of 5 with the lead applied. We'd been judging the raw synthesis and shipping something else.

The instrument now takes an explicit "as served" flag and applies the same processing the listener gets.

A cache hit skips synthesis entirely — and therefore skips every quality gate. Our phrase cache stored known-good audio for common lines. A cached take bypassed the detector completely, so a bad take that got cached once stayed bad forever, invisibly.

Now a cache hit re-judges once and records the verdict in a sidecar file; failures are renamed and re-rendered.

That second one generalizes past audio. Any cache in front of a quality gate is a hole in the gate. If validation happens on the generation path, cached results were validated under whatever rules existed when they were written — or never.

The defect was deterministic

With a working instrument, the finding reversed. This was never stochastic.

Same line, five votes per reference:

Entirely reference-dependent.

The mechanism, once visible, is obvious: our default reference's transcript ends on a sibilant, and the stray token was always sibilant — "is", "this", "yes". The reference tail was surviving the discard boundary and leaking into the output.

Note the best-performing candidate: the same recording as the default, just cut shorter. That's the ideal control — it can't change the speaker's timbre because it's the same audio.

The root cause was in the upstream docs

The framework states the requirement plainly, across three separate issue reports: the reference clip must be under 12 seconds and end in about 1 second of silence, or it truncates mid-word and leaks into the output.

All 25 of our reference clips had 0 ms of trailing silence. And the default was exactly 12.00 seconds — precisely the length at which the code clips.

We'd built every reference by cutting tightly at speech boundaries, which is what you'd do if you were optimizing for content density and didn't know the requirement.

Measured with all fallback and caching disabled, same recording, only the cut differing:

The leaked word is always the reference's last word. The old default ended on one phrase and leaked a syllable from it; the new one ends on a different phrase and leaks a syllable from that. That's a causal chain, demonstrated by changing the cause and watching the effect follow.

We rejected the zero-bleed variant. Cutting 2.5 seconds of speech costs speaker identity — 0.562 against 0.657 — and speaker identity is the entire point of a clone. A perfect score on one axis isn't a win if it's paid for on the axis that matters.

Never fix it downstream

We tried a structural detector — find the stray syllable in the waveform by amplitude, duration, and gap, and trim it.

Amplitude, duration, and gap cannot separate a stray syllable from a soft opening consonant. The detector flagged 24 of 32 cached takes. Head trimming turned "Have no fear" into "ave."

Reject and re-roll instead. When a generative system produces a bad output, the repair is another sample, not surgery on the sample you have. Post-processing that can't distinguish the defect from valid content will destroy valid content at whatever rate it appears.

Nothing had ever measured the actual goal

The most uncomfortable finding: our entire quality stack measured word error rate, pitch, static, and bleed. Nothing measured whether it sounded like him.

So we added a speaker-similarity gate — cosine similarity against a voiceprint built from a handful of real recordings — and, critically, reported it against a measured scale:

• floor (unrelated speakers): 0.028
• ceiling (his own held-out real audio): 0.608
• generated speech: 0.56–0.66

A bare similarity number is meaningless without that floor and ceiling. 0.60 sounds mediocre until you know the ceiling is 0.608.

Building the scale immediately paid for itself. One reference sitting at #2 in our fallback list generated speech at 0.247 against a 0.608 ceiling — the fallback chain was rescuing a stray syllable by handing back a voice half as much like him. Both fallback lists had been ordered on pitch alone, before any identity metric existed, and were reordered on measurement.

Two harness traps that faked entire result tables

dict(x.headers) breaks case-insensitive header lookup. The HTTP library's header object is case-insensitive; a plain dict built from it is not, and the server emits lowercase names. So h.get("X-Pitch", 1.0) on that dict returned the default every time.

It produced a plausible "1.00×" pitch column and a "0.0%" static column across many verification runs that measured nothing at all. The server was correct throughout.

The symptom is a metric that's suspiciously constant across takes of a stochastic process. Speech synthesis is sampling; if your pitch column reads exactly 1.00 on every row, you are not measuring pitch. Constant output from a random process is a bug in the measurement, not a well-behaved system.

Restarting by port leaves strays that re-bind. The log showed "address already in use" while the health endpoint still answered 200 from the old process — so a config change appeared to apply and hadn't. Kill by process name, wait for both the process table and the port to clear, then start one, and grep only the log lines from this start for bind failures. Grepping the tail matches the previous failure and cries wolf.

Key takeaways

1. If padding with silence changes your ASR result, ASR is not your detector. Vote across offsets.
2. Judge the artifact in the form it ships. Ours added a 55 ms lead after the gate ran.
3. A cache in front of a quality gate is a hole in the gate. Re-judge on hit and record the verdict.
4. Reference-clip requirements are in the upstream issue tracker. Ours needed 1 second of trailing silence; all 25 of our clips had zero.
5. Prove causation by changing the cause. The leaked word followed the reference's last word when we changed references.
6. Don't repair generative defects downstream. No signal separates a stray syllable from a soft consonant; trimming ate real words.
7. Report similarity against a measured floor and ceiling, or the number means nothing — and check your fallback chain isn't trading the thing you care about for the thing you're gating on.
8. A metric that's constant across a stochastic process is broken. Ours read exactly 1.00 for weeks.