64% of Our TTS Compute Synthesized Audio We Threw Away

Profiling a voice pipeline: the LLM was 5% of wall clock, three tuning knobs did literally nothing, and most of the work was generating a reference clip the code immediately discards.

Summary

We built a consented voice clone — a local pipeline that listens, thinks, and speaks in the subject's voice. It felt slow. Profiling showed the language model was not the problem: it answers in 0.15 seconds. Text-to-speech was 85–95% of wall clock, and 64–82% of that was synthesizing a 7-second reference clip that the code discards on the next line. Three of the tuning parameters the API accepted were being ignored entirely, including one we'd written a comment about optimizing.

A note on the subject: this was built with consent, for a small private audience. Nothing here names him and no audio is published — the findings are about the pipeline, and they transfer to any zero-shot voice cloning stack.

Where the time actually goes

The language model — the part everyone assumes is the expensive bit, the part we'd spent weeks fine-tuning — is about 5% of the wall clock.

This is worth internalizing for anyone building a voice agent. Your LLM is probably not your latency. A small quantized model on local hardware answers a conversational turn in under a second. Speech synthesis is an order of magnitude more expensive, and it's the part nobody profiles because it's a library call.

The model that explains every number

Reading the synthesis server: it splits text into sentences and synthesizes each one independently. Each call generates reference_frames + generated_frames of audio — and then throws the reference half away:

wave[_ref.shape[0]:]

The reference clip is 7.00 seconds. So:

cost ≈ (7.0s × n_sentences + generated_seconds) / 5.47

The fitted rate is near-constant — 5.46, 5.48, 5.47× realtime across the 2, 3, and 6-sentence cases. A three-line model that predicts every measurement.

For a three-sentence reply, that's 21 seconds of reference audio synthesized to keep about 9 seconds of speech. 64–82% of all TTS compute, discarded.

It isn't a bug. Zero-shot voice cloning conditions on a reference, and this architecture generates the conditioning and the output as one continuous sequence, then slices. The waste is structural. But it means reference length is a first-class latency parameter, and nobody had treated it as one.

Three knobs that did nothing

The request schema accepted steps, speed, and voice. All three were ignored. steps was hardcoded to 8 further down the file.

Measured proof: steps at 4, 6, 8, 10, and 16 produced byte-identical output and flat latency.

There was a comment in the application code that read, roughly, "steps=6 cuts synthesis time by about 27%; steps=4 slurs." Someone had tuned that value, observed a result, and written it down. It had never done anything. The observation was noise, and it had been load-bearing documentation for months.

A parameter that's accepted and ignored is worse than one that errors. It generates confident folklore. If you're building a service, either wire the parameter through or reject it.

What actually moves latency

Once the parameters were wired through, the picture became clean:

Latency is exactly proportional to the number of function evaluations. The solver family sets evaluations per step — a fourth-order method is 4, midpoint is 2, Euler is 1 — so the method is a much bigger lever than the step count. Fourth-order at 8 steps equals Euler at 32 steps, within noise. Both are 32 evaluations.

That's the useful framing: don't tune steps, tune total NFE, and pick the cheapest solver that holds quality.

Two parameters turned out to be dead ends worth naming:

cfg_strength has no speed effect at all. 2.0, 1.5, and 1.0 all land at 5.6–5.9 seconds. The framework computes both branches regardless of the weight. Standard advice says lowering it halves compute; on this implementation it does nothing.

Fourth-order at 4 steps is 55% faster and produces a 29.2% word error rate. Unusable.

The quality gate, and its limits

We used ASR word error rate as an automatic quality gate — synthesize, transcribe, compare to the input text. It catches slurring, static, and dropped words cheaply.

An interesting interaction fell out of it: Euler at 8 steps is worse on the 7-second reference (5.7% WER) but matches baseline on the 2.68-second one (1.9%). Fewer solver steps integrate a shorter conditioning sequence better. The two levers aren't independent, so sweeping them separately would have found neither.

But the honest caveat, which we wrote into the notes at the time: WER is an intelligibility proxy and says nothing about timbre or prosody. A synthesis can be perfectly transcribable and sound nothing like the person. It's a floor, not a measure of success — and treating it as the latter is how you ship something that passes every gate and sounds wrong.

That gap is exactly what bit us next, and it's part 2.

Streaming: predicted 40× win, measured rejection

The server already chunked by sentence. It just concatenated the chunks instead of yielding them. Yielding should take time-to-first-audio on a long reply from 11.96 s to about 1.6 s.

First measurement confirmed it: 13.79 s → 0.29 s warm.

We rejected it anyway, for two reasons that only appeared under real conditions.

Playback overruns generation. With the final production configuration, generation runs at 0.70–1.37× realtime while playback consumes at exactly 1×. Two of three test replies had audible gaps, with a worst-case slack of −3.95 seconds. Buffering enough to prevent gaps means buffering the whole reply — which is what we already do.

And it costs more total compute. Every sentence pays the full reference tax. A three-sentence reply synthesizes ~36 seconds of reference to keep ~9 seconds of speech. Synthesizing all at once pays it once.

The code stayed behind a flag, off by default, in case generation ever gets fast enough. But the lesson is that a streaming win depends entirely on your generation-to-realtime ratio, and ours moved below 1× as we improved quality elsewhere. A latency optimization that was correct in one configuration became wrong in the next.

Parallel synthesis was also worse — one GPU, so concurrent sentences just contend. 15.4 s parallel against 12.1 s sequential.

Key takeaways

1. Profile before optimizing a voice pipeline. The LLM was 5% of wall clock. Everyone assumes it's the bottleneck.
2. Zero-shot cloning synthesizes and discards the reference on every call. Reference length is a latency parameter, multiplied by sentence count.
3. Verify that a parameter does something before tuning it. Ours accepted three and honored none, and a false comment about one survived for months.
4. Tune total function evaluations, not step count. The solver family dominates; a 4th-order 8-step run equals a 32-step Euler run.
5. Sweep interacting parameters together. Euler was worse on the long reference and fine on the short one; separate sweeps find neither.
6. ASR word error rate is a floor, not a quality measure. It says nothing about whether it sounds like the person.
7. A streaming win depends on your generation-to-realtime ratio. Ours dropped below 1× as quality improved, and the optimization inverted.