The Fine-Tune Was Dead Weight. The Reference Clip Was Doing the Work.

We fine-tuned a TTS model for weeks. The base model beat it by ear and ran 2x faster. The actual quality ceiling was a 7-second audio clip with a 5.2 kHz bandwidth.

Summary

Our cloned voice sounded thin, and we assumed the fine-tune needed more data. It didn't. Zero-shot TTS reproduces whatever reference clip you hand it, and ours had a 5.2 kHz spectral bandwidth and a 16 dB signal-to-noise ratio — everything above 5.2 kHz simply did not exist in the source, so no model could synthesize it. An exhaustive scan of the corpus found a far better clip. Then the base model, cloning that clip, beat our fine-tune by ear and ran twice as fast.

Part 2 of a series on a consented, unpublished voice clone. Nobody is named and no audio is shared; the findings are about the pipeline.

The wrong hypothesis

Symptom: the voice was intelligible, passed every automated gate, and sounded thin. Not wrong — thin. Missing the air and presence of the real thing.

The natural theory when you've fine-tuned a model is that the fine-tune needs work. More data, more epochs, better hyperparameters. We had been operating on that theory for weeks.

The correct question turned out to be much cheaper: what is actually in the reference clip?

The reference was the ceiling

Everything above 5.2 kHz — sibilance, breath, air — was not present in the source audio.

Zero-shot cloning conditions on the reference. If the information isn't in the conditioning signal, the model cannot invent it. No solver setting, no step count, no amount of fine-tuning recovers absent frequency content.

The clip was also spliced — assembled from concatenated speaker-verification windows rather than being one continuous take. We'd built it to maximize speaker-similarity score, which is exactly the wrong objective if it produces discontinuities the model will faithfully reproduce.

Diagnose the reference before diagnosing the model. Two numbers, computed from the waveform in about a minute: bandwidth as the frequency below which 99.5% of cumulative spectral energy sits, and SNR as the ratio between the 90th and 5th percentile frame RMS. We'd never looked.

Exhaustive search beat sampling

The first attempt at finding a better clip had a sampling cap — stop after 60 candidates. It covered about 6% of the available recordings and topped out at 6.7 kHz. Marginally better, still bad.

The question that fixed it was blunt: did we check all the clips, or a few?

The full scan — 461 windows across 221 of 530 processed recordings — found a candidate at 10.9 kHz.

Two things about how that scan was run:

Confirm the speaker first, then rank by quality. An earlier quality-first search returned windows from other speakers entirely. Filter to speaker-confirmed segments, then sort by audio quality. Doing it the other way finds the best-sounding audio in the corpus, which is not the same problem.

A cap you set for speed becomes a cap on the answer. A 6% sample of a corpus is not a search, and the difference here was 6.7 kHz against 10.9 kHz — the gap between "still thin" and "solved."

The choice that wasn't the obvious one

We built new 12-second references with a light processing chain: a high-pass at 60 Hz, a de-essing cut around 4.5 kHz, dynamic normalization, and a limiter.

The winner was not the highest-bandwidth candidate.

Chosen by ear, and the reasoning holds up: a 36 dB noise floor gets cloned as part of "the voice." The model reproduces the room, the hiss, and the compression artifacts as faithfully as it reproduces the timbre. Cleanliness beat raw bandwidth.

That's a general property of zero-shot cloning worth stating plainly: everything in the reference is signal, including the parts you'd call noise.

Of all 461 scanned candidates, none beat the chosen one on both axes — the next best was a tie at 6.8 kHz / 54 dB. That decision closed.

The fine-tune lost

With a clean reference in place, we A/B'd our fine-tuned model against the stock base model, both cloning the same clip with the same solver.

The base model won by ear. It was also about 2× faster (4.5 s against 9.4 s on a short line) and loaded in 0.2 s instead of 3–5 s.

This confirms a documented failure mode we'd been warned about and dismissed: fine-tuning a TTS model on a small, noisy, spliced dataset produces output that goes monotone and mechanical while training loss keeps falling. That profile — isolated segments cut from multi-speaker broadcast audio — was exactly ours.

The reference clip had been doing the work all along.

Weeks of fine-tuning, and the deliverable was a model that was slower, larger, harder to load, and worse. We kept it behind an environment variable in case the judgment ever reverses, but the default is now the stock model.

The uncomfortable part is that a falling training loss looked like progress the entire time. There was no point at which the metrics said stop.

The final configuration, all chosen by ear

Fourth-order at 6 steps measured 6.00 s against 8.52 s — 1.42× faster, with quality approved by listening.

Note what happened to the speed work from part 1. Euler at 8 steps was a 5.7× win on the old thin reference and matched it on the automated gate. Against the clean reference, listening rejected it. The automated gate had been unable to distinguish "faster" from "worse" the entire time, because word error rate measures intelligibility and the defect was timbre.

Once the source was clean, a human could hear the difference immediately. Step count is now the speed dial, because the solver family is fixed by ear and cost is linear in function evaluations.

What we couldn't fix

Two ceilings we identified and left in place, because naming a ceiling is more useful than pretending it isn't there:

The vocoder. Our framework ships one vocoder; the original paper measures a different one as better. It isn't available in our stack. That's a fixed quality ceiling until it is.

Better systems exist and require CUDA. Two more recent TTS systems outperform ours generally. Both are PyTorch/CUDA, and we're on Apple Silicon. A real constraint, not a preference.

Key takeaways

1. In zero-shot cloning, the reference is the ceiling. Measure its bandwidth and SNR before touching the model. Two numbers, one minute.
2. Absent frequency content cannot be recovered. Nothing above 5.2 kHz existed in our source, so nothing above 5.2 kHz could be synthesized.
3. Everything in the reference gets cloned, including the noise floor. We picked 6.6 kHz at 55 dB over 10.9 kHz at 36 dB, by ear.
4. A sampling cap set for speed becomes a cap on the answer. 6% coverage found 6.7 kHz; the full scan found 10.9 kHz.
5. Confirm the speaker, then rank by quality. The reverse finds the best-sounding audio in the corpus, which is a different question.
6. Fine-tuning on a small, noisy, spliced dataset degrades TTS while training loss falls. There is no point where the metrics tell you to stop.
7. An automated gate that measures the wrong axis will approve regressions. Ours passed a configuration that a human rejected in one listen.