A 9.5x Pessimization That Looked Like a Hardware Limit

Batching made speaker embedding nine times slower for numerically identical output. That one bug made a multi-hundred-hour corpus look infeasible to process.

Summary

We needed to find one person's speech inside hundreds of hours of multi-speaker recordings. The first throughput estimate said 6× realtime, which made the job a multi-week affair and nearly killed it. The cause was a batching call that ran 9.5× slower than single-item processing for numerically identical output. With that removed, the pipeline ran at 57× realtime — a 55-minute recording processed in 58 seconds. Two other things saved as much time again: a cheap pre-scan that rejected files before paying for a full pass, and checking a classifier's output against someone's domain knowledge.

Part of a series on a consented voice clone. No names, no audio.

The batching trap

Speaker embedding is the core operation: take a window of audio, produce a vector, compare it to a target voiceprint.

The obvious optimization is batching — push 32 windows through the model at once instead of looping.

Measured: 8.6 seconds batched, 0.9 seconds single-window. For output that was numerically identical, maximum difference 0.0000.

A 9.5× pessimization, from the call everyone reaches for by reflex.

With single-window restored, the pipeline hit 57× realtime — a 55-minute file in 58 seconds. The original 6×-realtime estimate, the one that made the whole job look infeasible, was entirely this bug.

We didn't chase the mechanism far — on this CPU-bound path, the batched code path evidently does something that costs more than it saves. The point isn't the mechanism. It's that batching is an assumption, not a law, and it takes one timing comparison to check. We nearly abandoned a project over a number that a two-line experiment corrected by an order of magnitude.

The tell was available in advance: the throughput figure implied hardware performance far below what the machine does on comparable work. A throughput number that implies your hardware is bad is usually a bug in your code.

Reject before you pay

The corpus was several hundred recordings. Roughly half of it doesn't contain the target speaker at all — other speakers, other formats, entirely unrelated content.

Running the full isolation pass on every file to discover that would be about 12 minutes per file, most of it wasted.

The pre-scan: decode ten 12-second slices spread through the file, embed about 20 windows, decide whether to bother.

1.3 minutes for 75 files, against ~12 minutes per file for a full pass. Keep rate 45–49%.

Then the pipeline gates on it — the pre-scan runs first, and rejected files never cost a full pass. That structure is worth more than any single optimization in the project: a cheap sampling test in front of an expensive exhaustive one, where the cheap test only needs to be right about the negatives.

Note the asymmetry that makes it safe. A false reject loses one file out of hundreds. A false accept costs one wasted full pass. Neither is expensive, so the pre-scan can be aggressive.

There's a tension here with a lesson from part 3, where a sampling cap gave a much worse answer than an exhaustive search. The difference is what's being sampled: sampling to reject candidates is fine, because the exhaustive pass still runs on everything that survives. Sampling to pick a winner means the winner is only the best of your sample. Same technique, opposite consequence.

Checking a classifier against domain knowledge

The classifier was blind to titles and metadata — it only heard audio.

Someone who knew the material said: that kind of session isn't him — it's a different recording context.

The classifier, with no access to that, rejected 11 of 11 cases from that category (0% keep) against 78% for one category and 62% for another. It also rejected, outright, two other whole categories of unrelated material.

That's a free validation set nobody had to label. When a human who knows the domain can predict which items should be rejected, you have a labeled test set for your classifier and it costs one conversation.

It also converted "the keep rate is 45%, is that right?" from an unanswerable question into a confirmed one.

Settling the hard confusion case empirically

The obvious risk was confusing the target with a second frequent speaker — same setting, same vocabulary, similar recording conditions.

Rather than assume, we measured it on the six hardest cases: recordings the classifier skipped despite high similarity scores.

The other speaker beat the target by a median of 0.40–0.47, with a minimum gap of 0.28, and zero of 63 windows fell inside the decision margin.

Cleanly separated. Not a confusion risk. And the specific operational conclusion: don't loosen the margin threshold to chase those files — they're genuinely the other person, and loosening the threshold to capture them would start admitting real errors.

That's the kind of question that generates endless debate and is settled by one afternoon of measurement on the hardest cases you have.

The yield

From the first fully processed recording: under an hour of audio, a little over half of it target speech, the rest other speakers, several hundred speaker turns, and dozens of question-to-answer handoffs.

Extrapolated across the corpus: roughly tens of thousands of real dialogue pairs, against the several hundred authentic pairs in the existing dataset. An order of magnitude more.

That's what made it possible to drop every fabricated row from the training set without losing volume — the subject of part 4.

Broad, noisy harvesting is safe when you have a reliable filter downstream. Because speaker isolation rejects any voice that isn't the target, we could pull in everything and let the classifier sort it out, rather than curating the input list by hand. The filter's precision is what buys you the ability to be indiscriminate upstream.

Key takeaways

1. Batching is an assumption, not a law. Ours was 9.5× slower than looping, for bit-identical output.
2. A throughput number implying bad hardware is usually a bug in your code. Ours nearly killed the project at 6× realtime; the real figure was 57×.
3. Put a cheap sampling test in front of an expensive exhaustive one. 1.3 minutes for 75 files against 12 minutes each.
4. Sampling to reject is safe; sampling to select is not. Same technique, opposite consequences, and we made both mistakes in one project.
5. Ask a domain expert which items should be rejected. It's a free labeled test set for your classifier.
6. Settle confusion risk on the hardest cases, not in the abstract. Zero of 63 windows inside the margin ended a long-running worry.
7. A precise filter lets you be indiscriminate upstream. Reliable rejection is what makes broad harvesting safe.