max_tokens=900 Yielded Zero Pairs. 3000 Yielded 59.

An extraction job that reported 'no conversational material here' on every chunk. The model was spending its entire budget thinking and emitting an empty list as its answer.

Summary

We ran a local model over a transcript corpus to extract real question-and-answer pairs for training data. It returned nothing useful, and the output looked like a legitimate negative — an empty list, no error, no warning. The model was a reasoning model, and our token budget was low enough that it spent the entire allowance thinking and emitted [] as its visible content. Raising the budget from 900 to 3000 took the same 24 chunks from 0 pairs to 59. Separately, we discovered the original extraction had only ever seen about 30% of the corpus.

Part 4 of a series on a consented voice clone. No names, no audio.

The trap

Same transcript, same 24 chunks, same prompt, same model:

The model produced 2,749 characters of reasoning and then emitted [] as its answer.

It looks like a legitimate finding. "No conversational material in this chunk" is a perfectly plausible thing for an extractor to report, and it reported it for every chunk. There's no error, no exception, no truncation flag in a form the calling code was checking. The pipeline ran to completion and produced an empty dataset that looked like an honest one.

The rule: max_tokens is a ceiling, not a target. Observed completions run about 1,430 tokens, so 3,000 gives roughly 2× headroom and raising it further changes nothing. The only thing that would justify going higher is actually observing finish_reason: "length".

Which is the check that should have been there from the start. Assert on finish reason in any extraction pipeline. A length finish on a structured-output task means the result is garbage, and it's one field.

Disabling reasoning doesn't work either

The obvious response — this is an extraction task, turn off the thinking — fails differently.

With reasoning disabled, the model returns [] in two tokens.

It needs to think to do the extraction at all. Reading a transcript chunk and identifying which passages are genuine question-answer exchanges is not a pattern-match; it's the kind of task where the reasoning trace is the work.

So there's no cheap path. Budget for the reasoning or get nothing, and the failure looks identical either way.

The empty-list failure mode, generally

This is the third distinct place in our stack where a reasoning model returned a plausible-looking empty result rather than an error — and each time it cost real investigation:

• A serving stack returning content: None when the token budget was consumed by reasoning.
• A benchmark harness scoring those empty responses as wrong answers, producing a false capability finding.
• This extractor reporting an empty result set as a negative finding.

A reasoning model with an inadequate budget doesn't fail loudly. It returns the empty case for whatever type your prompt asked for — an empty string, an empty list, a null. Every one of those is a valid value in the domain, so every downstream consumer accepts it.

If you run reasoning models in a pipeline, the defensive check is the same everywhere: verify the response is non-empty and the finish reason is not length, before doing anything with it.

The other finding: 30% coverage

While fixing the budget, we read the extractor's constants:

TOP_N_FILES = 34          # of 104 transcripts
MAX_CHUNKS_PER_FILE = 6   # at CHUNK = 3500 chars

At most ~21,000 characters per file, from 34 of 104 transcripts, against a corpus of 403,351 words.

The 840 "authentic" pairs we'd been treating as our real-data baseline came from about 30% of the material, sampled.

Neither cap was wrong when written. Both were sensible during development — get a run finishing in minutes, see if the approach works. Then the approach worked, and the caps stayed, and their output became "the authentic dataset."

A development-time sampling limit that survives into production silently redefines your dataset. It's the same failure as the reference-clip search in part 2, where a break at 60 candidates covered 6% of the corpus and found a substantially worse answer. Twice in one project, from the same instinct.

The full sweep — no file cap, a minimum chunk size, 40 chunks per file — covers 88 eligible transcripts across ~669 chunks, yielding 60–72 pairs per transcript.

Two smaller things worth knowing

Round-robin across model instances, not just requests. One instance was the throughput bottleneck: system load sat at 2.9 on 32 cores while six concurrent requests queued behind a single server. Adding a second instance and alternating fixed it. If your local inference server processes one request at a time, concurrency in the client buys you nothing.

Two servers, two places the reasoning goes. One instance returned the thinking in a separate reasoning_content field; the other emitted <think>...</think> inside content. The parser has to strip the tags before the JSON regex runs or parsing fails on otherwise valid output.

That's an easy one to get wrong when you add a second backend, because the first one works and the failure looks like the model producing bad JSON rather than the client mis-parsing good JSON.

Key takeaways

1. A reasoning model with too small a budget returns the empty case, not an error. Empty string, empty list, null — all valid values your pipeline will accept.
2. Assert on finish_reason in extraction pipelines. length on a structured-output task means the result is worthless, and it's one field to check.
3. max_tokens is a ceiling, not a target. Set it to roughly 2× observed completions and stop tuning.
4. Disabling reasoning is not a workaround for extraction tasks. Ours returned an empty list in two tokens.
5. Development-time sampling caps silently redefine your dataset. Ours meant "the authentic corpus" was 30% of the transcripts, and nobody re-read the constants.
6. Client concurrency does nothing against a single-request server. Load 2.9 on 32 cores with six requests queued.
7. Different servers put the reasoning trace in different places. Strip inline thinking tags before parsing, or good JSON looks like bad JSON.