Our Chunked-Prefill Fix Was 72x Slower. Every Test Passed.

A scheduler change that removed a 14-second stall, plus the earlier version that was quadratic and undetectable by unit tests.

Summary

We added chunked prefill to our inference server's scheduler so that a long prompt arriving mid-stream stops freezing whoever is already generating. It worked: the worst inter-token gap dropped from 14.1 seconds to 0.95. The version before it was 72× slower than doing nothing — 236 seconds against 3.3 — and no unit test could have caught it, because it produced exactly the right answer. In the same week, a metrics endpoint revealed that the batching we'd shipped a day earlier was switched off in production. Both findings are about the same thing: the number you didn't measure.

The stall

Single-sequence serving hides this. As soon as two clients share an engine, one of them arrives with a long prompt, and prefill for a 4,008-token prompt is a lot of consecutive GPU work. The sequence already decoding gets nothing for the duration.

Measured, before any fix: a 14.1-second gap between tokens for the unlucky client. Their stream just stops. Every timeout heuristic downstream — proxy, agent framework, human patience — treats that as a dead connection.

The fix is standard: chop the prefill into chunks and interleave decode steps between them. After it, the worst inter-token gap is 0.95 s.

The trap: total time is unchanged

Here is the part worth internalizing. Chunked prefill does the same GPU work. Total wall time for the two requests together is identical before and after.

So if your benchmark measures total completion time — which is what almost every benchmark measures — this change shows nothing. Zero improvement. You could easily conclude the work was pointless and revert it.

The metric that moves is worst inter-token gap, and you have to go and instrument it on purpose. A latency fix that doesn't change throughput is invisible to a throughput harness, and the reverse is equally true. Decide which one you're fixing before you decide how to measure it.

The 72x pessimization that returned correct answers

The first implementation was 72× slower than not chunking at all: 236 seconds against 3.3.

The cause was one asymmetry in an internal API. The prefill routine reads a slot's cached prefix through a match function, but it never writes one — only a separate "keep this prefix" call does that. When prefill ran as a single blocking pass, that didn't matter; nothing else touched the slot in between. Chunked, every chunk found no recorded prefix, reset the slot, and re-fed the sequence from token 0.

Chunk one prefills tokens 1–512. Chunk two prefills 1–1024. Chunk three prefills 1–1536. It is quadratic, and it is a textbook accident: a function whose contract was implicitly "you are the only caller and you run once."

No unit test could have caught this. Re-prefilling from token 0 gives the identical output. The state is correct, the logits are correct, the generated text is correct. The only symptom is time. Assertions on values are structurally blind to it, and a test that asserted on elapsed time would have been the first thing anyone deleted as flaky.

What would have caught it: counting the prefilled tokens. We now have that as a metric, and a chunked prefill of an N-token prompt that processes substantially more than N tokens is an obvious, cheap invariant.

The corollary: our batching was off

The same week, we wired up a /metrics endpoint. The metrics registry had existed since the first sprint and had simply never been routed — a fair amount of careful instrumentation collecting numbers nobody could read.

Within an hour of it being readable, it showed the batch slots were never above one.

The cause: we'd added a default of max_batch_size: 2 to the code path that derives a model config from the checkpoint. Every model directory already on disk had a written-out config file from before that change, and every one of them still said 1. The feature was implemented, merged, tested, and inert everywhere it actually ran.

This is the standard shape of a defaults change. Changing a default only affects things created after the change. If your system materializes config to disk — and most do, for reproducibility — a default is not a rollout. You have to migrate the existing files, and you should have a metric that would notice if you didn't.

The same endpoint verified the slot-sizing logic: the engine clamps batch slots to 10% of unified memory, computed from config fields before any weights load, and it agreed with the engine's own post-load accounting to within 0.1 GB.

Four more defects, found by looking rather than by failing

We ran a deliberate hardening pass over the serving layer after the scheduler work. Four real defects, none of which had ever produced a bug report, because the server had only ever had one well-behaved client:

• No read timeout. A client that opened a connection and sent nothing wedged the slot forever.
• A second client got silence. Not an error — silence, for the entire duration of the first client's generation. It now returns 429 in 20 ms, which is a worse answer and a much better behavior.
• Every generated token retained a 608 KB logits row. A long generation was quietly holding hundreds of megabytes of tensors nobody would read again.
• max_completion_tokens was ignored. We honored the older field name only. A client using the current spelling got an unbounded generation.

The sampler got the same treatment in the other direction: we added top_k, min_p, and logit_bias, which are all pure per-row transforms and therefore safe for speculative decoding. Repetition, frequency, and presence penalties need generated history, which the speculative path doesn't have in the right shape — so they now return 400 rather than being silently ignored. Refusing a parameter is honest. Accepting and dropping it is how you get a week of "the penalty setting does nothing" debugging on the client side.

A fix that's seventy-two times slower while every test passes is why the setup audit benchmarks the hardware before trusting any change.

Key takeaways

1. A latency fix that doesn't change throughput is invisible to a throughput benchmark. Chunked prefill does identical total GPU work; only the inter-token gap moves — 14.1 s to 0.95 s.
2. Correct-but-quadratic is the hardest class of bug to test for. Re-prefilling from token 0 returns the right answer, so every value assertion passes. Assert on work done — tokens prefilled — not on output.
3. A function whose contract assumes a single call will break when you loop it. Read-without-write asymmetries are invisible until something calls the reader repeatedly.
4. Changing a default is not a rollout if your system writes config to disk. Every existing artifact keeps the old value.
5. Instrumentation nobody can read is not instrumentation. The registry had been collecting for months; routing it took an afternoon and immediately found a shipped-off feature.
6. Refuse unsupported parameters with a 400. Silently ignoring them moves the debugging into someone else's codebase.