We Spent a Week Tuning Prefill. The Bottleneck Was Decode.

A config knob that won't boot, a context sweep that moved throughput by 1.6%, and the vLLM histogram showing the effort was aimed at 13% of the problem.

Summary

We believed long-context prefill starvation was throttling our LLM cluster. So we tried to raise the prefill batch size. It wouldn't boot. Then we swept context length in both directions and found a 4× range moved throughput by 1.6%. Then we read the engine's own histograms and discovered prefill is 13.4% of inference time and decode is 86.6% — the entire effort had been aimed at the small number. Three negative results, one useful correction, and a reusable piece of KV-cache arithmetic.

The knob that refuses to move

The theory: bigger prefill batches would relieve starvation when a long request arrives. So, MAX_NUM_BATCHED_TOKENS from 8192 → 16384.

It failed to boot, cleanly and deterministically:

ValueError: To serve at least one request with the model's max seq len (768000),
9.8 GiB KV cache is needed, which is larger than the available KV cache
memory (6.9 GiB). Based on the available memory, the estimated maximum
model length is 35072.

Container exited 1. Not OOM-killed — the host had 96 GB free. The engine refuses to start rather than serve degraded, which is the correct failure and much better than the alternative.

The arithmetic is the reusable part. At 8192 the KV pool holds 1,110,255 tokens. Since 768,000 tokens of context requires 9.8 GiB, that pool is about 14.2 GiB. At 16384, only 6.9 GiB remained.

So +8192 tokens of prefill batch costs about 7.3 GiB of KV cache — roughly 0.9 MB per batch token on this configuration.

Run the intermediate: 12288 would leave about 10.5 GiB against a 9.8 GiB floor. Marginal, and not worth the risk of a boot failure in production.

8192 isn't an arbitrary default here. It's already near the ceiling imposed by keeping 768k context. Which also explains something we'd noticed earlier and not understood: a previous A/B had only ever tested this knob downward. Now we know why.

The real lever, if you want a bigger prefill batch, is max_model_len. And that's a product decision — how much context do you promise your users — not a tuning decision.

So we tested context length. It's a dead end too.

Three arms, one harness, one session, the cluster drained from the pool so nothing else contended:

A 4× context range and a 2× KV difference move throughput by 1.6%. That's noise. KV was never the constraint at this workload shape — peak usage was around 50%.

Two corrections fell out of this that are worth more than the result:

Lowering context does not free memory. 256k gave less available KV than 768k — 8.95 against 12.08 GiB — and roughly halved the token pool. That's the opposite of the intuition everyone has, including us.

Don't derive a per-token KV cost from any of these numbers. A later boot at the identical 768k/8192 config reported 8.44 GiB → 1,085,106 tokens, against 12.08 GiB → 1,110,893 earlier. Same config, nearly the same token count, 30% different reported memory.

The GiB figure is unreliable boot to boot. Only the token pool is worth comparing, and it varies about 2.3% across boots at fixed config — which is how we know the 256k halving is a real effect and not variance.

We settled on 768k: same throughput as 1M, more concurrency headroom (1.45× vs 1.11×).

Then we read the histograms

Everything above was driven by a framing: prefill starvation is the bottleneck.

The engine exports its own accounting. Over 401 production requests:

Prefill is 13.4% of inference time.

We had spent the week tuning a lever that can touch, at absolute maximum, about a seventh of the problem.

Where the wrong number came from. We'd been quoting "1,860 tok/s prefill," derived from total prompt tokens over total wall clock. That figure silently includes decode time, so it is not a prefill rate. It's a made-up unit that looks like one.

Read vllm:request_prefill_time_seconds_sum against vllm:request_decode_time_seconds_sum instead. The engine bursts to 11,000 tok/s of real prefill — six times our fake number. And 83.8% of scheduler iterations process eight tokens or fewer, meaning they're decode iterations.

Scale it up to a 38,000-token prompt and it barely moves: prefill about 28s, decode about 137s. Decode is still 83%.

Consequences. Every prefill-side lever — batch size, context length, and prefix caching — can only ever touch 13–17% of inference time. Decode is memory-bandwidth-bound on this hardware's 273 GB/s and scales with tokens generated.

Which is what pointed us at the actual biggest lever: reasoning mode, which was on by default and multiplies output tokens. That's the thinking-mode cost post, and it's where the real wins turned out to be.

Two things we found while we were in there

69% of requests finish by length, not stop — 280 against 125. Full decode cost paid for answers that got cut off mid-sentence. That's a client configuration problem, not an engine problem, but you only see it in the finish-reason distribution.

The prefix cache was saving nothing. request_prefill_kv_computed_tokens exactly equalled request_prompt_tokens, with a 3.7% lifetime hit rate. Since prefill is only 13.4% of the time anyway, fixing that is worth at most a percent or two here — but on a prefill-dominated workload it would be the first thing to chase.

How to not do this to yourself

The mistake wasn't any individual experiment. Each one was competently run and each produced a real answer. The mistake was that we never checked the premise before spending a week on it, and the premise was available in a metric the engine was already exporting.

The specific trap was a derived number that looked authoritative. "1,860 tok/s prefill" appeared in our notes, our dashboards, and our comparisons. Nobody re-derived it, because it was already written down. It was wrong by a factor of six, and it encoded an assumption — that prefill was where the time went — that was itself the thing under investigation.

A derived metric can quietly carry a hypothesis inside it. Then every measurement you take with it confirms the hypothesis.

Checking the premise before tuning — prefill versus decode, straight from the engine's own counters — is exactly what the setup audit does before you spend a week on the wrong lever.

Key takeaways

1. Read the engine's own accounting before tuning anything. Prefill vs decode time is a single query. It would have redirected the entire week.
2. A rate computed over total wall clock is not that rate. Prompt tokens ÷ total time includes decode. Ours was off by 6×, in the direction that supported our assumption.
3. +8192 prefill batch tokens ≈ 7.3 GiB of KV at 768k context. Run the arithmetic before the config change; the engine will refuse to boot, which is safe but slow to learn from.
4. Lowering context length does not reliably free KV memory. 256k gave less available KV than 768k on our config. Measure it; don't reason about it.
5. Compare token pools, not reported GiB. The GiB figure moved 30% across boots at identical config. The token pool moved 2.3%.
6. A "clean, deterministic refusal to boot" is a good failure. It cost an afternoon. Serving degraded silently would have cost weeks.