One Slow Request Locks Out Everything Else

Requests decoding at 1 token/second on hardware that does 42. Not a roofline — a scheduler default and a gated-off config field behind it.

Summary

After a 165-deep queue burst, some requests on our cluster were decoding at roughly 1 token per second. The same hardware, same recipe, decodes an isolated 512k-context request at 42.4 tok/s. That gap is a factor of forty, and it isn't a hardware limit — it's scheduling collapse. We found the mechanism, found that the documented fix won't boot on our image, and came away with a ranked list of levers plus a longer list of things that definitively don't work.

The gap

The symptom: during a burst, individual long requests crawl. One token per second, sometimes worse.

The comparison that matters: an isolated 512k-context request on this exact recipe and hardware decodes at 42.4 tok/s.

We weren't at the roofline. The theoretical floor for a dense long-context request here is roughly 5–15 tok/s before speculative decoding, and the model's sparse attention path — a top-k indexer that caps KV reads at about 18 MB — is what makes 40+ tok/s at depth possible at all.

So the 1 tok/s wasn't physics. It was batch-sharing arithmetic: 16 concurrent slots plus 8192-token prefill chunks dividing up the available forward passes per second. Each request gets a slice, and when the slices are small enough, everything crawls together.

An upstream issue documented our exact symptom — concurrent lanes at 1–4 tok/s, degradation correlated with concurrent prefill — which was both reassuring and a little deflating.

The smoking gun

max_num_partial_prefills defaults to 1.

That means one enormous prefill — a 359,000-token request arriving — locks out all other prefills for as long as it takes. Minutes. Every other request in the queue waits, including the tiny ones that would have finished in two seconds.

The documented fix, which upstream adopted for the same issue:

--long-prefill-token-threshold 1024
--max-num-partial-prefills 2
--max-long-partial-prefills 1

The fix doesn't boot

NotImplementedError: Concurrent Partial Prefill is not supported

The config fields exist. The feature is gated off in our image. Proved on a drained cluster, auto-rolled back.

This is the second time we hit this exact pattern: a parameter that exists in the build, appears in the config schema, and is rejected at runtime because the feature isn't wired up on this code path. The first was thinking_token_budget (see the thinking-mode cost post), which had the worse variant of the failure — it returned HTTP 200 and silently did nothing.

A field existing in a config object tells you nothing about whether the feature is implemented on your build. Read the validation code, or test on a drained node.

What's still untested: threshold-only, leaving partials at 1. It may pass validation, and the upstream lineage does set that threshold. But its semantic value without concurrent partials is unverified — it might do nothing at all. That's on the list, not in the write-up, because we haven't measured it.

The traffic shape during a burst is not what you'd guess

658 requests over three hours:

• p50: a 3,000-token prompt with a 37-token completion. Tiny.
• Nine "monsters" at ≥100k tokens, each producing about 124 output tokens. Pure extraction — read an enormous document, return a short answer.
• Prefix cache hit rate: 4.2% during the burst, against 98% in quiet hours.

That last line explains a lot about why bursts feel so much worse than their request count suggests. The requests that arrive together are all different, so the cache that carries you through normal operation evaporates exactly when you need it.

And the shape is adversarial for scheduling: a handful of requests with enormous prefill cost and negligible decode cost, sharing a queue with hundreds of requests that are the opposite. Any fairness policy that treats a request as one unit gets this wrong.

What we actually shipped

Update (2026-08-28): We later removed this gateway router. Its premise expired when the heavy cluster was reassigned to another model, which left a single cluster behind it. A lane slot was released only on success or failure, never on a client disconnect, so slots leaked until the heavy lane wedged while the GPUs sat idle. Concurrency now comes from the engine's sequence cap and a per-deployment parallel-request limit at the gateway. The scheduling analysis in this post still stands.

Since the scheduler fix wouldn't boot, we worked at the gateway instead — routing by request shape.

Pin the monsters to one cluster. Requests above a size threshold go to a designated heavy cluster only, with the other cluster available as an emergency lane if the heavy one is unhealthy. If the heavy cluster is full, monsters wait rather than spilling onto the cluster serving everyone else.

Three operational lessons came out of running it:

Measure in the units your threshold uses. Our router measured characters, and we set the threshold from a token estimate. Dense content — code, JSON — runs about 3 characters per token rather than the ~4 you assume from prose. A 113,000-token request measured about 340,000 characters and slipped under a 400,000-character bar. Two of fourteen monsters leaked onto the wrong cluster before we dropped the threshold to 300,000.

A bare except: return data in a routing hook hides every bug it causes. Ours did, silently, for a while. Adding explicit decision logging — what size was measured, which lane was picked, what the health cache said, and any swallowed exception — turned the router from a black box into something debuggable. If a hook can fail open, it must log when it does.

Watchdog timeouts have to account for legitimate slowness. Once the heavy cluster was deliberately pinned with monsters, it became legitimately slow — a 719,000-token decode outlives a 90-second liveness probe. Our watchdog started paging on a perfectly healthy engine. Raising the probe timeout to 300 seconds cost no safety: timeout-class alerts never triggered an automatic restart anyway, and the connection-refused and crash paths still fire immediately.

That last one is a general shape. When you deliberately make one component slow, every timeout that was tuned for the old behavior becomes a false alarm generator.

Ranked levers, for anyone in the same position

1. Shape-shard the clusters. One tuned for fast short requests (smaller context, more sequences), one for monsters (huge context, few sequences), with token-count routing at the gateway. A published four-node benchmark measured about 2.4× aggregate from partitioning this way. This is the big one, and it's mostly configuration.
2. Prefill-fairness flags, if your build supports them. Ours doesn't yet.
3. Pull the upstream scheduler fixes. A "bounded decode service" commit landed upstream while we were tuning around the symptom. Worth checking before you build anything: someone may have already fixed your problem.
4. Priority scheduling. Orders the waiting queue only, doesn't preempt running requests. Modest, and there's an open crash bug with n>1.
5. GPU memory utilization 0.78 → 0.82–0.85. Every published recipe for this hardware runs in that band; we were being conservative.

Anti-levers: measured or reasoned out, don't retry

• Raising max_num_batched_tokens to 16384. Won't boot at our context length. The prefill post has the KV arithmetic.
• More sequences without a context cap. More KV per forward pass makes every request slower, including the ones you're trying to help.
• Four independent single-GPU nodes. The weights are 148.7 GB. They don't fit.
• Disabling speculative decoding. Speculation is most profitable at long context, and the crossover where it stops paying is around 64–80 concurrent requests. We run at 16. There are also open bugs in the batch-size-based auto-disable.
• CPU offload / KV tiering. This hardware has unified memory — it's the same 273 GB/s pool. There is no slower tier to offload to, so there's nothing to gain.
• Prefill/decode disaggregation across all four nodes. Loses high availability, halves decode capacity, and shape-sharding gets you the same benefit for free.

There is no single tokens-per-second number

One more thing worth publishing, because people ask for a single figure and there isn't one.

Single-stream, idle engine, reasoning off, same max token budget, distinct prompts:

Code decodes about twice as fast as prose, and it tracks speculative-decoding acceptance almost exactly. The draft head agrees with the target model on structured code and disagrees on novel prose. That's the whole mechanism.

Quote a range, and always say which workload it describes. Lifetime acceptance — 55.7% for us — just averages the two regimes into a number that describes neither.

Measure acceptance with counter deltas around a single request, not from the raw cumulative counters. We got this wrong first.

The lifetime numbers describe the felt experience better than decode rate does: mean time-to-first-token of 3.77 seconds across 34,061 requests, on an average prompt of 34,573 tokens, with a prompt-to-generation ratio of 144:1. Our workload is overwhelmingly about reading, not writing — which is consistent with the traffic shape in the idle-cluster post, and worth knowing before you optimize the generation path.

This is the failure mode an on-site setup is built to prevent: one pathological request should not be able to hold every other tenant hostage.

Key takeaways

1. Compare against an isolated request before blaming hardware. 1 tok/s under load against 42 tok/s isolated says "scheduler," not "roofline."
2. max_num_partial_prefills defaults to 1, and one huge prefill can block every other prefill for minutes. Check it on any long-context deployment.
3. A config field existing is not evidence the feature works on your build. Twice now: one rejected at boot, one silently ignored at HTTP 200.
4. If your router measures characters, calibrate on dense content. Code and JSON run ~3 chars/token; a prose-derived threshold lets real monsters through.
5. A hook that fails open must log when it does. A bare except: return data turns every bug into invisible mis-routing.
6. Deliberately slowing one component invalidates every timeout tuned for the old behavior. Budget for the false-alarm cleanup as part of the change.
7. Publish a tok/s range with the workload attached. Ours differs by 2× between code and prose, entirely through speculative acceptance.