Speculative Decoding Won Alone and Lost Batched

+25% decode at one sequence, −27% at four. The draft head barely engages once sequences batch — 126 drafted passes solo, 1 batched — while still paying for itself. Which way you set it is a question about your traffic, not your engine.

Summary

We turned on the full feature set for a 177B mixture-of-experts model — depth-1 speculative decoding, the sparse-attention indexer, Neural Engine prefill offload, larger prefill chunks, an 8 GB prefix cache — and measured it against the same model with all of that off. Solo it is +29% prefill and +25% decode. At four concurrent sequences it is −12% and −27%. The configured arm is worse than the stripped one under concurrency, and the mechanism is visible in the log: the draft head ran 126 times on solo passes and once on a batched pass. It stops engaging and keeps costing.

Update (2026-09-25): On a different model, a 320B MoE, making drafted passes cheaper turned MTP at two sequences from −1.5% to +8.6% against MTP off. That fix targets per-pass cost. The loss here was drafting that stopped engaging under batching, and we haven't re-measured this 177B model at four sequences since.

The measurement

Both arms max_batch_size: 4, two rounds, reproduced to 0.1 tok/s. Same bundle on both sides, so this is a settings A/B and not the different-conversions problem that preceded it.

Draft acceptance stayed at 82% in both regimes, so this is not a quality-of-draft problem — the drafts are good, they just stop being produced. Per-sequence state rises from 235 MB to 353 MB with the features on, which is the other half of the cost.

The same inversion has been reported for this model family on a completely different runtime, where the guidance is to require single-stream operation for the speculative path. Measuring it on our own engine makes it look like a property of this model's draft head, not of either implementation.

Why a solo win inverts

Speculative decoding buys latency by spending capacity: draft k tokens, verify them in one pass, keep the accepted prefix. At one sequence the machine has capacity to spare and the trade is free. At four sequences the verify pass is already wide — the batch itself is filling the machine — and the scheduler stops finding passes where drafting is worth attempting. What remains is the bookkeeping, the extra per-sequence state, and a wider verify shape than the plain path would have used.

So the feature does not degrade gracefully into neutrality under load. It degrades into a cost.

Choosing, which is a traffic question

Neither setting is correct in the abstract:

• Optimizing latency or low concurrency → speculative decoding on.
• Optimizing aggregate throughput under real concurrency → off, and batch instead.

We measured our own traffic before deciding: the fleet runs at one to two concurrent requests, with 74% of busy time at three or fewer. At that shape, the batch-4 arm's aggregate advantage never materializes, and the solo latency win is what users actually experience. So: on, with a batch-4 ceiling, which costs nothing at one sequence and still beats batch 1 at four (97.4 vs 70.9 aggregate).

A different deployment — a nightly extraction job, a fan-out that fires seven agents at once — should read this table and choose the other way.

The general form

We have now been bitten twice in one week by the same shape, in two directions:

• A win at one sequence can be a loss at four. This post.
• A win on a small chip can be a third of itself on a large one. The same sub-expert gather bench measured 4.57× on a 10-core laptop GPU and 1.49× on an 80-core workstation, because at one row the kernel is launch-bound and a wider chip runs out of work before it runs out of bandwidth.

Both are the same failure: a result measured in one regime, quoted as a property of the code. The fix is cheap and boring — measure at batch 1 and at the configuration's real batch ceiling, on the chip you will run it on, and publish both rows.

Related: one slow request locks out everything else and 2× decode from block-diffusion drafting.