The Win Was Four Slots, Not the Prefix Cache

Three agent conversations against two serving slots: time to first token grew from 16 to 61 seconds a turn. Against four slots it stayed flat at 1.6-1.9 seconds, and the prefix cache we had just switched on recorded zero hits.

Summary

A model on our second workstation was being served from the recipe that shipped inside its converted bundle: two slots, 512-token prefill chunks, no prefix cache, the sparse-attention indexer and speculative decoding off, 32K context. That is a conservative recipe written to prove the conversion runs, not to serve anything. An agent workload running three conversations at once timed out on 7 of 12 tasks at a 15-minute limit. We wrote a serving recipe — four slots, 2,048-token chunks, an 8 GB prefix cache, the indexer and the Neural Engine prefill offload on, speculative decoding off — and measured it with a synthetic agent load. Every turn after the first went from a growing 16-61 seconds to first token down to a flat 1.6-1.9 seconds. The obvious explanation was the prefix cache. The cache recorded zero hits. The win was the two extra slots.

The load

Three concurrent conversations of ten turns each. Turn one is a ~2K-token shared system prompt plus a ~4K unique task, about 6.1K tokens; every later turn appends ~1.6K tokens of synthetic "tool output", growing each conversation to about 20.5K. Each turn generates up to 200 tokens at temperature 0 with thinking off, streamed. The server's queue depth was sampled every half second. One run per arm, not interleaved — so treat the walls as indicative. Time to first token is the clean comparison.

Single-stream, the new recipe is no faster — slightly slower, if anything. Everything it wins, it wins under concurrency.

Why it wasn't the cache

With three conversations on four slots, each conversation kept its own slot between turns. The engine prefers to resume a conversation in the slot that already holds its state, so every turn prefilled only the ~1.6K new tokens at its tail. The prefix cache never had anything to do: 32 lookups, 32 misses.

With three conversations on two slots, the third conversation queued behind the other two. Every time a slot turned over to a different conversation, that conversation's entire history — up to 20K tokens — had to be prefilled again, in 512-token chunks. And while one stream was re-prefilling, the other stream's decode starved: 1.7 tokens per second. That is where 16 seconds became 61: the history being re-read grew by 1.6K tokens every turn.

The prefix cache matters in the other regime — when conversations outnumber slots, so that a conversation comes back to a slot some other conversation has used. We left it on, because that regime is real. But had we credited the result to it, we would have tuned the wrong thing next.

Two settings we measured rather than assumed

• Interleave chunk at the engine default of 1,024. A smaller value, 256, had starved agent tails in an earlier test queue, so we kept the default rather than guess again.
• Speculative decoding off. On this bundle it loses at a batch of four, as it did on another model. It is the right default solo and the wrong one here.

The cost of four slots was modest: 0.94 GB of per-slot state, 112.8 GB resident in all.

Verdict

The recipe inside a converted bundle is a proof that the conversion runs; do not serve from it. For agent traffic, count your concurrent conversations and give each one a slot before you tune anything else — slot affinity makes each turn prefill only its new tail, and that is worth more than any cache. Then check the cache's hit counter before crediting it with the result. Ours said zero.

Related: our prefix cache died at token 1,067 and our two machines ran different models under the same name.