Your Headline tok/s Is a Short-Prompt Number

118 tok/s at a 30-token prompt, 35.7 at 86,000. Decode halves for every 4× of context and prefill halves past 20k — and on this engine there is no configuration lever for it, because every cache compressor drops you off the fast path.

Summary

We put our own engine into production, published its throughput, and then twice in one session mis-sized a long-context job by 2–4× using those numbers. Everything in the README was measured on short prompts. Decode on this model falls from 118 tok/s at a 30-token prompt to 35.7 tok/s at 86,000 — roughly halving for every 4× of context — and prefill rate halves too past about 20k. Neither number is wrong. They answer a question nobody running an agent is asking.

Update (2026-09-25): The memory was there, but the engine sized its cache pool wrong. It reserved room for about two full-length sequences instead of one per slot, so four concurrent sessions averaging past ~65K tokens would have ended in a "KV cache full" error. Production never hit it: at one to two concurrent requests its logs held zero such errors. The fix landed 2026-09-24. The sizing advice above stands.

The curve

Measured through the gateway against the live production model, four slots, one client:

Our published figures for this model are 155–236 tok/s depending on the workload, with prefill at 2,731–2,764. Those are all the top row of this table.

Batch scaling measured the same day: one stream at 118 tok/s, four concurrent at ~57 each, so 226 aggregate — 1.92×, matching what our engineering log records for this architecture. Worth one warning that falls out of the same mechanism: never set the slot count to 3. A decode pass runs as ceil(N/2) two-token groups, so three slots measures worse than two.

Sizing a job

total ≈ prefill + output_tokens / decode_rate_at_that_context

The second term is the one people get wrong, because they take the decode rate from the headline. An 86,000-token document with 7,000 tokens of output is about 266 seconds with the machine to itself, and comfortably over 400 under any contention. Estimated from the headline number it looks like 110 seconds, which is how a job that was going to be fine ends up timing out at the client.

There is no config lever, and that is the useful part

The obvious reach is for key-value cache compression, and the recipe format offers it: 3-bit and 4-bit quantized cache, and a heavy-hitter eviction policy. Both are traps here.

Reading the engine source rather than the recipe documentation: two separate places gate the GPU cache pool on the compressor being none. Enabling any compressor silently drops the whole path onto a slower one — you lose more than the compression buys. And memory is not the constraint anyway: the per-sequence state is 329 MB, on a machine with hundreds of gigabytes.

Eviction is worse than useless for our workload: it drops tokens, and the job is clinical fact extraction where the dropped token may be the answer.

The lever is input size. Compaction, summarization, chart-first prompt layout, deterministic extraction of the fields that do not need a model — those move the number. Configuration does not.

One hard edge, documented

The engine's context ceiling is 131,072 tokens. Over it, it returns a clean HTTP 400 "prompt is too long" rather than hanging, which is the behavior you want. The gateway's own max_input_tokens: 120000 is advisory and does not pre-check — a 152k-token prompt sailed through the gateway and was rejected at the engine. Fine in practice; worth knowing when you are reading which component said no.

What we do now

Every throughput claim in our README carries the prompt length it was measured at. A number without one is not a number. And the benchmark set includes a long-context row by default, because the short-prompt row is the one that gets quoted into a capacity plan.

Related: we spent a week tuning prefill, the bottleneck was decode, our prefix cache died at token 1,067, and our baseline was the three worst samples in the log.