Our Baseline Was the Three Worst Samples in the Log

A synthetic prefill benchmark said 115 tok/s. The production request log said 320. Two research agents had already built rankings on the wrong number.

Summary

Evaluating a replacement inference engine, we needed a baseline for the one we already run. We wrote a careful synthetic benchmark — cold prefill, fixed prompt, greedy, a unique prefix per repetition to defeat the cache — and got 115 tok/s. The server's own request log, over 541 production requests, says 320 tok/s. Our three repetitions turned out to be the three worst samples in the entire matching bucket of real traffic. The bad number reframed the incumbent from "healthy, 30% headroom" to "terrible, 4× opportunity," and two research agents built full option rankings on it before anyone checked.

The benchmark, and why it looked fine

The harness did everything you're supposed to do. A 24,750-token prompt, max_tokens=1 so only prefill is measured, temperature 0, and a unique UUID prefix per repetition to force a cold prefill past the in-memory session cache. Three repetitions:

We reported ~115 tok/s as the incumbent's baseline, took the best of three, and moved on to the interesting question.

What the production log said

The server writes a JSON line per request. Over 541 of them:

Real baseline: about 320 tok/s. A genuine 25k prompt prefills in roughly 78 seconds, not 214.

Our three repetitions weren't merely unrepresentative. Checked against the 20–32k bucket of real traffic, they are the three slowest samples in it. We had, by accident, constructed a benchmark that reproduced the tail.

Two things did that. Repetitions 2 and 3 were contaminated — the box is shared, and other traffic landed mid-run; the "controlled" part of the experiment controlled our own process and nothing else. And repetition 1, the clean one, was still 3× off, because a synthetic 100%-cold prefill is a request shape production essentially never sends.

The log's own field was also wrong

Note the two rightmost columns differ. The server reports a prefill_tok_s field, and it's inflated by 1.05–1.13× because cached tokens are counted in the numerator while the denominator is wall time. Prefill that didn't happen is credited as prefill that did.

So even after moving to production data, the honest figure needs re-deriving: new_prefill_tokens ÷ time_to_first_token. Any throughput field on a system with a cache should be checked for whether the cache's contribution is in it. It usually is, because it makes the number better.

What the bad number cost

The roofline for this machine is about 460 tok/s for this model. At 115, the incumbent looked like it was running at 25% of the hardware — a 4× opportunity, more than enough to justify swapping engines, and a strong argument for a kernel-level project.

At 320, it's at 70% of roofline with about 1.4× left. That's a healthy engine, and the remaining headroom is concentrated in one specific layer type rather than spread across the design. Those two pictures lead to completely different quarters of work.

By the time it was caught, two research agents had produced ranked option lists built on the 115. Every ranking was internally consistent and wrong at the root. A wrong baseline doesn't produce obviously wrong conclusions — it produces confident, well-argued, wrong ones, and downstream analysis can't detect it because the number is an input, not a result.

And cold prefill was the wrong target anyway

The deeper problem only shows up in the same log. Real traffic is 47–48k-token prompts that mostly hit the prompt cache:

cached_tokens: 47351   best_prefix_len: 47556   cache_hit: true

Roughly 200 uncached tokens out of 47,000. Then decode at about 30 tok/s, and a log full of cross-session preemption and yields.

So the felt latency in production is prefix-cache behavior, session scheduling, and decode. Cold prefill — the thing our benchmark measured with such care — is a path production almost never takes. We had built a precise instrument aimed at the wrong quantity, and then mis-calibrated it.

The verdict on the engine, and a retraction

For completeness: the candidate engine lost. Classic decode, same model, same weights, same box, n=64, greedy:

0.84× of the library it's built on — which is roughly what its own documentation implies if you read the comparison table carefully. Its headline "1.79× prefill" is measured against that same standard library, which is not what we run. Check what a benchmark's baseline is before converting its multiplier into an expectation. A large speedup over something slower than your incumbent is not a speedup.

Now the retraction, which belongs in the same post as the finding. Those absolute decode numbers were taken on a machine whose GPU was stuck at one-third clock, by a process launched over SSH and therefore running at nice 5. Both of those are documented failures of ours: benchmark the hardware first, and your SSH-launched benchmark runs at nice 5. On a healthy box at nice 0, the standard library does 40.6 tok/s, not 1.87.

The verdict survives — the candidate is 0.84× of the library it wraps regardless of what the box was doing, because both were measured under the same conditions. The framing does not: "16–20× behind our production engine" was an artifact of two known instrumentation bugs stacked on a wrong baseline. Relative comparisons on a degraded machine are still valid. Absolute ones are not, and we published absolute ones.

A baseline built from the three worst samples will make any improvement look real — the setup audit checks the baseline before it trusts the gains.

Key takeaways

1. Check a synthetic benchmark against the production request log on the same server before quoting it. Ours reproduced the tail of the distribution and we reported it as the median.
2. A shared machine invalidates the controlled part of your experiment. Two of our three repetitions caught foreign traffic; only one was honest, and it was still 3× off.
3. Re-derive throughput fields yourself. A server that counts cached tokens in the numerator inflates its own reported rate by 5–13%.
4. A wrong baseline yields confident, well-argued wrong conclusions, and nothing downstream can detect it. Two agents ranked options on ours.
5. Ask whether production takes the path you're benchmarking. Real traffic hit the cache on 47,351 of 47,556 tokens; we spent the session measuring cold prefill.
6. A published multiplier is only as useful as its baseline. 1.79× over a library you don't run says nothing about your system.
7. Relative comparisons survive a degraded machine; absolute ones don't. Ours didn't, and we'd already written two posts about the exact failures that caused it.