We Evaluated a Faster Inference Engine and Rejected It

A Rust engine that beats our incumbent on prefill by 4-8% and loses decode by a third — plus the benchmark table everyone quotes, measured on swapping hardware.

Summary

We spent a day evaluating an actively developed Rust inference engine as a replacement for our production server on an M3 Ultra. It wins cold prefill by 4–8%, loses decode by 30–40%, and its prefix cache does not hit on identical prompts. The verdict is no, for our workload — and the reasoning generalizes, because the decode gap traces to one architectural difference we could have predicted from reading the source. Along the way: a widely-quoted benchmark table that turns out to measure swap, a device gate that panics on our hardware, and a one-line patch that builds the whole thing on the public SDK.

The benchmark everyone quotes

The engine's README once carried this table. It has since been deleted, but it's what circulates:

A 10× speedup on some rows. Read the columns against each other and a different picture appears.

Where models fit comfortably in RAM — the 1B rows — the difference is 5 to 10 percent. The 10× rows are exactly where the comparison engine posts 1.08 tok/s on a 4B model. That's a base M2 swapping on unquantized bf16 weights, which the authors themselves note.

Base M2, models at 4B and below, bf16, against one competitor. None of that is our regime: an M3 Ultra with 256 GB, a 27B model, 8-bit quantization.

The honest reading is that the engine is somewhat faster than llama.cpp when everything fits, and that the dramatic rows measure memory pressure on a laptop. (The unified-memory version of this trap, with the arithmetic: RAM vs VRAM for local LLMs.) To their credit, they deleted the table. We couldn't find published benchmarks anywhere else.

This is the most common shape of misleading benchmark we run into: the numbers are real, the methodology is disclosed, and the headline ratio comes entirely from a regime nobody runs in. Read across the columns for the row where both systems are healthy. That row is the real result.

Getting it to run at all

The released binary panics on our hardware before loading anything:

unsupported Metal device "Apple M3 Ultra"; add its DeviceIdentity mapping

A documented tier-override environment variable does not bypass it — separate gate.

The device profile system had been deleted from main two days earlier, so building from source was the path. Four blockers, each worth knowing:

The engine ahead-of-time compiles 64 Metal shaders, so it needs full Xcode rather than Command Line Tools. Worth noting: the build directory is a directory named build/, not a build.rs script, so searching for build.rs finds nothing and you conclude there's no build step.

On macOS 26 the Metal compiler ships separately — xcodebuild -downloadComponent MetalToolchain, 704 MB.

main needs a Metal SDK newer than public. The 4-bit tensor formats it references don't exist in the public SDK, whose matrix-multiply primitive accepts only int8, float, half, and bfloat. They're building against an Xcode beta.

The fix is one line — alias the 4-bit format to int8_t — plus a small if constexpr gate around a function body that would otherwise instantiate on an impossible type. This is safe for a specific reason worth checking before you patch anything: the matrix unit in question is M5-only hardware. The engine gates on a runtime capability check that returns false on M3, every affected kernel carries a non-matrix-unit variant, and the routing layer filters those paths and falls back by GPU family. We were patching a branch that can never execute on this chip.

Verified on the M3 path: the patched build runs at 388.2 tok/s prefill and 21.1 tok/s decode against the official prebuilt's 390.5 and 21.1. Zero cost, as predicted. A public-SDK build is a legitimate substitute if a future release regresses.

And then a same-day release made all of it unnecessary. Version 0.5.23 was tagged while the build was running, with a prebuilt binary that clears the device gate. Roughly 90 minutes spent on a build that a release published that afternoon obviated. Check the releases page before building from source — especially on a project shipping this fast.

The result

Same box, same input bytes, box exclusive to one engine at a time. (This is the second running of this evaluation — the first day's numbers came off a machine wedged at one-third clock, which is why the protocol now starts with a hardware probe.)

It wins prefill by a few percent and loses decode by a third.

We predicted the decode loss before measuring it, from reading the source. Our incumbent uses the checkpoint's jointly-trained multi-token-prediction head, which yields 2.10 accepted tokens per round across ~17,800 production requests. The new engine has no native MTP support — its speculators are draft-model only. To tie, it would have to be more than twice as fast per raw token.

It isn't. The measured gap is almost exactly what that arithmetic predicts.

That's the transferable part. Before benchmarking an engine swap, check whether it supports the one feature your current stack gets its speed from. Ten minutes in the source told us the answer, and the measurement confirmed it rather than discovering it.

The prefix cache doesn't hit

This is the finding that settled it.

Identical 26,015-token prompt, sent a third time, non-streaming: TTFT 66.6 seconds, cached_tokens: 0. A full re-prefill, equal to the cold time.

Streaming the identical prompt was worse than cold — about 200 seconds to first token. The cache lookup appears to cost something and save nothing.

Reading the documentation more carefully: the prefix cache only helps when the new prompt extends the previous state. Identical resends miss.

For our workload that's decisive. Agent harnesses re-send a growing conversation constantly. Our incumbent's session bank and SSD cache can skip prefill entirely on a hit — when the prompt doesn't invalidate it at token 1,067. The new engine's +4–8% prefill advantage applies to every request, while the incumbent can skip prefill on some of them. At even a 10% cache hit rate the incumbent's effective prefill cost is already lower.

A percentage win on an operation you can sometimes skip entirely is worth less than it looks.

Doing the crossover arithmetic properly

Rather than argue from the two rates, we computed where they cross, then checked our actual traffic against it.

The new engine saves ~4.1 s on a 26k prefill and costs ~15.1 ms per output token. Crossover ≈ 271 output tokens.

Then, real production replies, prompts ≥20k, n=1,400: p25 123 / p50 201 / p75 421 / p90 881 / mean 565 tokens.

So: 62% of requests would be faster on the new engine, 38% on the incumbent. At the median reply the difference is 1.1 seconds — a wash. But the mean reply of 565 tokens is far above the crossover, so total daily compute time favors the incumbent. The new engine only helps the short-reply tail.

A verdict of "faster on 62% of requests and slower overall" is only visible if you compute the crossover and weight it by your own distribution. Either rate alone tells you the wrong thing.

One trap in producing that distribution: the un-rotated request log after a reboot was mostly our own benchmark traffic — its p50 of 64 was literally our max_tokens setting. Use the rotated files or a time cutoff for anything you'll call a production statistic.

Six measurement traps, all real

Each of these cost us a run or produced a wrong number:

Slice bytes, not characters. Our reference prompt file was 400,000 bytes but 399,369 characters. The incumbent had been measured on the first 100,000 bytes; the new harness took the first 100,000 characters — a different prefix, different hash, invalid comparison.

Select log records by content, not position. tail -1 on the request log returned a stray 11-token probe that landed after our request. The first "control" reading was garbage. Select by new_prefill_tokens > 20000.

Their SSE format has no space after data:. A parser requiring "data: " sees nothing and reports ttft=None.

The first streaming chunk arrives before prefill. It's an empty role delta. Any TTFT that fires on "first chunk" reads about 0.1 s on a cold 26k prompt. Fire on the first actual content piece.

Weights are mmapped lazily. /v1/models returns 200 with GPU memory at 1.0 GB; the first real request maps them and memory jumps to 29.4 GB. A warmup request before timing is mandatory or your first measurement includes a 28 GB load.

Thinking is streamed as a separate field. Count both content and reasoning deltas or decode throughput is undercounted. Relatedly, at max_tokens=64 our incumbent returned empty content because all 64 tokens went to reasoning — give any quality probe ≥600 tokens or disable thinking on both engines.

And two process-management ones that produced false "the process died" alarms:

pgrep -f <script> self-matches. A waiting loop whose own command line mentions the script counts as a running instance. Use a bracket pattern: [b]ench_uzu\.py.

python3 on these machines is a framework path, so pgrep -f "python3 [^ ]*script.py" never matches — argv[0] isn't python3. Match the script name with a bracket pattern and don't assume the interpreter path.

Verdict, and what would change it

No, for this workload. Our incumbent wins on decode, wins on effective prefill once caching is counted, and wins on total daily compute against our actual reply-length distribution.

What would flip it: native MTP support closing the decode gap, or a prefix cache that hits on identical resends. Either one alone might do it. Both are plausible for a project moving this fast — hence writing down the specific numbers rather than just the verdict.

We'd re-run against: prefill ~360–377 tok/s, decode ~28–34 tok/s at 26k context, 8-bit, M3 Ultra. That's the bar.

Key takeaways

1. Read benchmark tables across the columns. A 10× row where the loser posts 1.08 tok/s is measuring swap. The rows where both systems are healthy show 5–10%.
2. Check for your key feature in the source before benchmarking. Ten minutes told us the decode gap was coming; the measurement only confirmed the size.
3. Check the releases page before building from source. Ninety minutes lost to a build that a same-day release made unnecessary.
4. A cache that misses on identical prompts changes the whole calculus. A percentage win on an operation you can sometimes skip is worth less than it looks.
5. Compute the crossover, then weight by your own distribution. "Faster on 62% of requests, slower in total" is invisible from either rate alone.
6. Verify a patch is on a dead branch before trusting it's free. Ours gated M5-only hardware, which is why it cost exactly nothing on M3 — and we checked that rather than assuming it.
7. Warm up before timing anything that mmaps weights. The first request pays a 28 GB load that has nothing to do with inference.