Our Two Machines Ran Different Models Under the Same Name

A clean A/B, two rounds, token-identical arms, +23.3% — measured on a conversion with the model's speculative decoding, sparse attention, coprocessor offload and prefix cache all switched off. The other machine's copy had them all on.

Summary

We ran a careful A/B on a 177B mixture-of-experts model: two interleaved rounds, both arms from one invocation, generated tokens compared byte for byte, arms non-overlapping in both rounds. Larger prefill chunks came out +23.3% from 3.8× fewer dispatches, and the model turned out to batch at 4 for a 2.36× aggregate after our own notes said it could not batch at all. Every one of those numbers is internally valid and none of them is this model's speed, because the copy on the benchmark machine is a different conversion from the copy on the serving machine — not drifted settings, two separate builds from two different source checkpoints. The benchmark copy has speculative decoding, the sparse-attention indexer, the Neural Engine prefill offload and the prefix cache all off. We had been quoting its throughput next to numbers from the other machine for days.

The two copies

Same model name. Same file name, near enough. A twenty-thousand-parameter difference in the totals, which is the kind of thing nobody reads.

So the +23.3% chunk-size result applies only to the stripped copy — the serving machine had already been running the larger chunk — and the headline prefill and decode figures from that A/B are a floor, not a speed.

What caused it

A converted model carries its own recipe inside the bundle. Editing the recipe in the repository does not touch a model that has already been converted, and nothing in the serving path re-reads the repository. Two people converting the same weights on two machines at two times produce two bundles, and only the bundle knows what it is.

The check that closes this: when a served model is slower than its published number, read the deployed bundle's own recipe — and check whether it is even the same conversion. Parameter count is a decent fingerprint; the recipe id is a better one.

We also found the inverse of the same problem in the serving map: a row named the speculative-decoding variant of another model while the machine had in fact been serving the mixed-precision variant by hand. The two differ in one field — attention at 8 bits versus 4 — and 4 is the setting we had measured and rejected elsewhere for costing 5.3 points of top-1 agreement. Installing the service from that row would have silently downgraded production to a setting we had already ruled out. A map that has never been reconciled against the machine is a hypothesis.

Five traps from the same afternoon, all ours

Writing this A/B produced a small collection of ways to get a confident wrong answer:

• A phase gate that tests for the absence of an error string passes when the run produces nothing at all. Test for the presence of a result.
• Filtering each arm through grep -E '^prefill:|tok/s' silently swallowed eight REFUSED lines when another campaign held the machine lock. Log the whole output; grep for display only.
• A pre-flight that greps the process table for serve cannot see a benchmark run. It reported an idle machine in the middle of somebody else's campaign.
• A 9,511-token prompt with 64 generated tokens is ~99% prefill and cannot see decode concurrency. It reported +3.9% for a 4× batch. The concurrency arm needs a short prompt and a long generation.
• A batch-1 arm built by setting the recipe's slot count to 1 is refused outright on an engine that validates it. The --batch N flag already runs its own batch-1 baseline in the same invocation, which is also how both rows end up immune to drift between runs.

What we changed

The serving map now records the parameter count and recipe id it expects, and the deploy tool reads the bundle rather than the repository. The stripped copy was left stripped rather than half-patched into a hybrid matching neither machine — a third configuration is not an improvement on two.

The real measurement, on the configured copy, still has not been taken. It needs a quiet window on the serving machine, and that machine is 11 points of free memory away from a kernel panic when it loads a 104 GB model beside what it already serves.

Related: one environment variable cost us 45× and four reasons your config edit didn't take.