We indexed 45 local-inference benchmark results — the 4.7x gap nobody looks for

45 measured local-inference figures, transcribed verbatim with sources, whose headline is that two identical Mac Studios differed by 4.7x — and the cause was invisible until you measured it.

We run local inference on a small fleet of machines, and for a while every number we produced lived in a chat log or a terminal scrollback and then vanished. That was the mistake. Last week we stopped doing that and wrote everything down in one place: a benchmark index of 45 measured figures, each one transcribed verbatim from its source with the machine, the date, and the command that produced it. The headline finding, the one worth the whole exercise, is that two Mac Studio M3 Ultra units with the same model spec, the same chip, and the same memory were not the same machine at all. On an 8-bit quantized matmul, one sustained 22.91 TFLOP/s and the other sustained 4.83 TFLOP/s. A 4.7x gap between two boxes that are, on paper, identical.

The gap was not subtle in end-to-end terms. On a 25,970-token prompt, the healthy unit prefilled at 373.7 tokens per second for a time-to-first-token of 69.5 seconds; the degraded unit prefilled at 118.2 tokens per second for a TTFT of 219.8 seconds. Same weights, same prompt, same serving stack. Nothing about the machine would have told you which one you were on until you measured it.

What it was not

The obvious explanations were ruled out one by one. It was not memory pressure: the degraded box reported 57 GB free and 147 GB inactive with zero swap, and its memory bandwidth was fine — clean reads came back at 535 GB/s, and a twin measured 538 GB/s. It was not thermal throttling. It was not a competing job. It was not the server or the scheduler or the quantization. The tell was in how the numbers moved. Read bandwidth on the sick unit was erratic — 94 GB/s, dirty and jittery — where the healthy unit's was steady. And the compute deficit was steady too: a consistent 3.3x shortfall across the SoC, with the CPU running about 2.4x slow, not a dip you could blame on a burst of load.

The smoking gun was the power telemetry. Under a sustained 7.0 TFLOP/s matmul load, the degraded unit reported the GPU at 100% idle, with 0 MHz and 0 mW of draw. A box that is measurably doing work while reporting its GPU asleep is a box whose clock governor has wedged. Nothing in the OS was going to notice or fix that on its own, because the OS still believed the hardware was idle.

The fix and the recovery

A warm reboot did not fix it. A full shutdown — shutdown -r now, enough to reset the system management controller — did. Before the reboot the degraded unit managed 7.23 TFLOP/s on an fp16 matmul; after, 22.61. Its twin, measured the same day for reference, scored 22.60. Prefill jumped from 118.2 to 377.2 tokens per second, against the twin's 373.7. Clean read bandwidth rose from 535 to 597 GB/s, and the small 1024-cubed matmul went from 2.81 to 6.10 TFLOP/s. The recovery was a 3.2x jump in real throughput, which is to say the machine had been running at roughly a third of its capability and nobody had noticed, because the work still got done.

The free health check we were sitting on

Here is the part that stings, and it is why the index exists. We had a warmup ladder in the logs the whole time: a short-prompt prefill measurement run on a schedule. The healthy range is about 350 tokens per second at 2560 tokens; under 150 means something is wrong. On the 31st the box read 359.3 and 370.1 — healthy. On the 2nd at 17:21 it read 122.8 and 107.3, and by 20:16 it was 35.9 and 28.5. The machine had been degrading for hours, visibly, in a measurement we were already taking, and the anomaly only got connected to the hardware after the manual investigation. The lesson is not that you need exotic tooling. It is that the cheapest, most boring measurement — the same small prompt, at the same time, every day, compared against the same machine's own history — is worth more than a one-off, state-of-the-art benchmark you run once and never look at again. The roofline you need is not the spec sheet; it is yesterday's number.

The other traps in the index

The 4.7x gap is the headline, but the rest of the 45 rows earn their place. A few of the ones that keep recurring:

Your SSH-launched benchmark is running at nice 5. A hand-launched inference server inherited a nice level of 5 from its parent shell, while the production service, started by launchd, ran at nice 0 with priority 4. The hand-launched server prefilled at 28.5 tokens per second on a 2560-token prompt where production managed about 320. If you benchmark through a terminal session and conclude the machine is slow, check ps -o pri,ni first.

The kernel benchmark measured the cache, not the chip. A naive matmul microbenchmark reported 850–910 GB/s when the working set fit in cache and about 600 GB/s when it did not — a 40% overstatement that had nothing to do with real memory bandwidth. Keeping the working set honest changed the number by the same margin as the hardware defect it was trying to find.

The benchmark said 18%, production said 0.07%. A throughput metric counted completions that hit the token limit as successes, producing an 18.3% "empty output" rate where the real, semantics-correct figure was about 0.07% — one in 1,520. The difference was a max-tokens setting, not a model quality issue.

The unifying theme across all of them is that most benchmark numbers are measuring something other than what you think they are — the cache, the shell's priority, the token limit, the clock governor. Writing the number down, with its source and its exact command, is the only way to catch that drift later.

How to detect a slow twin

Pulling it together, the checklist that would have caught the 4.7x gap in an afternoon:

1. Kill your own processes first. Re-run the measurement from a clean launchd or systemd service, not a terminal, and check ps -o pri,ni. A priority artifact looks identical to a hardware defect and is far more common.
2. Compare against a twin or against the machine's own history. Two boxes on the same job, or the same box against its own warmup ladder. The spec sheet will not tell you anything; a roofline will not tell you anything; a second measurement will.
3. Look at the shape, not just the average. Steady 3x shortfalls are hardware or firmware. Jittery, dirty, erratic numbers are contention or cache. The two have different fixes.
4. Read the power telemetry under load. A GPU reporting 100% idle at 0 MHz while sustaining 7 TFLOP/s is a wedged governor, and it will not resolve with a warm reboot.
5. Benchmark what production runs, not what the benchmark harness runs. The 18%-versus-0.07% case and the cache case both came from measuring an idealized workload that production never actually issued.

The reason this matters more than most tuning work is that it is invisible. A local model that runs at a third of its real speed does not fail; it just answers slower, and the staff concludes the machine is underpowered rather than sick, and someone buys a bigger box to solve a firmware problem. That is the 4.7x gap in one sentence: identical hardware, dramatically different machines, and the only instrument that catches it is a number written down yesterday.

Notes: RobotNurse is a New York City business that installs and maintains private, on-site AI systems; the benchmark index and the field notes this piece draws on are public at https://robotnurse.io/.