Benchmark the Hardware Before You Benchmark the Software

We spent a day evaluating a prefill engine on a machine at one-third clock speed. A power meter reporting zero watts under load cracked it.

Summary

A complaint that "prefill is slow" on one of our two identical Mac Studios turned out to have nothing to do with the inference server, the model, the flags, or the prompt. The machine's SoC was wedged at roughly one-third clock — a rock-steady 3.3× compute deficit with completely normal memory bandwidth. A warm reboot fixed it and prefill went from 118 to 377 tok/s. The tell was power telemetry reporting 0 MHz and 0 mW while the GPU was demonstrably sustaining 7 TFLOP/s.

The situation

Two Mac Studios, identical on paper and, as far as we could measure, identical in fact: M3 Ultra, 80 GPU cores, 256 GB, same macOS build, same MLX version, same Python, same management agents. One of them felt slow.

"Feels slow" was, again, a real signal with the mechanism removed.

The measurement that settled it took ten minutes — a bare quantized matmul with no inference server involved at all. Identical script, identical shapes, forced evaluation each iteration, median of ten:

And end to end, identical 25,970-token prompt, same model, same server version:

Having a twin machine is the single most valuable diagnostic asset here. Without a healthy reference, 118 tok/s is just a number — you'd tune the server for a week.

(If you're speccing a box like these, our hardware notes are here: what I'd actually deploy.)

Cleaning the box changed the diagnosis

The first numbers were partly self-inflicted. Our own test server was still resident, holding a 96 GB session bank and crushing both bandwidth and dispatch latency.

After killing every process we'd started:

The real deficit is a rock-steady 3.3× on compute only, with zero jitter. Memory bandwidth is identical to the healthy twin. Quantized and unquantized are equally affected, so it isn't kernel selection.

The cleanup mattered. The dirty numbers said "erratic, bandwidth-starved, 4.7× slow" and pointed at memory. The clean numbers said "steady, bandwidth-normal, 3.3× slow" and pointed at clocks. Those are different investigations.

What we ruled out, and how

Worth listing because each is a plausible story that had to die:

• A competing GPU job — no training or transcription process on the box. (This had been the diagnosis for a different problem on the same machine, which is exactly why we checked.)
• Memory — 57 GB free, 147 GB inactive, zero swap.
• Thermal — no recorded thermal warning. Weak evidence on Apple Silicon, where that reading is unreliable, but combined with the steadiness it's strong: thermal throttling ramps and oscillates. This didn't move at all.
• The inference server — the deficit reproduces in a bare matmul with nothing else loaded.
• Process priority — the server was confirmed at nice 0 (see the nice-5 post, where it wasn't, and that cost us a different session).

Everything measurable was identical between the two machines. Same OS build, same framework versions, same power mode, same AC power, same endpoint-management agents — the healthy machine actually had more of them.

The diagnostic that cracked it

GPU HW active frequency: 0 MHz
GPU Power: 0 mW
GPU idle residency: 100%

Three consecutive samples, while MLX was demonstrably sustaining 7.0 TFLOP/s on that GPU.

That's not a slow reading. It's an impossible one. Real GPU work with dead telemetry means the thing reporting on the power state has lost track of the power state — which identifies a wedged power-management subsystem rather than thermal, contention, or software.

One check before trusting it: make sure the work isn't silently falling back to CPU. On this box GPU measured 7.23 TFLOP/s against CPU at 3.10, so the GPU really was running, just slow. Without that check, "GPU idle 100%" could have meant the GPU genuinely was idle and everything had moved to CPU — a completely different bug.

The CPU was also about 2.4× slow, so this was SoC-wide, not GPU-specific.

The fix, and the worry that turned out to be wrong

Our assumption was that a wedged Apple Silicon power state needs a full shutdown and cold power-on to clear — which would have meant someone physically at a headless machine in another location.

Wrong. shutdown -r now and a disk-unlock over SSH did it.

3.2× recovery, and both system daemons came back on their own.

We'd written up "this needs physical access" as a blocker before testing the cheaper option. Try the reversible thing first, even when you have a theory saying it won't work.

The signature, for next time

If you see all of these together, it's a wedged clock governor and not anything you're going to fix in software:

• Compute down ~3×, perfectly steady, zero jitter (thermal throttling ramps; this doesn't)
• Memory bandwidth completely normal — the memory controller is independent of the wedged governor
• CPU also slow, so it's SoC-wide rather than GPU-only
• No GPU faults in system logs, low-power mode off, no competing process, no swap
• Power telemetry reporting all zeros under demonstrable load — the smoking gun

The free health check we'd been sitting on

Our MLX server runs a warmup ladder on every start — a 512-token then 2560-token prompt, logged distinctively. It had been recording the machine's health, automatically and historically, the entire time:

The degradation was visible in a log file hours before anyone noticed the machine felt slow, and it brackets onset to within a day.

~350 tok/s at 2560 tokens = healthy; under 150 = the box is sick. That's now our standard check, and it costs nothing because the server already does it.

Most systems have something like this — a startup self-test, a warmup pass, a periodic probe — whose output is already being logged and never read. Finding one and giving it a threshold is the cheapest monitoring you will ever build.

The actual cost of not doing this first

A full day of prefill-engine evaluation ran on a machine at one-third speed. Two harness bugs were investigated and fixed during that day; both turned out to be trivial next to the hardware problem, and neither would have mattered if we'd spent ten minutes on a matmul probe first.

The evaluation had to be redone. The conclusions from it were worthless — not wrong in an interesting way, just measured on a machine that wasn't the machine we thought we were measuring.

Benchmark the hardware against a known-good reference before you benchmark software on it. On a fleet with identical machines that's a ten-minute check. Without a twin, keep a historical roofline number for each box and compare against that.

If you'd rather not run this diagnostic yourself, on-site local AI setup and tuning is the version of this I do for hire.

Key takeaways

1. A twin machine is the best diagnostic tool you have. 118 tok/s means nothing alone and everything next to 377.
2. Kill your own processes before measuring. Ours changed the diagnosis from "erratic and bandwidth-starved" to "steady and clock-limited" — two different investigations.
3. Steady vs. jittery separates clock-wedging from thermal throttling. Throttling ramps. A wedged governor sits flat.
4. Telemetry reporting impossible values is itself the finding. 0 MHz under demonstrable load named the subsystem.
5. Rule out silent CPU fallback before trusting "GPU idle." One comparison probe, and it changes what the reading means.
6. Try the reversible fix before accepting a blocker. We'd written off a warm reboot on a theory. It worked.
7. Find the self-test your system already runs and give it a threshold. Ours had been logging the machine's decline for two days.