Our Kernel Benchmark Was Measuring the System Cache

A microbenchmark reusing one weight matrix reported 850 GB/s; cycling distinct matrices reported 600. Every tuning call from the first number pointed the wrong way.

Summary

Our 4-bit matrix-vector kernel benchmarked at 850–910 GB/s. The real figure under production conditions is about 600 GB/s. The difference is that the benchmark allocated one weight matrix and hammered it, so after the first iteration it was resident in the chip's system-level cache and we were measuring cache bandwidth. Retuning against a cold benchmark reversed one of our configuration choices and produced +7% decode throughput, bit-identical output, from a one-line constant change.

What the bench did wrong

The original harness did the obvious thing: allocate the weights for one layer shape, run the kernel in a loop, take the best of five after a warm-up.

Every part of that is standard practice, and the warm-up is exactly what makes it wrong. Apple Silicon has a large system-level cache shared across the chip. A single quantized weight matrix for one projection fits in it comfortably. So iteration one loads from memory and iterations two through five load from cache, and best-of-five reports the cache.

Production decode does not look like that. It streams every weight in the model once per token and comes back to none of them until the next token, by which point the whole 15.7 GB has evicted anything you cared about.

The fix

The replacement benchmark cycles through 768 MB of distinct matrices per shape, round-robin, so nothing survives to the next iteration. That is enough to blow past the system-level cache on this chip. It's tagged separately ([cold]) and it's the one we tune against now; the warm number is still useful for spotting a kernel regression in CI, where relative change is all that matters.

Why it mattered, concretely

We had tuned the kernel's threadgroup layout — how many output rows each simdgroup is responsible for — against the warm number, and landed on 4 rows.

At 850 GB/s the kernel looks close to bandwidth-saturated, so the sensible move is to give each simdgroup more work and amortize the launch. At 600 GB/s it isn't saturated, and the right move is the opposite: fewer rows per simdgroup, more simdgroups, more memory-level parallelism in flight.

Retuned cold, we went to 1 row per simdgroup for the main decode kernel and 2 for the two-token variant. That is +7% on the decode step and +7% on the speculative pass, with byte-identical output. Both are single constants.

The warm benchmark wasn't just imprecise. It pointed in the wrong direction, and it did so consistently, because a cache-resident kernel always looks more bandwidth-bound than it is.

Two neighboring traps on the same chip

The synchronous kernel profiler lies about absolute time. Ours has roughly a 200 µs floor per measurement from the synchronization it inserts, and forcing a sync between dispatches drops the GPU clocks. It's fine for ranking kernels against each other and useless for "this kernel takes 40 µs." Use GPU counters for absolute numbers, and never let a profiler's own overhead into a budget.

Absolute throughput on a shared box drifts ±20–30% depending on what else is resident. Ours also hosts a couple of long-lived model servers. Any A/B comparison has to be interleaved back-to-back in one chain — A, B, A — and never compared against a number written down yesterday.

The general rule

Before trusting a microbenchmark, check whether its working set fits in cache. If it does, you are benchmarking the cache.

The tell is easy: multiply your buffer size by the iteration count and ask whether the hardware could plausibly have kept it. If the answer is yes, size the benchmark up until the answer is no, and re-run. Then check whether any tuning decision you made on the old number still holds — some will, and the ones that don't tend to be the layout and blocking choices, because those are precisely the ones that trade bandwidth against parallelism.

The warm-up loop that everyone adds to remove noise is the same loop that installs your data in cache. It is doing two things and only one of them is the one you wanted.

Key takeaways

1. A microbenchmark that reuses one buffer measures cache bandwidth, not memory bandwidth. Ours was off by 40%.
2. Cycle a working set larger than the last-level or system cache. We use 768 MB of distinct matrices per shape.
3. The error has a direction. A cache-warm kernel looks bandwidth-saturated, which pushes you toward more work per thread — exactly backwards.
4. Re-derive tuning constants after fixing a benchmark. One reversed constant was worth 7%, bit-identical.
5. Synchronizing profilers report a floor, not a time. Rank with them; budget with hardware counters.
6. Interleave A/B runs on any shared machine. Absolute throughput moved ±20–30% with unrelated resident processes.