We Wrote Our Own Inference Engine. It Started at 0.9 tok/s.

Six weeks building a Metal LLM runtime for Apple Silicon: the six fixes that took a 27B model from 0.9 to 41 tok/s, and the optimizations that lost.

Summary

We built a native Metal inference runtime for Apple Silicon from scratch — C++20 and Objective-C++, recipe-driven so model architectures aren't hardcoded, with a NumPy reference for parity. The first working decode on a real 27B checkpoint ran at 0.9 tok/s. It now runs at 41.2 tok/s autoregressive and 66.5 with multi-token prediction, against 41.9 for the mature framework we're competing with on the identical model and prompt. This post is the ranked list of what actually moved the number, the list of textbook optimizations that lost with measurements attached, and an honest answer to whether writing your own engine was worth it.

Update (2026-09-25): Depth 3 later won on this same 27B and its built-in MTP module. A verify kernel that runs a 3–6-row verify as one weight stream cut depth 3's pass from about 45 ms to 34.7–34.9. Depth 3 then decoded 69.5–70.1 tok/s against 58.6–60.2 at depth 1 over 256 tokens, and 72.0 against 60.3 served. Depths 4 and 5 still lose. Depth 1 with exact verify stays the default: the fast kernel isn't bit-identical, and one 1,024-token code run diverged at token 383.

The numbers

A 27B hybrid model (48 linear-attention layers, 16 grouped-query layers), 4-bit, 15.7 GB, on an M3 Ultra workstation:

The plain autoregressive step is at parity. The speculative path is the margin. Prefill is 1.7× — and that number is worth less than it looks, which we'll come back to.

What actually moved it, in impact order

The 45× from 0.9 to 41 was not one clever kernel. It was six fixes, and the largest one is embarrassing:

1. Copy the weights into Metal buffers. We were running directly off the memory-mapped checkpoint file. Unified memory makes that look free — the GPU can address those pages. It cannot keep them. The file pages were being re-faulted every single token. This was the single biggest win in the project, and it is not an optimization; it is a bug we shipped because "zero-copy on unified memory" sounded correct.

2. A dedicated 4-bit GEMV. The decode step of a quantized model is bandwidth-bound matrix-vector, and a general GEMM does it badly. Ours runs at 850–910 GB/s warm. (That "warm" is doing a lot of work — see our kernel benchmark measuring the system cache.)

3. A GPU-resident paged KV cache. Page table on device, append on device. The version before it flushed per layer, which meant a CPU-GPU round trip 64 times per token.

4. A split-K fp16 GEMV for the unquantized matrices. Those were costing 45 ms per token on their own.

5. Load-time projection fusion and fp16-output GEMVs. Fold what you can fold once, at load, rather than every token.

6. A bit-identical verify kernel. Not a speed fix — a correctness prerequisite for speculative decoding, and the thing that made #7 through #12 possible.

Per-pass time is now 26.6 ms and flat across context: 28.2 ms at 1.9K tokens, one command buffer and 770 dispatches per token.

Correctness first, or none of it counts

We kept a NumPy reference implementation of every layer alongside the Metal one, and a parity gate in CI. Greedy generation matches the reference framework token for token — 32 of 32 — with logit cosine similarity 0.994.

This mattered more than we expected, because of one specific hazard: kernels with different summation orders drift about 1e-3 in whole-model logits once fp16 casts are in the path. For ordinary decode that's invisible. For speculative decoding it is fatal, because the verify pass has to agree with the decode pass exactly — if it doesn't, you reject correct drafts at random and your acceptance rate silently craters. Our verify kernel is checked against the decode kernel with memcmp, not with a tolerance. Anyone optimizing one without the other breaks the test, which is the point.

Things that lost, with numbers

Publishing the failures is most of the value here. Each of these is a standard, sensible optimization that measured worse:

• Split-K GEMV for the quantized path. Slower.
• Software-prefetch GEMV. The extra registers cost more occupancy than the prefetch bought.
• 8 rows per simdgroup. We ended up at one row per simdgroup for the main decode GEMV and two for the multi-token variant — the opposite direction from where we started, worth +7%.
• Fusing SwiGLU and normalization into the GEMV prologue. Saves a 5 µs dispatch, costs far more in L2 re-reads per threadgroup.
• Multi-token prediction at depth greater than 1 (on the 8-bit-attention mixed build, hence 55.3 rather than the 66.5 above). 55.3 → 43.8 → 41.4 tok/s for depths 1, 2, 3. The chain already runs in one command buffer; the cost is a 3–4-row verify that streams the weights about 1.8 times a pass, and acceptance falls 85% → 76% → 51%. We went in believing the verify kernel was the constraint. It wasn't, so the whole premise of the research scan was wrong. Depth 1 is optimal for this draft head; the only remaining lever is draft quality.
• Whole-head fused attention kernels. They lose as soon as you process more than one token.

The pattern across all of these: on a wide chip, occupancy and cache behavior beat dispatch count almost every time. Fusion is not free, and "fewer kernel launches" is not a goal.

One hardware finding worth the whole post

We tried to hand off between threadgroups inside a single dispatch — a last-arrival pattern with device atomics, so a later stage can consume an earlier stage's output without a second launch.

It produces garbage tokens on the M3 Ultra and byte-identical output on the laptop GPU.

There is no ordering guarantee for that across threadgroups, and the narrower chip happens to schedule in a way that hides it. If you develop on a laptop and deploy on a workstation, this class of bug will find you. Don't rely on cross-threadgroup handoff inside one dispatch, and don't trust a passing test on the smaller machine.

Quantization, measured rather than assumed

Once you have your own converter you can also have a real precision dial, and measure it instead of arguing about it. Against bf16, on the same model:

The mixed build is the finding: six times closer to bf16 than the 4-bit build, for 15% of the decode speed. Attention is where the precision matters; the MLP tolerates 4 bits fine. If you only ever ship one uniform bit width you never see that.

The recipe rule

Early on, every model needed a hand-written YAML recipe. That is a treadmill, and the owner said so directly: nobody wants to hand-write a recipe per model. So the converter now derives the recipe from the checkpoint's own config.json and ships it in the model directory. Adding an architecture means adding a derivation, once. Hand-written recipes are for tuning experiments only.

That decision is why bringing up the second architecture — a sparse-attention MoE model — took days rather than weeks.

Was it worth writing?

Honestly: parity with a mature framework is not, by itself, a business case. Six weeks to match something you can pip install is not a win.

What it bought is everything parity enables:

• Speculative decoding that actually ships, worth 1.6× — because we control the verify kernel and can make it bit-identical.
• A precision dial with measured quality, not a choice between two published quants.
• A place to put research. Every idea in the "things that lost" list above was testable in an afternoon.

And one caution about the prefill number. 318 tok/s is 1.7× the reference — but with no cross-request prefix cache, a 25,000-token agent prompt is still around 100 seconds to first token. For agent workloads, the lever is caching across requests, not the GEMM. We spent time on the GEMM first because the GEMM is the fun part. The same mistake, from the other direction.

Key takeaways

1. The biggest win was a bug, not an optimization. Memory-mapped weights on unified memory look free and get re-faulted every token. Copy into device buffers.
2. Fewer dispatches is not a goal. Five separate fusions lost on a wide chip; occupancy and L2 behavior dominate launch overhead.
3. Bit-identical, not approximately equal. Speculative decoding requires the verify path to match the decode path exactly; fp16 casts turn a different summation order into a 1e-3 logit drift and a collapsed acceptance rate.
4. Cross-threadgroup handoff inside one dispatch is not guaranteed — and it can be byte-perfect on a narrow chip while producing garbage on a wide one.
5. Speculative depth greater than 1 lost, badly. Acceptance falls faster than the extra positions pay, and a wider verify streamed the weights nearly twice a pass.
6. Derive model recipes from the checkpoint config. Hand-written per-model config is a treadmill you will still be on a year later.
7. Mixed precision beat uniform precision — 8-bit attention with a 4-bit MLP was 6× closer to bf16 for 15% of the speed. You only find that with a converter you own.

Update, 2026-09-12

Six days on, prefill is the part of this post that has changed most. The engine now runs three mixture-of-experts models alongside the 27B, and prompt processing is well past the numbers above. These are from the same M3 Ultra with the shipped recipes, on a 9.5K-token prompt unless noted:

• This isn't the same benchmark as the 318 tok/s above. That figure came from a 1,327-token prompt in 128-token chunks. Prefill speed depends on both prompt length and chunk size, so read the two as a trend, not a ratio.
• The reference framework wasn't the fastest thing on the machine. On the same 9.5K-token prompt, the fastest MLX-based server we compared against, oMLX, prefills the 27B at 451 tok/s on the GPU and 539 with its Neural Engine offload. We trail it by 6% on both. Two of the changes that got us here have their own posts: 17 GB of attention buffers nobody read and putting the Neural Engine to work on prefill.
• The prefix-cache lever from the last section has shipped. Requests that share a prefix can now reuse it through a cross-request prefix trie that persists to SSD. It's off by default.