We Put the Neural Engine to Work on Prefill. It Added 19%.

Splitting each MLP's up projection between the GPU and the M3 Ultra's two Neural Engine instances took prefill from 425 to 508 tok/s with no fidelity change.

Summary

The M3 Ultra has two Neural Engine instances, and our engine now uses both as a co-processor during prompt prefill. While the GPU runs each MLP's gate projection, the Neural Engines run most of the up projection from an int8 copy of the weights. On a 27B hybrid model (4-bit, a 9.5K-token prompt in 2,048-token chunks), prefill goes from 425 tok/s on the GPU alone to 508 with the Neural Engines, +19.5%. Paired per-chunk NLL shows no measurable fidelity change. Two results surprised us. First, the best split leaves 12.5% of the up rows on the GPU, because when every row is offloaded the Neural Engines finish last and the GPU sits waiting for them. Second, the Neural Engine's throughput barely moved with procedure size or QoS class, so there was nothing to tune on that side. On mixture-of-experts models only the shared expert fits: +3.0% and +2.7% on two models, +1.4% on the third.

Why a Neural Engine can help at all

On this model, prefill time is mostly matmuls. After the attention and recurrence kernel work, GEMMs were 88% of GPU prefill time. So the question was whether anything else on the chip could take a share of the matmuls without slowing down the rest.

We probed that first through the private Neural Engine framework, with an int8 17,408×5,120 linear over 2,048 tokens:

• one instance: 31.25 ms per call;
• both instances at once, one matrix each: 31.57 ms for the pair;
• output within 3.6e-4 of a CPU matmul.

The GPU runs a 4-bit matrix of the same size at 2,048 tokens in about 15.5 ms. Working together, the two instances match the GPU's matmul rate. Then we tried it on a real layer:

The split takes as long as the slower of its two halves, and the GPU and the Neural Engines don't slow each other down.

Shapes are static, and a process gets about 120 programs

Two constraints shaped the design.

A Neural Engine program is compiled for fixed shapes. Offload runs only on chunks of exactly the bank's token count, and that count has to be a multiple of 32. At 8, 16 or 48 tokens the Neural Engine lays out its surfaces differently, and the up projection came back with a relative error of about 1. At 32 and 64 tokens it matched within 5.2e-3. A tail chunk with at least half the bank's tokens is padded to fit.

A process can hold only about 120 resident Neural Engine programs. Splitting 64 layers over two instances needs 128. So each instance gets a single program, a bank, that holds every layer as a separate procedure. Each bank is 64 up-projection halves at 8,704×5,120 int8, 2.85 GB in total. Requantizing a bank takes 2.5 s, compiling it about 4 s, and loading it 0.2 s. One undocumented detail: every request has to name its own procedure's input and output symbol indices. If you pass index 0, which belongs to procedure 0, every later procedure fails with "Program Inference error".

The probe projected +19%. The first build got +9.5%

The layer probe projected about +19% end to end, but the first integrated build measured 336 → 368 tok/s, +9.5%. The GPU profile shows where the difference went: of an ideal 4.55 s saved, only 2.46 s was realized.

• GPU idle, about 0.95 s. Each offloaded layer committed and waited on three command buffers, where the GPU-only pass has one.
• The tail chunk, about 0.75 s. Its 1,329 tokens didn't match the bank, so they stayed on the GPU.
• Surface copies, 0.49 s. Packing the input and unpacking the output.

We fixed these in three steps. The completion of the pack's command buffer now starts both Neural Engines, the gate is committed right behind it, and a merge kernel builds SwiGLU directly from the gate rows and both halves (+10.8%). The tail chunk now pads into the bank (+0.6% more). The DeltaNet layers' z projection is token-local, so it joined the banks and starts before the GPU's qkv matmul (+3.5%, 462 → 479). The GPU baselines kept improving between these steps, so each percentage is measured against its own interleaved GPU-only run.

Give it less work

Even with z on the Neural Engines, the profile showed about 2.1 s of the GPU waiting. The Neural Engines' up projection was outlasting the GPU's gate matmul by about 6.7 ms per layer-chunk. So we added a dial: a set fraction of the up rows goes to the Neural Engines, and the GPU runs the rest right behind the gate.

0.875 and 0.9375 tie within noise, and the shipped recipe uses 0.875. The goal isn't to offload as much work as possible. It's for both sides to finish at the same moment.

No knob on the Neural Engine side

A public write-up on the M4's Neural Engine reported a throughput cliff at about 32 MB per procedure, and our 44.6 MB halves were over it. Based on that, we estimated that splitting procedures or changing the QoS class could buy up to 25% of Neural Engine speed. Here's what we measured on the M3 Ultra (int8, 5,120 columns, 2,048 tokens, both instances):

QoS 33 was within noise of QoS 21, and there was no cliff. The fixed cost of each evaluate call (about 0.3 ms) only shows below 2,176 rows. The banks stay at one procedure per half projection.

Fidelity: measured, not assumed

The Neural Engine path isn't bit-identical to the GPU. Requantizing the dequantized 4-bit weights to one int8 scale per row puts the up projection 9.2e-3 (relative) away from the GPU's result. A scale per 64-column block would shrink that, but the Neural Engine compiler rejects blockwise scales on this convolution (InvalidMILProgram, macOS 26.5).

So we measured the effect instead of assuming it. On a fixed six-document corpus, paired per 512-token chunk against GPU-only prefill:

• up projection on the Neural Engines: NLL delta +0.0000 nats/token [−0.0003, +0.0003], mean KL −0.0000 [−0.0001, +0.0001], top-1 +0.0 [−0.1, +0.2];
• z on the Neural Engines versus off: NLL delta −0.0000 [−0.0004, +0.0004].

Mixture-of-experts: only the shared expert

Routed experts don't work on the Neural Engine, and the arithmetic shows why. At 2,048-token chunks an expert sees 40–48 tokens on average. One expert's matmuls over 40 tokens come to about 25 µs of Neural Engine work, against a fixed cost per request estimated at 120–230 µs. And 512 experts × 48 layers is 24,576 procedures, against about 120 programs per process and our 256-procedure bank limit, for a routed pool of about 80 GB in int8.

The shared expert is different, because every token passes through it, as with a dense MLP. Each MoE layer's shared expert becomes one procedure, with the gate rows on one instance and the up rows on the other. It fires on the layer's input before the routed expert kernels start. The down projection and the combine stay on the GPU.

For quality, paired per 512-token chunk over 18 chunks, the NLL deltas were +0.0003 [−0.0045, +0.0055], −0.0004 [−0.0061, +0.0055] and +0.0008 [−0.0035, +0.0054] nats/token, in table order. All three are neutral. The gains are small because the shared experts are small: we estimate them at 3.4% and 1.9% of prefill multiply-adds on the two sparse-attention models.

Where this leaves us

The fastest MLX-based server we compared against, oMLX, offloads the same parts of the model (the MLP up rows and the z projection). On the same 9.5K-token prompt it still leads: 451 tok/s on the GPU and 539 with its Neural Engine offload, against our 425 and 508. It's 6% ahead in both cases, so the Neural Engine adds about the same share in both engines and the remaining gap is on the GPU.

If you'd like the Neural Engine working on prefill rather than sitting idle, it's the kind of thing we configure in an on-site engagement.

Key takeaways

1. The M3 Ultra's two Neural Engine instances together match the GPU's rate on a large int8 matmul, and running them doesn't slow the GPU.
2. Aim for both sides to finish together, not for maximum offload. Putting 0.875 of the up rows on the Neural Engines beat putting all of them there by about 3%.
3. A probe's projection is an upper bound. It projected +19%, and the first build got +9.5%. Command-buffer waits, a tail chunk and surface copies ate the difference.
4. Measure the co-processor's knobs before designing around them. Procedure size and QoS made almost no difference on this chip, even though the M4 data suggested they would.
5. Static shapes drive the design. Chunks must be a multiple of 32 tokens, and with only about 120 programs per process, layers have to be packed into banks as procedures.
6. int8 per row puts the output 9.2e-3 off the GPU's, so measure fidelity paired per chunk. Ours came out neutral.
7. On MoE models, only the shared expert pays off, by +1.4% to +3.0%. For routed experts, the per-request overhead is larger than the work and the procedure count is far too high.