Five Kernel Ideas That Measured Neutral or Worse

An exact split-K tile, a cache-friendly tile order, resized expert tiles and a fast verify kernel, each with the number that killed it.

Summary

We keep a ledger of kernel ideas that didn't pay on the 256 GB M3 Ultra we tune our engine on, each with the number that killed it, so nobody queues them twice. Five came from the last week. An exact split-K tile cost 1% of prefill, and its 64 MB of scratch pushed the largest model into a GPU out-of-memory. A band-swizzled tile order changed nothing. Tile rules tuned at 128-token chunks were about 7% slow at 2,048. Resizing the expert tile lost in both directions. And a fast verify kernel for speculative decoding measured exactly neutral at 4 rows, with identical tokens and identical timing. It was neutral because the engine never runs the fast kernel at 4 rows, a rule already written in a comment beside the dispatch branch. We queued the experiment anyway.

Update (2026-09-25): The fix this pointed at arrived two days later. A verify kernel that does take 3–6-row verifies, as one weight stream, made depth 3 win on the same 4-bit 27B: 69.5–70.1 tok/s against 58.6–60.2 at depth 1 over 256 greedy tokens with the confidence gate off, and 72.0 against 60.3 served. Depth 1 with exact verify is still the shipped default: the fast kernel isn't bit-identical, and one 1,024-token code run diverged at token 383.

How these were measured

Unless noted, each A/B is two interleaved rounds on the same box, reported as "round 1 / round 2", with generated tokens compared between arms. On a shared machine only back-to-back pairs count (here's why). The models are Qwen3.8-27B (a hybrid), Qwen3.8-Flash-Next (a mixture-of-experts model) and GLM-5.3-Flash, which pins 185 GB of weights.

1. An exact split-K tile: −1%, and a GPU out-of-memory

Split-K cuts a matrix's input dimension into partials so more of the multiply runs in parallel, then reduces the partials in a fixed order. We tried an exact split-K tile in place of the chunked GEMV in Flash-Next's prefill:

On a laptop's 10-core GPU the split had come out ahead, but only when the GEMV left the GPU under-filled. On the workstation, the GEMV's own threadgroups already fill all 80 cores, so the partials had no idle capacity to use. (A later fp16 split-K tile did win, by 3.1%. It's this exact variant that's dead.)

The second number mattered more. That build's 64 MB of scratch pushed GLM-5.3-Flash into a GPU out-of-memory at 512-token chunks. The ceiling there isn't RAM. It's Metal's GPU working-set limit, which defaults to 75% of the machine no matter how much memory is free. The model's 185 GB of pinned weights, its per-sequence state and its activation scratch left so thin a margin under that limit that 64 MB was enough to go over. (More on that limit, and on getting its units right.)

2. A band-swizzled tile order: ±0

The prefill GEMM loops weight-row tiles against token tiles. The idea was to walk the weights in bands of 8 row tiles across all the token tiles, so each band's weights stay in cache while every token tile uses them. On the laptop it won at mid-size shapes. On the workstation it was neutral: 425.0 / 425.1 tok/s against 424.9 / 425.2 on the 27B (GPU only), and 970 / 974 against 968 / 971 on Flash-Next.

On the M3 Ultra this tile is bound by the simdgroup multiply-accumulate itself, not by weight traffic, so better cache locality has nothing to win back.

3. Tile rules tuned at 128-token chunks: ~7% slow at 2,048

We swept the prefill tile shapes, and the rule for which matrix gets which tile, at 128- and 256-token chunks. That sweep had its own dead ends:

• A 64×64 tile was never the best at 128 tokens.
• 256-token chunks changed nothing per token.
• One 64-row layout spilled its accumulator array and collapsed to 0.8 TFLOP/s, against 18 for the kernel it was meant to improve.

The rule that survived gave the tall tile only to matrices above 16,384 rows. Then the shipped recipes moved to 2,048-token chunks, where the tall tile is faster on every shape (4-bit, ms per call):

The old rule sent every shape except w13 to the wide tile, about 7% slower. With the rule re-derived by chunk size and matrix height, 27B prefill went from 405.9 / 406.1 to 415.9 / 416.1 tok/s GPU-only, and from 478.5 / 477.9 to 491.6 / 492.7 with the Neural Engines. What lost here wasn't a kernel. It was the assumption that a rule tuned at one chunk size still holds at another.

4. Resizing the expert tile, in both directions

MoE prefill packs each expert's tokens into 32-slot tiles. Both obvious tweaks lost:

• 64 slots, on Flash-Next (one dequantization pass per expert instead of two): 949.4 / 951.9 → 914.6 / 914.7 tok/s, −3.9%. The larger tile's occupancy cost more than the saved pass.
• 16 slots, on GLM-5.3-Flash (about 14 tokens per expert at 512-token chunks): 326.6 / 327.5 → 323.3 / 323.1. The 32-slot tile already skips its empty 8-slot blocks, so the smaller tile saved little and paid more for per-tile dequantization.

The 16-slot entry came with a pointer: the lever is more tokens per chunk. It was right. Once GLM's scratch fit at 2,048-token chunks (17 GB of attention buffers nobody read), it got 57 tokens per expert instead of 14, and prefill went from 327.0 to 399.8 / 399.1 tok/s, a 22% gain.

5. A fast verify kernel that measured exactly neutral

Speculative decoding checks drafted tokens in one verify pass over several rows. That pass can use an exact multi-token GEMV, bit-identical to the decode kernel, or a fast simdgroup-matrix GEMM (decode.verify_kernel: fast) that is 25–40% faster in isolation at 8 and 16 rows.

The question this week was whether the fast kernel makes it worth running multi-token prediction at depth 3, gated at confidence threshold 4 so a verify is at most 4 rows. The setup was Qwen3.8-27B at 4-bit, 256 greedy tokens, one prose prompt and one code prompt, two interleaved rounds each. Depth 1 took 140 passes at 29.9–30.1 ms each, with 83% of drafts accepted. Depth 3:

Depth 3 is 3–5% faster than depth 1 on code and about 3.5% slower on prose, so depth 1 stays the default. The verify kernel changed nothing: the tokens were identical in all four comparisons, and the timings were within a tenth of a millisecond.

That was the tell. The fast kernel isn't bit-identical to the exact one (it has fp16 rounding and a different summation order), and it isn't the same speed. Identical output and identical timing means both arms ran the same code. A comment on the dispatch branch says as much:

// 3-4 rows stay on the 2-token multi kernel even in fast mode.
// Tried and rejected 2026-09-06: routing them to the single-stream
// fast GEMM instead (multi_max = 2) made the real model SLOWER --
// Qwen 27B mixed, MTP depth 2: 58.5 -> 64.9 ms/pass, depth 3:
// 61.7 -> 68.0.

For the quantized matrices, the flag only changes the kernel at 5 to 16 rows. The week before, the same comparison at 64 tokens had read 34.0 vs 34.2, 58.4 vs 58.5 and 61.8 vs 61.7 ms per pass at depths 1–3, for the same reason. Where the fast kernel does run, it earns its place: short-chunk prefill goes from 56.0 to 71.1 tok/s, and it's what makes a 6-row verify round affordable on DeepSeek-V4-Flash.

What a ledger entry needs

• The switch and the queue that measured it, so the arm reproduces exactly.
• Both rounds and the number that killed the idea, not just "slower".
• The mechanism, which tells you when the result stops applying. The split lost because 80 cores were already full; the tile rule was right at 128 tokens.
• The lever it points at, if any. "More tokens per chunk" was worth 22%.
• Proof the arm ran. Before recording "exactly neutral", confirm the arms dispatched different code.

The verify kernel's entry already existed, numbers included, one line above the dispatch branch. Writing negative results down is half the job. Reading them before you write the next queue is the other half.

Five kernel ideas that all measure neutral is the point where someone has to keep the ledger honest — the on-site packages are built around keeping it.

Key takeaways

1. Keep the number that killed each idea, with both rounds and the switch that reproduces it, or the idea gets re-run. Ours did.
2. Scratch memory is a cost, not a detail. 64 MB turned a 1% loss into a model that couldn't run, because that model already sat just under Metal's GPU working-set limit.
3. Wins on a narrow GPU don't carry over to a wide one by default. The split and the band order both looked good on a 10-core laptop GPU; on 80 cores one lost and the other did nothing.
4. Tuning rules carry the conditions they were tuned at. Tile rules from 128-token chunks were about 7% slow on most shapes at 2,048.
5. Record the lever a negative result points at. "More tokens per chunk", from a losing tile size, became a 22% gain.
6. "Exactly neutral" is a reason to check that the arm ran. Identical tokens and timings from a kernel that is neither bit-identical nor equally fast meant the flag never reached the 4-row dispatch.