Turning On the Drafter Moved the Whole Model to the Slow Path

Three times, enabling a speculative-decoding module added a few hundred experts to a mixture-of-experts model, overflowed a count derived without them, and silently dropped the entire model onto the paged fallback: 485 ms per pass instead of 28, and different tokens. The different tokens were a second bug.

Summary

Our engine keeps a mixture-of-experts model's experts resident in GPU-owned memory, and falls back to a paged LRU cache when they don't fit. The resident set is sized by a count the converter derives from the model: how many experts, how many gigabytes. A speculative-decoding module — a multi-token-prediction layer, or a separate drafter — brings its own experts. Three times, on three models, turning one on pushed the total past a count derived without it, and the loader silently put the whole model, not just the extra experts, on the paged path. On one model: 9.5 GB pinned instead of 180.6, 484.6 ms per pass instead of about 28, and different tokens by position 8 — which is what made it look like a bug in the drafter's rollback. It wasn't. It was a budget — and the different tokens most likely a second, separate bug.

Three times

The mechanism was the same each time. The loader warms the resident set from the derived count; with the drafter's experts added it had two early-return checks — the expert count, then the gigabyte budget — and tripping either one abandoned residency for the entire model. It returned quietly, and the engine carried on correctly, and slowly. In the third case the imported file's config simply lacked the field that says how many prediction layers there are, so the derivation could not have counted them.

The fixes were boring. The converter now counts the prediction layers' experts for every family that has them — one branch had done this for a drafter already, with a comment describing the identical failure; the others hadn't. The loader now prints a warning on either fallback instead of returning silently. And the startup log has a line that must appear for a mixture-of-experts model: N experts resident. If it isn't there, or the pinned figure is far below the model's size, you are on the slow path, whatever the tokens look like.

With the counts fixed, the drafter in the second row was byte-identical to the baseline, and the GGUF import's speculative decode went to 160-175 tok/s against 107-109 without it.

The different tokens were a second bug

Paging should change speed, not output. The first incident's different tokens were the reason we spent time on the drafter's rollback logic, and we did not pin them to a cause at the time. Nine days later we found one that explains them, and it is unrelated to drafting: resident experts and paged experts ran different kernel families — the resident path through the expert-slot kernels, the paged path through a routing read-back and a per-group matrix kernel. Same weights, same bytes, other arithmetic. On a small test model, a resident budget of 20 experts against 19 moved the logits by up to 1.4e-3 from the first row, and a second test model's greedy tokens split at step 5.

Output that depends on the memory budget is a bug even when both outputs are plausible. The paged path now stages the routed experts into the same layout the resident path uses and runs the same kernels; tokens and logits match between resident and paged on a real model, and a test fails if they ever don't. The paged path is still slow — 85.5 ms per pass against 9.6 resident on that model — but it is now only slow.

Verdict

A fallback that returns quietly will eventually be the configuration you are running. Make it print, make the startup log state residency outright, and read that line before any benchmark. And when enabling a feature changes the tokens, check whether it changed the memory plan before you debug the feature: our "rollback bug" was a budget, and the tokens were most likely a kernel mismatch the budget exposed.

Related: 17 GB of attention buffers nobody read, every GPU out-of-memory error was one sysctl and can SSD offload fit a model twice your RAM?.