Every Sub-4-Bit Number We Had Was Uncalibrated

Our calibrated quantizer had never once run on a mixture-of-experts model, because the dispatch guard looked at layer 0 and layer 0 is dense. The tell was a calibrated arm that tied its baseline to four decimal places.

Summary

We spent weeks deciding how few bits our mixture-of-experts models could take. Every one of those conversions had fallen back to round-to-nearest, silently, because the branch that selects the calibrated quantizer tested whether layer 0 has a router — and in these architectures layer 0 is a dense block. The fallback is not an error. It produces a model, reports a successful conversion, and scores a hair away from the baseline, which is what you expect a calibrated quantizer to do. The signature we should have caught immediately: the calibrated arm tied its round-to-nearest baseline exactly. Same size, same worst-error figure, NLL delta +0.0000 with a bootstrap interval of [+0.0000, +0.0000], top-1 agreement +0.0. Two more guards were wrong underneath it. And there is a rank bound that decides how much calibration could have bought us anyway, which for the large models is more tokens than we were feeding it.

The check is the equality itself

A calibrated conversion that exactly reproduces round-to-nearest has not calibrated anything. That sentence is obvious written down and is not obvious at 11pm looking at a results table, because every other number in that table is the number you wanted:

A real calibrated arm moves. It moves a little, and sometimes in the wrong direction on one document, and the bootstrap interval straddles zero — that noisy, unsatisfying shape is what a working quantizer looks like. A perfectly clean zero is a fallback.

We now treat exact equality between a calibrated arm and its baseline as a failed conversion, not a passed one.

Three guards, three different ways of reporting success

Wiring the calibrated path to this model family took three attempts. Each failure lied differently.

1. The guard keyed on layer 0. The dispatch test was "layers.0.mlp.router" in tensors. Several of these architectures open with a dense block before the mixture-of-experts stack starts — it is a common design, not an oddity — so the key was absent and the converter took the fallback branch. It said so, in a log line, among several thousand other log lines. Now it matches any layer's router.

2. The reference implementation was not on the machine. The converter resolves its NumPy references relative to the script's own directory. Run it from the laptop checkout and they are there; run it from the working directory on the workstation, which is where a 5-hour conversion actually runs, and the path does not exist. ModuleNotFoundError. This one at least failed loudly.

3. The architecture string is not enough to pick a reference. Two model families derive to the same internal architecture name and use different token mixers. The reference we had models one of them; the other has no lin.w_z tensor at all, so the reference died on a KeyError deep inside the calibration loop. The guard now tests for a tensor the reference actually reads, which means one family calibrates and the other falls back with a message instead of crashing. That is the honest outcome: we can calibrate the model we serve, and we cannot yet calibrate the other one.

Three guards, and the common shape is that each one asked a question adjacent to the one that mattered. "Is this a mixture-of-experts model" instead of "does this model have a router anywhere". "Is the architecture supported" instead of "does the reference read the tensors this checkpoint has".

What this invalidates

Everything we concluded about sub-4-bit experts on these models is a round-to-nearest result. That includes the quality gate we ran at 2 bits, which failed — a fair failure, but a failure of RTN at 2 bits, not of 2-bit quantization. Those conclusions need re-running before they mean what we wrote down.

Our published 4-bit and mixed conversions are unaffected: they were always RTN and always labelled as such.

The rank bound, which caps what calibration can buy

Worth knowing before anyone budgets five hours per conversion for this.

In each expert, the first and third projections read the same input, so one Hessian over the block's hidden dimension serves every expert in that block, built from all T calibration tokens. It is full rank, it is nearly free, and it gets better as the expert count rises. The second projection is the problem: its Hessian is per-expert, sized by the intermediate dimension, and it only sees the tokens actually routed to that expert. Full rank needs

T >= moe_intermediate * n_experts / top_k

Below that, the Hessian is rank-deficient, the Cholesky accepts it after damping without complaint, and the result is mostly the damping term. We were calibrating on 2,048 tokens.

There is a cheap diagnostic that separates "the wiring is broken" from "this Hessian is noise": calibrate the first and third projections only, and score that against full calibration. If they match, the damping is doing all the work on the second one, and more calibration text is the lever, not a better solver.

What we changed

• Exact equality with the RTN baseline is now a conversion failure, checked in the fidelity gate rather than read off a table by a human.
• The dispatch guards test for tensors the code actually reads, not for names that describe the model.
• The references ship next to the converter, so an off-laptop run fails on the box the same way it fails locally.

Related: our kernel benchmark measured the system cache and we pinned a hash of floating-point output — the same family of mistake, where the measurement apparatus quietly answers a different question than the one asked.