Our Model Conversion Ran Five Times at Once

A conversion driver without an `if __name__ == "__main__"` guard, a worker pool that starts with spawn, and every worker re-ran the whole conversion. Three more converter bugs from the same day, none of which the test fixture could reach.

Summary

We started the first real conversion of a 475 GiB checkpoint on the only machine that had room for it — which is also the machine serving production. We ran it beside production under a guard script that kills the converter's process tree if the kernel reports critical memory pressure twice, and samples memory every five seconds. Four runs died before they reached the experts. The main cause was one missing line: the small driver script that called the converter had no if __name__ == "__main__": guard. The converter's expert pool starts its workers with spawn, spawn re-imports __main__, and so every worker ran the entire conversion again from the top. The log printed "derived recipe" five times. Each worker grew to between 15 and 77 GB. Three more bugs turned up in the same session, and the pattern across all four is the useful part: every one of them only exists at the real model's size or in the real checkpoint's config, and the synthetic fixture that gates the converter is neither.

The one that mattered

On macOS, Python's multiprocessing starts workers with spawn by default. A spawned worker is a fresh interpreter that imports your main module to find the function it was asked to run. If that module does its work at import time — a script that just calls main() at the bottom — the worker does the work too.

The converter itself was fine; it guards its own entry point. The small driver we wrote around it for this run set arguments and called the converter's main, unguarded. So:

• the parent derived the recipe and started the expert pool;
• each worker imported the driver, derived the recipe, and started loading the model;
• memory climbed until the production machine hit the kernel's pressure warning, three times across the runs;
• run 4 then died with multiprocessing's own bootstrapping RuntimeError, which is the error Python raises precisely for this mistake — the one time it said so plainly.

The fix was the guard, in the driver. Run 5 printed "derived recipe" once.

The tell is cheap: any line your conversion prints once per run appearing N times, where N is your worker count plus one.

Three more, from the same afternoon

The embedding and output head were quantized whole, in float64. Each is a 129,280 × 5,120 matrix. The quantizer upcast the full matrix to float64 and worked on it whole: 662 million values, 2.65 GB as float32, and with the float64 copy and its intermediates each quantization peaked around 26 GB, roughly ten times the input. It now quantizes in row chunks, byte-identical to the old output at 4 and 8 bits, with a 4.7 GB peak. Real, and worth fixing, but secondary: on its own it would not have taken the box to warning.

A config key name derived a drafter width of zero. The model's speculative-decoding stages route to 3 experts per token. The released config calls that dspark_num_experts_per_tok; the vendor's own model-args class calls it dspark_n_activated_experts. Our converter read the second name, found nothing, and wrote 0 — and downstream, 0 means "use the backbone's width", which is 6. The drafter would have routed to twice its trained width with no error anywhere. It now reads either name, and a test feeds it the released one. This is the one most likely to have survived into production: the model would have run, and drafted worse, with nothing pointing at the config.

A sidecar export wrote 205 GB of the wrong tensors. A flag that splits out the small drafter module into its own file was supposed to write only the drafter's tensors. The new model's remapping step merges its own extra tensors back in after the older family's split, so the "drafter" file started receiving the model's lookup tables. We stopped it at 205 GB. It now writes only the drafter.

Why the fixture passed

The converter is gated by a synthetic fixture: a tiny model built from the same architecture with random weights, converted and compared token for token against a NumPy reference. It is a good gate and it caught nothing here, for reasons that are structural rather than careless:

• the tests call the converter from the test runner, never through a script — the unguarded driver only existed for the real run;
• a tiny embedding does not care whether you quantize it in float64;
• the fixture's config was written by us using the vendor's model-args key name, so it agreed with our converter and not with the released config file;
• the fixture has no lookup tables large enough to notice where they went.

A fixture proves the converter agrees with our understanding of the model. The real config and the real sizes are a separate test, and the cheapest version of it is a dry run on the real checkpoint that derives the recipe, prints every derived field that fell back to a default, and stops before touching a weight.

Verdict

Put the __main__ guard in every script that calls into code which might use a process pool, including throwaway drivers. When a conversion runs on a machine that is also serving, run it under something that kills only the conversion's tree on memory pressure — that guard is why a missing line cost us four failed runs instead of a production outage. And treat "fell back to a default" in a converter's derived config as a warning to read, not a convenience.

The conversion itself, at about 2.8 expert tensors per second over 46,080 tensors, was expected to take four and a half to five hours. Whether the converted model is correct is a separate question; this post is only about getting it to run once.

Related: our two machines ran different models under the same name and every sub-4-bit number we had was uncalibrated.