The Model Booted, Returned 200, and Spoke Fluent Nonsense

A filtered download skipped one shard of 141, the one holding the final norm; the server booted with it uninitialized and raised nothing. A shard audit then crashed and reported green on a node holding 17 of 141 shards. Liveness checks cannot see any of this.

Summary

Three failures on our GPU cluster, all of the same shape: the server is up, every request gets an HTTP 200, and the output is wrong. The first was a download that skipped one shard of 141 — the one holding model.norm.weight, the final normalization before the output head. The inference engine booted with that tensor uninitialized, logged no error, and produced fluent nonsense at temperature 0. The second was a missing expert-index remap in a different model, which answered every request with zum zum zum & & &. The third was the check we built after the first: a pre-flight shard audit that crashed on a missing index file, printed a traceback, and — because the gate counted error lines and a traceback has none — reported green on a node holding 17 of 141 shards. None of this is visible to a liveness probe.

A filtered download is a different checkpoint

The download was filtered on purpose: we only needed some of the shards, the thinking went, because another part of the pipeline supplied the rest. The filter dropped one shard too many. It held one tensor that matters enormously and is tiny — the final RMSNorm's weights.

A missing tensor ought to be a load error. In this engine, on this path, it was not: the parameter stayed uninitialized and was used anyway, so the last normalization multiplied by whatever was in memory. Text came out. It was grammatical and meant nothing.

Counting files proves nothing about coverage. The audit that would have caught it maps every tensor in the checkpoint's index to its shard and checks that the shard is present — 118,629 tensors in this one — and classifies what is missing: some expert tensors may legitimately be supplied elsewhere; attention, norm, embedding and router tensors are fatal. And the cheapest fix of all is to not filter: disk was cheaper than the debug cycle.

The audit that could not fail

So we added that audit to the pre-flight, and gated on the count of lines it printed saying a needed tensor was missing. On one node the checkpoint's index file had not arrived — small files often land last. The audit script raised on the missing file. Its output was a traceback, which contains zero "missing" lines, and the gate read zero as clean. That node had 17 of 141 shards.

A gate that counts findings has to prove the audit ran first: a zero exit code and the audit's own summary line present in its output. Otherwise "no findings" and "no audit" read the same. The same class of mistake gave us more green signals about the wrong thing within one night: a build step that exited 0 on a stale image, an image load that succeeded with the wrong content, and a cached layer replaying an old version string. Verify the artifact, not the transport.

Fluent failure needs a coherence probe

The expert-remap failure was a quantized mixture-of-experts model missing its expert-index remap. It emitted zum zum zum & & & at temperature 0, with a 200 on every request. The server was healthy by every measure a liveness probe has.

What catches it is a question with exactly one right answer, asked on a schedule: What is 17 times 23? Reply with only the number. If the reply doesn't contain 391, the model is not working, whatever the status code says.

And what you do on that failure matters: alert, but do not auto-restart. A restart reproduces a wrong conversion or a missing runtime patch exactly. Auto-restart is for a process that died, not for one that is running the wrong thing.

Verdict

Serving returns 200 for a model that is missing its last layer, and for one routing every token to the wrong experts. Audit tensors, not files; make every audit prove it ran before you believe its silence; and give every model that can fail fluently a probe that checks an answer, not a status code.

Related: every failure silently scored 0.5 and your engine isn't hung, it's saturated.