Two Crash Fixes That Cost Throughput and Fixed Nothing

A daily inference-server crash got two mitigations, both matched to the symptom by name, both 'validated' by a load test that passed 12 of 12. One mechanism was impossible, the other flag routed traffic into the failing path, and one of them cost 24% of decode.

Summary

Our GPU cluster's inference engine was crashing about once a day. We shipped two mitigations. Each was chosen because the symptom's name matched something — an upstream issue title, a flag named after the feature we suspected. Both were declared validated after a concurrent load test passed 12 of 12. Neither could have worked. The first depended on a mechanism the engine's own configuration makes impossible, and it took decode from 39.4 to 29.9 tokens per second — 24% — until we reverted it. The second flag, meant to switch a feature off, instead pushed more traffic into the code path the crash was actually in. And the load test that "validated" both of them could not have failed, because the crash happens once a day and the test ran for minutes.

Fix one: a matching title

The crash looked like an open upstream issue about full CUDA graphs colliding with chunked prefill, and that issue suggests a piecewise graph mode. We set it.

The engine's graph modes are not single settings; they are pairs — one mode for pure decode batches, one for mixed batches that include prefill chunks. Our configuration only ever used a full graph for pure-decode batches; it never ran one on a chunked batch at all. So the mechanism in the issue could not produce our crash on our configuration. The issue itself, still open, listed the piecewise mode as "no improvement over baseline".

The real cause turned out to be a bug in the collective-communication library, fixed in a later version — and the box had already been running the fixed version since before the crashes started. The mitigation fixed nothing and cost a quarter of decode throughput for as long as it was in.

Lesson: a matching issue title is where the investigation starts. Before changing a flag, read what the flag does in the code, read the issue's own status and results table, and check the value the engine actually booted with — not the one in the compose file, which may have drifted.

Fix two: the flag named after the suspect

A second crash was a cuBLAS internal error. The engine has a multi-stream GEMM feature, and a threshold that controls it; we set the threshold to 0 to "disable multi-stream GEMM".

The gate reads enable = batch_tokens <= THRESHOLD: multi-stream is for small batches. And the faulting frame in the stack trace was in the sequential branch, on a prefill chunk of more than 1,024 tokens that was already running serially. Setting the threshold to 0 did not move the crash out of anything; it moved small decode batches into the same serial path, losing their overlap for no stability gain. We reverted it to 1,024.

Lesson: with a stack trace in hand, find which side of the gate the faulting frame is on before touching the flag that gates it.

The validation that could not fail

Both mitigations went in as "VALIDATED" after a 12/12 concurrent load test. The crash rate was about one per day. A test that runs for minutes against a once-a-day fault passes whether or not anything was fixed; its pass carried no information about either change.

Silence afterwards is no better. The same stack had once gone six clean days on an older library version against the same rate, so a quiet week proves nothing. The informative event is the opposite one: a single recurrence after the change refutes it outright. We now write that falsifier down before calling anything fixed — "any engine-dead error after this timestamp" — and call the change "unverified, watching cadence" until the fault's own timescale has passed.

One more misfiling

Three different faults had all been filed as "the crash": a collective-communication hang that sits for the full 600-second timeout and leaves a flight-recorder dump; an instant cuBLAS crash in the sparse-attention indexer; and a Triton "operation not permitted" crash. The instant crashes die in about two minutes with no dump. Our probe log made the split countable — 13 connection-refused failures, the instant crashes, against 4 timeouts, the hang. Time-to-death and whether a dump exists are the first fork; merge the classes and every fix gets credited or blamed for the wrong fault.

Verdict

A mitigation is a hypothesis with a cost. Check that its mechanism can produce your symptom on your configuration, check which branch the faulting frame is actually in, and define the observation that would prove it wrong before you declare it right. Both of our wrong fixes were plausible, both passed a test, and one of them cost a quarter of decode throughput for as long as it stayed in.

Related: the crash we couldn't reproduce on purpose and your engine isn't hung, it's saturated.