The Crash We Couldn't Reproduce on Purpose

Five hours of deliberate abuse produced zero failures. Ordinary traffic killed the same engine in about an hour. The difference was load shape, not load size.

Summary

A distributed inference engine kept dying with a deadlock in its inter-process message layer. We built a harness to provoke it: twelve concurrent 100k-token streams, random mid-stream aborts, five hours. Zero crashes. That same afternoon, ordinary mixed traffic at thirteen concurrent requests killed the other cluster in about an hour. The lesson is that we'd been stress-testing the wrong dimension — and that a thermal fix which looked like the obvious cure demonstrably wasn't.

The fault

TimeoutError: RPC call to sample_tokens timed out, from an acquire_read timeout in the shared-memory broadcast layer. One worker in a two-node tensor-parallel group stops responding, and the executor waits until it gives up.

No GPU exception in kernel logs. No out-of-memory. No GPU reset. The hardware is fine; the processes have stopped talking to each other.

The pattern over four events across four days: backlog → hang → death. Around thirty minutes of genuine saturation, then a timeout-class alert, then the engine core dies.

The newest variant is worse — a silent deadlock. No crash capture, the container still reports Up, and the only symptom is that the metrics counters have frozen. Nothing pages, because nothing crashed. You find it because requests stopped completing.

Building a reproduction

Waiting for organic crashes means a two-to-four week feedback loop on any hypothesis. So we tried to provoke it: twelve concurrent ~100k-token streams, random-depth mid-stream aborts, five hours, one cluster drained from the pool so real traffic wasn't affected.

The watchdog was left armed deliberately — its crash captures are the evidence we wanted.

Result: zero crashes. 295 heavy requests, 276 completed, 19 aborted, p50 763 seconds per request, 96% utilization the whole time, container unbroken.

A clean negative. The provocation failed.

Then we found the crash we'd missed

Reviewing logs the next day, we found that the other cluster had genuinely crashed the previous afternoon — same signature, same timeout, no GPU fault. We'd missed it for eleven hours because during the alert storm we checked one cluster's uptime and not the other's.

That single oversight nearly cost us the whole finding. Check every node's crash logs before declaring "no crash" during an incident. An alert storm makes you want to answer the question quickly, and "is the engine up?" has a different answer per node.

The recovered crash carried two decisive facts.

First: load at death was 13 running, 2 waiting. Moderate, mixed, ordinary traffic. Not extreme saturation.

Second, and this killed our leading hypothesis: the crash capture contained the marker showing our thermal patch was applied. The patched build crashed anyway.

The thermal fix that wasn't the fix

The hypothesis had been good. A published analysis of vLLM on this hardware showed that the shared-memory broadcast layer spins CPU cores for a full second after every inter-process message. During decode, messages arrive every few milliseconds — so it never sleeps. It just spins, continuously.

We measured it: 291% CPU on the worker process and 91% on the engine process during a single decode request. The SoC thermal zone went from 57°C to 81°C while the GPU itself sat at 68°C under 96% utilization. The chip was cooking on message-passing overhead, not on inference.

And the crash died in that exact file.

The fix is one constant — the spin duration from 1 second to 2 milliseconds. Deployed across all nodes. Measured: decode CPU fell from ~382% to ~200%, the engine process went from 91% to 0%, the hottest thermal zone dropped 81°C → 75°C. No throughput cost.

That's a genuinely good change and we kept it. It is not the fix for the crash. The crash capture proves the patched build still dies.

This is the value of a crash capture that records which patches were live. Without that marker we'd have counted the next quiet week as confirmation and moved on with a false belief.

Shape, not size

Put the two results together:

• Homogeneous load — twelve identical 100k-token streams for five hours — no crash.
• Mixed organic load — small, medium, and monster requests interleaved, streaming, real client aborts, ~13 concurrent — dead in about an hour.

The synthetic test was harder by every metric we'd thought to control: more concurrency, bigger requests, longer duration, higher sustained utilization. It didn't reproduce anything.

What the organic load had that ours didn't was heterogeneity and real abort behavior. Requests of wildly different sizes finishing at unpredictable times, streaming connections dropped mid-generation at shallow depths.

Our harness configured 50% aborts and achieved about 6% effective ones — the aborts were firing too late, after the interesting window. That gap is probably the whole story.

It also matches what the upstream bug report describes as the potent trigger: agentic traffic with streaming and mid-generation aborts.

The generalizable lesson: if your stress test is more uniform than your production traffic, it is not a harder test. It's an easier one wearing a hard test's numbers. Uniform load lets a scheduler settle into a rhythm. Heterogeneous load is what keeps it in the transitional states where the races live.

Our next attempt replicates the shape — mixed sizes at realistic proportions (p50 3k, p90 32k, a few above 100k), streaming, forced shallow aborts two to five chunks in — rather than adding more depth.

Related: we'd already learned that ordinary traffic is a better provocateur than our synthetic harness, and the cheapest experimental setup turned out to be letting real bursts be the test while reading the crash captures.

Operational rules we now follow

Never restart the head node alone on a multi-node tensor-parallel stack. It orphans the other rank. You get a half-dead stack that reports partially healthy, which is harder to diagnose than a cleanly dead one.

Expect a rendezvous race on relaunch. One restart hit a transient failure during initialization — the head rank died and respawned mid-rendezvous. A clean stop followed by a start fixed it. It looked patch-related and wasn't, which cost an hour of investigating the wrong thing.

Read an asymmetric A/B carefully. We were mid-trial on two versions of the collective communications library when we introduced routing that sent all heavy requests to one cluster. That broke the symmetry — but usefully. The cluster on the new version now carries the harsher load, so "it crashes no more than the control" is strong evidence for the new version, while "it crashes more" is ambiguous. Worth writing down which direction of an accidentally-confounded result you're still allowed to believe.

Don't raise the execution timeout. Our failure is the model-execution timeout firing at 300 seconds. Raising it doesn't fix a deadlock, it just makes you wait longer to find out about it.

Check the power envelope before worrying about hardware damage. All of this was engine-process death, never host instability. Clock caps hold power well below the threshold where the platform protects itself. Knowing that turned "is this damaging the machines?" from an open worry into a settled question.

A crash you can't reproduce is usually a missing piece of state — the kind of thing an on-site Rescue finds by reading the machine instead of re-running the request.

Key takeaways

1. A stress test more uniform than production is easier than production, whatever its concurrency number says. Reproduce the shape — mixed sizes, streaming, real aborts — before adding depth.
2. Verify your harness achieves what it configured. Ours asked for 50% aborts and delivered 6%, which was probably the difference between reproducing and not.
3. Record which patches are live in every crash capture. Ours is the only reason we know the thermal fix didn't fix the crash, rather than believing it did.
4. A real improvement that doesn't fix your bug is still a real improvement — but only if you resist filing it as the fix. Ours dropped SoC temperature 6°C for free.
5. Check every node during an incident. We missed a crash for eleven hours because we confirmed one cluster was up and inferred the other was.
6. Never restart the head of a multi-node TP stack alone. Orphaned ranks are worse than clean deaths.
7. A silent deadlock with frozen counters and a healthy-looking container is a distinct failure class. If your monitoring only catches crashes, it will not catch this one.