Your Engine Isn't Hung. It's Saturated.

A twenty-message alert storm, an auto-restart that threw away thirty in-flight requests, and why the obvious watchdog fix would have broken real failure handling.

Summary

Our monitoring paged twenty times in an hour with "engine hung." The engine was not hung — it was working through a queue 99 requests deep. The watchdog's automatic restart made it substantially worse, discarding thirty in-flight requests and triggering a retry storm. The fix looks obvious (suppress restarts when the queue is deep) and is wrong, because the fleet's most common real fault produces an identical signature. Here's the distinction that actually works, and the twenty-day-old config comment at the root of the whole thing.

The signature

A twenty-message alert storm starting at 10:22. Every message: vLLM completion probe FAILED — engine hung or still loading.

What was actually happening: one API key had started a long-context batch around 09:43, ramping hard by 10:08. Seven to sixteen requests per minute, average prompt 10k–40k tokens, one at 196,763 tokens.

The cluster saturated. Running sat at 4–12, Waiting peaked at 99, aggregate decode collapsed to 2–7 tok/s, per-request latency ran 100–400 seconds. Both accelerators at 96% utilization.

The watchdog's completion probe has a 90-second timeout. It queued behind the backlog like every other request, timed out, and reported a hang.

The distinguishing signature, and it's clean:

• /v1/models returns 200 the entire time
• Completion probes time out
• Waiting is deep and rising

That combination is backlog. A genuinely hung engine doesn't serve /v1/models.

The diagnosis is two commands: read the Waiting: series out of the container logs, and check the queue depth metric on every cluster in the pool. We initially checked one and assumed the other was fine. It was worse — 107 waiting against the first one's 39.

Auto-restart is actively harmful here

At 10:34 the watchdog restarted the stack across both nodes. That threw away roughly 30 in-flight requests. The client retried them. The queue rebuilt to 99 within ten minutes.

Fifteen minutes later the watchdog emitted this, which reads like a disaster:

SUPPRESSED ... COOLDOWN for another 165min ... NOTHING will recover it

That message is the correct outcome. Leave it suppressed. Let the queue drain.

There's a broader point here about restart-on-failure as a policy. It's the right default for a crashed process and precisely the wrong response to an overloaded one, and the two are hard to tell apart from outside. A restart under load doesn't just fail to help — it converts queued work into retried work and deepens the queue you were trying to relieve.

The retry storm we caused ourselves

Restarting the two clusters sequentially, to avoid going fully dark, had a cost we hadn't anticipated.

When the first cluster went down for its restart, its ~77 queued requests failed. The gateway retried them (two retries configured). Every one landed on the surviving cluster, which went from 0 to 134 queued in seven minutes.

The new rate limits were working the entire time — the gateway was actively returning 429s. The queue was retry fallout, not fresh demand.

Then, after both restarts completed, the first cluster sat at 0/0 while the second held 149, because the gateway hadn't re-added the first to the pool yet (health check interval 120s, cooldown 300s).

Drain the stack at the load-balancer layer first, or expect the survivor to eat the entire backlog and plan to restart it again once the pool is whole.

Zombie queue entries

The engine's access log only prints on completion. Queued requests are invisible in it.

That's why the gateway looked idle at 1–4 requests/minute while 95 were queued. Two views of the same system, one of them structurally blind.

Worse: requests whose client already gave up — our gateway times out at 600 seconds — never leave the engine's waiting queue. A queue can be almost entirely work that nobody is waiting for any more. Which is why Waiting can sit static while completions flatline.

The fix that would have broken everything

The obvious patch: if the queue is deep, don't treat a probe timeout as a fault.

We shipped that. It was wrong, and we caught it on the second pass.

The fleet's most common genuine fault class is a distributed collective hang — the engine's worker processes deadlock waiting on each other. Its probe_reason is literally timeout >90s. The same signature saturation produces.

Depth-only logic would have suppressed every collective hang that struck under load. Forever. One page, no restart, and a dead engine sitting there. And collective hangs preferentially strike under load, so we'd have disabled the watchdog exactly when it mattered.

The working distinction is progress, not depth.

The guard now sums vllm:prompt_tokens_total + vllm:generation_tokens_total and compares against the value from the previous run:

• Counters advanced → the engine is doing work. Page SATURATED, do not restart.
• Counters frozen across a full ~3-minute cycle → engine HUNG. Fall through to the normal restart path.

Including prompt tokens matters: a single very long prompt in prefill generates no output tokens for a long time, and a generation-only counter would misread that as a hang.

Non-timeout failures — connection refused, empty reply, HTTP 5xx — still restart immediately, as before. Those are unambiguous.

The general lesson: "busy" and "stuck" look identical from outside if you only measure depth. They differ in whether anything is moving. Any saturation guard needs a liveness signal that a wedged process cannot fake, and a monotonic work counter is the cheapest one available.

The root cause was a comment

None of this should have happened, because rate limits should have kept the queue shallow.

The gateway config had said this since three weeks earlier:

TESTING (operator request): rate limits lifted GLOBALLY … ORIGINAL values to restore when done: rpm_limit: 300 / tpm_limit: 200000 / max_parallel_requests: 8.

Nobody restored them.

max_parallel_requests: 8 is the lever that matters. It sits just above real measured capacity, so the queue stays shallow without throttling anyone below what the hardware can actually do. The rpm and tpm limits are loose by comparison — 200k tokens/minute still permits about 5.5 requests/minute at 36k tokens each.

The file the comment referenced, containing the original values, no longer existed. The config comment was the only surviving record of what the numbers had been.

Two things worth stealing from this:

A "temporary" change with no expiry is permanent. Three weeks is enough for everyone who knew about it to stop thinking about it, and for the cluster's behavior under load to become a mystery.

If you write "restore these values later," put the values in the comment, not in a file you'll reference. The comment survived. The file didn't. That accident is the only reason the restore was possible at all.

Measured capacity, for sizing limits

Worth writing down, because "how many requests can we take" is otherwise a guess:

The pool completes about 1.6 requests/minute across both clusters on a 36,000-token average prompt, averaging 186 seconds with a p95 of 546 seconds. By Little's Law that's roughly 5 requests genuinely in flight.

KV cache is not the constraint — 1.1M tokens holds about 30 of those requests. The 8192-token prefill batch budget is: a 36k prompt costs about 4.5 scheduler steps of pure prefill, which starves decode down to 2–7 tok/s.

The arrival rate during the incident peaked at 7–16 requests/minute. Five to ten times capacity.

That's the number that makes the whole event legible. No amount of watchdog tuning fixes a 10× overload; only admission control does.

Bugs found while fixing this

Two, both embarrassing, both worth checking on your own stack:

A readiness probe using curl -f with no auth header had been 401-looping for 25 minutes on every single boot since authentication was enabled. The stack came up eventually, so nobody noticed the delay was self-inflicted.

A watchdog that can only see one cluster's queue. One cluster's engine binds to localhost and the other host has no trust relationship to it, so the second watchdog reads only its own queue and uses it as a load proxy. If the first is saturated while the second is idle, it still pages with the old wording. That's the safe direction to fail, and it's written down rather than fixed, because fixing it means opening a network path we'd rather not open.

Saturated and hung look identical from a dashboard; telling them apart is the first thing the setup audit checks before any tuning.

Key takeaways

1. /v1/models returning 200 while completion probes time out is saturation, not a hang. Add rising queue depth and you have a positive identification in two commands.
2. Never auto-restart on a probe timeout alone. Ours discarded 30 in-flight requests and rebuilt the queue in ten minutes.
3. Queue depth cannot distinguish busy from wedged — the most common real fault has the identical signature. Key the guard off a monotonic progress counter instead.
4. Include prefill tokens in your progress signal. A long prompt generates no output for minutes and would otherwise read as a hang.
5. Drain at the load balancer before restarting one member of a pool, or the survivor absorbs the entire retried backlog.
6. Queued requests are invisible in completion-only access logs, and requests whose clients gave up never leave the queue. The gateway can look idle while 95 requests are stacked up.
7. Put restore values in the comment. The file ours referenced was gone; the comment was the only surviving record.