One Abandoned Request Wedged the Whole Server

We blamed GPU contention, then reproduced the failure on demand on the machine that had none. The fix was to raise a timeout — the opposite of what it looks like.

Summary

An inference server kept wedging: it would accept connections, serve /v1/models with a 200 forever, and return zero bytes on every generation request. We diagnosed GPU contention from a training job sharing the accelerator — plausible, well-evidenced, and wrong. The real cause was that the server doesn't release its scheduler slot when a client disconnects mid-prefill, and because it runs one request at a time, a single abandoned request blocks everything after it. Permanently. The fix was to raise the client timeout.

The symptom

The server accepts connections. /v1/models returns 200 instantly. Generation returns zero bytes indefinitely — even for an 18-token prompt.

Once in that state it never recovers on its own. A restart clears it, and it re-wedges within about an hour.

Its own request log tells the story in retrospect: completion_tokens: 0, decode_tok_s: 0.0, cancellation_reason: "client_disconnected" on prompts of 25k–77k tokens, while small or well-cached prompts still finish in 1–20 seconds.

That last detail — small prompts still working, sometimes — is what kept sending us down wrong paths. It made the failure look load-dependent.

The wrong diagnosis, and why it was convincing

The affected machine was doing double duty. Alongside the inference server, it was running a PyTorch training job. On Apple Silicon, PyTorch and MLX share one GPU.

The timing was compelling: training started around 14:22, the server first wedged around 14:53.

The comparison was compelling: the second machine ran a byte-identical server command — we diffed them, only the port and key file differed — and did not wedge. It had no training job.

Memory was ruled out: 256 GB with about 137 GB free.

So: GPU contention. We pulled the affected machine from the load-balancer pool, let training continue, and the coding agent that had been timing out went back to completing runs in about 90 seconds.

Problem apparently solved. It wasn't.

Reproducing it on the healthy machine

The thing that broke the theory was trying to reproduce the failure deliberately, on the machine that supposedly couldn't have it.

Sent a 41,818-token prompt to the healthy machine — no training job, nothing else running. The client timed out at 280 seconds. The very next request, an 18-token prompt, then hung for 90 seconds with zero bytes.

Restarting the server fixed it instantly.

That single experiment rules out GPU contention, RAM, thermals, and any machine-specific explanation. Both machines have the bug. One just gets provoked more often.

The general move: try to reproduce your failure on the machine your theory says is immune. If it reproduces, the theory is dead in one experiment. We'd had a correlation — training job present, wedging present — and correlation across two samples is not much of an argument, however good it feels at the time.

The real cause

The server does not release its scheduler slot when a client disconnects mid-prefill.

Both machines run a serial scheduler — exactly one request at a time, which is the server's own default rather than a choice we made. So one abandoned request blocks every later request, forever, until restart.

And there's a feedback loop that makes it trigger constantly:

Prefill runs at only ~200–330 tok/s on this hardware. A 25–50k-token prompt therefore needs 100–250 seconds before the first token appears. Every client in the path — the coding agent, the agent framework, the load balancer's own 240-second timeout — gives up before that. The disconnect wedges the server. Every subsequent request hangs.

Large agentic prompts wedge these servers routinely, not rarely. Our workload had been generating the trigger condition several times an hour.

For scale, the same 41,818-token prompt through the same load balancer: our GPU cluster answered in 25.5 s; the Mac produced 0 bytes in 280 s. Different hardware, different engine, real prefix caching on one side.

The fix is counterintuitive

Since the wedge is caused by the client disconnecting mid-prefill, the fix is to give big prefills room to finish so no slot ever leaks.

Raise the timeouts. Pool timeout and stream timeout from 240/180 to 900/900.

Verified: a 39,623-token prompt completed in 138.7 seconds with a correct answer, and the machine stayed healthy afterward — 2.6 seconds on the next request. The identical prompt had previously produced zero tokens and wedged the box.

This is worth sitting with, because every instinct says a hanging server needs a shorter timeout. But the timeout wasn't protecting us from the server. The timeout was causing the failure. A client that waits gets its answer; a client that gives up destroys the server for everyone.

Any system with a non-releasing resource on cancellation has this property. Aggressive client timeouts convert a slow system into a broken one.

Two fixes we tried and reverted

Restarting the server. Treats the symptom, returns in about an hour. Worth naming explicitly because it's the first thing anyone does and it teaches you nothing.

Capping context at the load balancer. We set a maximum input size on the Mac pool with automatic fallback to the GPU cluster for oversized prompts. It worked exactly as designed — a 41.8k prompt rerouted and answered in 38 seconds, the Mac untouched.

We reverted it. Rerouting around the problem wasn't the goal; the Macs were supposed to serve this traffic. It's documented here so nobody re-proposes it as a discovery.

There's a real judgment call in that. The cap was a working mitigation and we chose the root fix instead. That's the right call when the root fix exists and is one config line. It would have been the wrong call if the fix had been "wait for upstream."

Three more things found along the way

The fallback list retried the sick machine first. The load balancer's fallback order listed the degraded machine ahead of the healthy one, so a failed request retried onto the box that had just failed it — turning one agent run into a 6m40s stall. Check the order of your fallback lists, not just their membership.

A watchdog probing /v1/models is useless here. It returns 200 throughout the wedge. Any liveness probe for this failure mode has to probe generation. This is the Apple-Silicon twin of a lesson from our GPU cluster, where the same distinction mattered for the opposite reason — there, /v1/models returning 200 was the evidence that the engine wasn't dead.

Would a different engine have been better? We had a llama.cpp instance serving the same model with the same speculative decoding on the same machine, so we measured it rather than speculating:

Our server wins both. Swapping would have been a downgrade — its single-stream is about 2× slower as configured, and it would need a parallelism flag to show any batching benefit at all. Keep the server, fix the timeout.

A single abandoned request taking down the whole server is the kind of thing we untangle in an on-site Rescue — and keep untangled.

Key takeaways

1. Try to reproduce the failure on the machine your theory says is immune. One experiment killed a diagnosis that had a compelling timeline and a compelling control.
2. A correlation across two machines is two data points. "Training job present ⇒ wedging present" felt like evidence. It was a coincidence of provocation frequency.
3. A resource not released on cancellation turns client timeouts into the cause of outages. Raising the timeout was the fix; every instinct said lower it.
4. A serial scheduler makes one leaked slot fatal. Concurrency would have masked this for a long time — arguably worse.
5. Check the order of fallback lists. Ours retried the failing machine first and turned a fast failure into a six-minute stall.
6. Probe generation, not /v1/models. A model-list endpoint answers 200 through this entire failure.
7. Measure the alternative before switching to it. The obvious replacement engine was 2× slower single-stream on the same hardware and model.