Our Benchmark Queues Deadlocked on Each Other's pgrep

Chained benchmark queues waited for their predecessors with `pgrep -f`, which also matched launch wrappers and the other waiters. None of them started.

Summary

Our benchmarks run on a shared 256 GB M3 Ultra as chained shell queues. Each script waits for the one before it with while pgrep -f "queue_gNN.sh", then runs its measurements through a bounded runner that holds a global GPU lock. One batch of waiters never started. pgrep -f matches full command lines, and a queue's name appeared in far more of them than the queue's own: in the screen, login and caffeinate wrappers that launch queues, and in the other waiters' pgrep calls, whose argument is the name. The fix is an anchor: ^/bin/bash .*queue_gNN\.sh. The same system failed twice more that week. First, a 12-hour CPU-only quantization job took the GPU lock, and two GPU benchmarks behind it printed skip. Then the fix for that exposed memory, not the lock, as the real constraint. Each time, a coordination primitive was answering a different question from the one we were asking it.

The setup

Each queue is a bash script launched in a detached screen session under caffeinate, so the box doesn't sleep. The first thing a queue does is wait:

screen -dmS queueg52 /bin/bash -c "exec caffeinate -i -s /bin/bash ~/work/queue_g52.sh"

# first lines of queue_g52.sh
echo "=== waiting for the earlier queues $(date +%H:%M)"
while pgrep -f "queue_g51.sh|queue_g50.sh|queue_g49.sh" >/dev/null; do sleep 60; done

Every measurement then goes through a runner that does three things. It takes an atomic lock (a mkdir with a pid file and stale-lock detection), refuses to start below a free-memory threshold computed from the model's size, and imposes a wall clock. An arm that can't start is logged, and the loop moves on.

The lock exists because pgrep had already failed us in the other direction. We used to serialize runs with pgrep -f on the engine's command line. A driver looping over several runs shows no match in the gap between two of its own iterations, so a second launcher read "idle" and started a second large model. The box hit 2% free memory with swap at 84.2 of 85 GB and went down. That was its third panic (the first two are here). That time, pgrep said "nothing is running" when something was. This time it said "something is running" when nothing was.

Failure 1: every waiter found somebody

pgrep -f matches against every process's full argument list. While the chain waited, a queue's script name appeared in three places:

• the queue itself, /bin/bash ~/work/queue_g50.sh. This was the only match we meant.
• launch wrappers: the screen session, a login shell and caffeinate all carry the script path as an argument.
• other waiters' pgrep processes. A queue waiting on g50 runs pgrep -f "queue_g50.sh|…". While that process exists, its argument list contains queue_g50.sh, so every other waiter whose pattern names g50 matches it.

The last one is the hard one to see. Waiters polling on the same 60-second loop keep a constant offset from each other. If their pgrep calls coincide once, they coincide every minute. Each waiter finds the other's pattern, concludes its predecessor is still running, and sleeps again. The chain phase-locked. Nothing was running and nothing started, and the last line of every queue log was === waiting for the earlier queues.

The fix anchors the pattern to the interpreter that actually runs a queue:

while pgrep -f "^/bin/bash .*queue_g5[012]\.sh" >/dev/null; do sleep 60; done

The argument lists of screen, caffeinate, login shells and pgrep itself all start with their own names, so ^/bin/bash rules them out. \. makes the dot a literal dot. Two conditions keep the anchor honest:

• Nothing else may start with the same interpreter path and contain the name. That's why the launcher uses exec caffeinate inside bash -c: without exec, a /bin/bash -c "… queue_g52.sh" parent could be left in the process tree, and it would match.
• A range must not include the waiting queue's own number. queue_g6[0-9] inside queue_g64.sh matches the queue itself, so it waits forever.

Failure 2: a CPU job holding the GPU lock

Later that week the batch included a quantization run: re-converting a 27B model with calibration in NumPy, on the CPU. At 2,048 calibration tokens it took 15 minutes a layer, so it was relaunched with 1,024 tokens and a 12-hour wall clock. It needs about 37 GB.

Like every long job on that box, it ran under the same bounded runner, so it held the global lock for twelve hours. That lock refuses callers rather than queueing them: a second caller exits with an error, and a benchmark queue logs that and moves on. The GPU benchmarks behind the conversion printed skip twice and carried on without their numbers.

The lock exists to stop two GPU-saturating runs from overlapping, and a CPU-only converter doesn't saturate the GPU. So the runner got a no-lock mode that keeps the other two protections:

# A CPU-only job (a converter run) may share the box with a GPU benchmark.
# It still gets the wall clock and the memory refusal, just not the lock.
if [ -n "${NOLOCK:-}" ]; then
  LOCK=/tmp/runner-nolock-$$.lock   # private to this process, never contended
fi

(Names simplified.) The conversion now runs as NOLOCK=1 nice -n 10 ./runner 43200 python convert.py ….

Failure 3: memory was the real constraint

With no lock in the way, the conversion could now run alongside any GPU queue. The biggest model is the exception.

The runner derives each model's admission threshold from its size: model bytes as a percentage of RAM, plus a 12-point margin. The largest model we benchmark needs about 79% of memory free to start. A 37 GB process is roughly 14% of the machine on its own, and this box also hosts long-lived model servers. Running both side by side is too tight: the big-model queue waits at its memory gate and skips. That's Failure 2 again, with the memory gate refusing instead of the lock.

So the conversion moved to the end of the chain. It now waits for every GPU queue in the batch. The big-model queue that used to wait on the conversion now waits on the other big-model queues. Re-chaining queues that are already running has two traps of its own:

• Killing a queue that others wait on starts the next waiter immediately. Its next poll comes back empty, so it starts in the old order, possibly right into the job you're moving. Change the successors' wait conditions first, then kill.
• Don't edit a waiting bash script in place. Bash reads a script from the file as it runs, so an edit under a live shell lands at a shifted offset. Edit the file, then quit and relaunch that queue.

What each primitive actually answers

• pgrep -f answers "does any command line match this regex right now?" That's a search. As a wait condition it gave us a false negative (a driver between runs, which ended in a panic) and a false positive (wrappers and other waiters, which ended in a deadlock).
• A mkdir lock with a pid file answers "does a live process hold this?" That's exclusion. Ours refuses rather than waits, so every caller has to decide what a refusal means.
• A free-memory gate answers "does this model fit right now?" That's admission, and it's the only one of the four that knows how big the job is.
• A wall clock answers "how long may this job hold the machine?"

None of them answers "in what order do these run?" We had been getting order as a side effect of pattern matching. Order needs one owner: either a single runner that works through the queue files in sequence, or a successor that waits on its predecessor's recorded PID with kill -0.

Two benchmark queues deadlocking on each other's pgrep is the kind of thing an on-site Rescue untangles before it becomes your next weekend.

Key takeaways

1. pgrep -f matches whole command lines, including launch wrappers and other waiters' pgrep calls. Our waiters polled on the same period, phase-locked, and none of them started.
2. Anchor to the interpreter and escape the dot, as in ^/bin/bash .*queue_gNN\.sh, and keep the waiting queue's own number out of its pattern.
3. The same primitive failed in both directions. It said "idle" between a driver's runs, which caused an overlap that panicked the box, and "busy" with nothing running, which caused a deadlock.
4. A lock that refuses, inside a loop that skips, loses measurements silently. Two GPU benchmarks printed skip behind a CPU job that never needed the lock.
5. Scope each protection to what it protects. The CPU-only job keeps the wall clock and the memory refusal and drops the GPU lock.
6. Find the real constraint before removing the apparent one. The biggest model needs about 79% free, so a 37 GB job next to it is too tight with or without a lock.
7. Re-chain before you kill, and relaunch waiting scripts rather than editing them in place.