We Panicked the Same Machine Twice in One Day

A guard with a wall-clock cap and a free-memory floor didn't prevent the second kernel panic. A userspace watchdog is starved by exactly what it exists to detect.

Summary

A long inference run pinned 169 GB of unified memory and saturated the GPU for eleven minutes on a 256 GB workstation. The kernel panicked — not from an out-of-memory kill, but because WindowServer missed its watchdog check-in for 120 seconds. We built a guard the same afternoon, with a wall-clock cap and a free-memory floor, and then panicked the identical machine again forty minutes later with the guard running. The wall-clock half didn't fire because the guard is a userspace process and was starved along with everything else. The memory half didn't fire because we had lowered its threshold by hand to get past a box that idles near the limit.

The panic

panic(cpu 0): userspace watchdog timeout: no successful checkins
from WindowServer (2 induced crashes) in 120 seconds

This is not an out-of-memory condition, and reading it as one sends you to the wrong fix. Nothing was killed. The kernel's userspace watchdog took the whole machine down because a system process it supervises stopped responding.

What provoked it: an inference run whose sidecar pinned 168.7 GB — mlock plus a GPU residency set, so the kernel cannot page it out — and kept the GPU saturated for eleven minutes. Free memory fell to 22% and swap grew from 14 GB to 43 GB. WindowServer needs both GPU time and memory to check in. It got neither.

Duration, not peak

The obvious lesson would be "169 GB is too much on a 256 GB box." That's wrong, and we had the data to know it was wrong: a 158 GB baseline configuration had run repeatedly, dozens of times, at 32 tokens per run — about a minute each — with no trouble at all.

It is the duration of GPU saturation that kills the machine, not the peak footprint. A large allocation that releases the GPU regularly is survivable. The same allocation held under continuous load starves the system processes that must be scheduled to keep the kernel happy.

That reframes the guard you need. A memory ceiling is the intuitive control and it is the wrong one. The control that matters is how long you hold the machine down.

The second panic, forty minutes later

We wrote the guard: a wall-clock kill and a free-memory floor, both enforced by a supervisor around the run.

Then, launching the next experiment, the floor refused to start because the box idles near 20% free. So we passed a lower threshold on the command line to get past it, on a machine that had already panicked once that day.

It panicked again.

Two separate failures, and the second one is the interesting one:

The threshold override was the human failure. A guard you can wave through at the prompt is a guard that will be waved through, and it will be waved through at exactly the moment it's most needed — when the machine is already tight, which is why the guard is complaining. If a floor is right, it should not have a flag.

The wall-clock cap was the design failure. The guard is a userspace loop that sleeps ten seconds between checks. When the system thrashes, that loop is starved on the same runqueue as WindowServer, for the same reason. It cannot wake up, so it cannot measure, so it cannot kill anything. It was, in effect, not running during the entire window it was written to cover.

A userspace watchdog cannot protect a machine from a workload that starves the kernel's own watchdog clients. The starvation is the failure mode; anything scheduled by the same kernel is inside it.

What actually holds

The only reliable lever is not starting the run.

• Never start a ~169 GB pinned workload on a box whose free memory is already at or below 25%.
• Never lower the floor to get past a refusal. The refusal is the measurement working.
• Keep any single GPU-saturating run to a couple of minutes, or leave headroom well above 22% free.
• Watch swap growth, not just free percentage. Swap going 14 → 43 GB in eleven minutes is the same signal, arriving earlier.

Note that the existing watchdog's free < 12% floor never fired in either panic, because the machine died at 22%. A floor calibrated to "how close to zero can we get" is calibrated for the wrong failure. The box does not die when memory runs out; it dies when the kernel's clients can't get scheduled, which happens with a fifth of memory still free.

"I freed memory" is not "there is headroom"

Before the second run we took down the other model servers on the box, which releases about 64 GB.

Free percentage still read 20 immediately afterwards.

The pages had been released and the page cache had not been reclaimed yet. If you take that reading as your go/no-go, you conclude that stopping the servers did nothing and you either give up or override the floor — which is precisely what we did. Wait for free percentage to actually rise before starting. The number lags the action by long enough to be misleading.

The cost of a panic is not the panic

The reboot is the cheap part. What follows is:

1. The machine comes back to the encrypted-disk pre-boot screen, where nothing runs — no VPN, no jobs — until somebody unlocks it. See the four wrong diagnoses of a locked-out Mac.
2. The panic is invisible from anywhere else. The mesh network simply reports the host offline, which looks like a hundred other things.

So: after any unexplained blackout, check ls -t /Library/Logs/DiagnosticReports/*.panic before theorizing. We spent weeks blaming automatic OS updates for reboots that were panics the whole time.

Recovery itself was clean, and worth recording because it validates an earlier migration: swap read 0.00M on the fresh boot, and all six services restarted themselves with nobody logged in, because they are system daemons rather than user agents.

One trap on that recovery: two of those services take minutes to load model weights. Their ports are closed while the processes are already running. A port-based health check reports them down and a naive supervisor will restart them into a loop. Check for the process, not the socket, during the startup window.

A watchdog that starves alongside everything else isn't a watchdog — it's on the list the setup audit works through.

Key takeaways

1. no successful checkins from WindowServer is not an OOM. Nothing was killed; the machine was taken down for unresponsiveness. Reading it as memory exhaustion sends you to the wrong fix.
2. Duration of GPU saturation kills the box, not peak footprint. The same 158 GB run was fine dozens of times at one minute and fatal at eleven.
3. A userspace watchdog is starved by the condition it watches for. Ours slept ten seconds between checks and never got scheduled during the window it existed to cover.
4. A guard with an override flag has no floor. We disabled the memory half by hand, on a machine that had already panicked that day, to get past a correct refusal.
5. Calibrate for the real death point. A free < 12% floor was useless on a machine that dies at 22%.
6. Freed memory is not immediately visible as free memory. The page cache lags; wait for the number to rise before treating it as headroom.
7. A slow-starting service has closed ports and a live process. Health-check on the process during startup, or your supervisor will restart it forever.