The Machine Was Up. Everything On It Was Dead.

93 background jobs vanished while the host stayed up with unbroken uptime and working SSH — all in a launchd domain that only exists while someone is logged in.

Summary

A headless Mac running ~93 background agent jobs went dark. Every outward symptom said the machine had crashed: it disappeared from the VPN, nothing responded, no job was running. The machine was fine — 15 days of unbroken uptime and working SSH. All 93 jobs lived in the gui/501 launchd domain, which exists only while a graphical login session is active. The disk filled, the login session died, and the entire fleet went with it while the kernel never noticed.

The trap

macOS launchd has multiple domains. LaunchAgents in ~/Library/LaunchAgents load into gui/<uid> — a domain tied to an active console session.

For a desktop Mac that's correct. For a headless machine running background services, it means your entire service fleet is a child of a graphical login you never use and can't restore over SSH.

Every outward symptom says "the machine is dead":

• The VPN daemon also dies (it couldn't write state to a full disk), so the host vanishes from the network entirely.
• Nothing responds on any service port.
• Nothing is running.

Meanwhile the plists are all still on disk. Nothing is disabled. The kernel is fine. Uptime is unbroken.

Diagnosing it over SSH

The whole recipe, and every step distinguishes a real hypothesis from a plausible wrong one:

stat -f "%Su" /dev/console — root means no GUI session. A live session shows the user.

Use this rather than who, which is unreliable here. Ours showed a stale console entry from two weeks earlier and would have led us to the wrong conclusion.

ps -u 501 | wc -l — about 13 processes means no desktop. A real session has hundreds, plus a window server and a dock.

launchctl print gui/501 — Could not print domain: 125: Domain does not support specified action.

That error is the finding. It doesn't mean a job is missing; it means the domain itself is gone.

launchctl print-disabled user/501 — confirms nothing is actually disabled, which rules out the first theory everyone reaches for.

The single clearest indicator turned out to be a drift watchdog we'd written for an unrelated reason. It reports DRIFT: not-loaded -> <85 labels> and it keeps running, because it isn't in the GUI domain. A monitor that survives the failure it's monitoring is worth more than a more sophisticated one that doesn't.

What doesn't work, so you don't spend a day on it

• launchctl kickstart — only works on already-loaded jobs. These aren't loaded.
• launchctl bootstrap user/501 <plist> — fails for every agent with Bootstrap failed: 5: Input/output error. launchd will not accept GUI-domain agents from an SSH context. Only 3 of 93 plists even set a session-type restriction, so the plists aren't the blocker. The domain is.
• Legacy launchctl load -w — silently no-ops. No error, nothing loaded. The worst of both.
• sudo requires a password, so you can't enable screen sharing remotely to get a GUI session that way.

We concluded "nothing can be done remotely." That was wrong.

Worth flagging as its own item, because we wrote it down as fact and it cost us.

launchd is not required to run a process. nohup <cmd> > log 2>&1 < /dev/null & disown survives SSH logout. Confirmed by starting a process in one session and finding it alive in a later one — a one-line test that settled more than any documentation.

One trap in verifying that: macOS has no setsid. A test written with it fails in a way that looks exactly like the process dying on logout, which is how we talked ourselves into the wrong conclusion the first time. Use plain nohup.

cron runs as root, independent of any GUI session. It's the headless replacement both for launchd scheduling and for KeepAlive — a watchdog entry that restarts a dead daemon covers most of what you lose.

To recover a job's actual command from its plist:

python3 -c 'import plistlib; p=plistlib.load(open("<plist>","rb")); print(p["ProgramArguments"])'

The durable fix

Move everything to /Library/LaunchDaemons — the system domain, root-owned, with an explicit UserName so jobs still run as the right account.

That domain loads at boot with no login required, and it keeps full-disk encryption on. Seventy-nine of eighty-one jobs converted cleanly with a script that adds the user and group keys and strips the graphical-session-only keys.

Three things learned during the cutover:

Move the old agents out, don't leave them. If both copies are active, a console login runs every job twice. For jobs with real side effects — anything that sends, orders, or writes — a duplicate run is much worse than a stopped one.

Manually-started processes hold the ports. Eight daemons wouldn't start with Address already in use, because our own emergency nohup copies were still running. Kill the orphan and let KeepAlive reclaim the port. One job self-healed because its plist passes a --replace flag, which is a nice pattern to copy.

One job ignored SIGTERM entirely and needed kill -9. Worth finding out during a planned migration rather than during an outage.

Also surfaced by the conversion: two plists pointed at files that no longer existed. Pre-existing rot that had been silently failing, and that fails identically under the new domain. A migration is a decent excuse for an inventory.

Auto-login is not the fix

It's the obvious suggestion, and it's wrong here for two independent reasons.

Full-disk encryption forbids it. macOS won't enable auto-login while FileVault is on. Turning it off to get auto-login means giving up disk encryption on a machine holding credentials — a much worse trade than the problem it solves.

It wouldn't have helped anyway. The auto-login setting is consumed at boot. Our outage was a session death with the machine still up for 15 days. Auto-login would never have fired.

Two reasons is the useful part. The first is a policy objection someone can argue with. The second means the fix simply doesn't address the failure mode, which ends the discussion.

The tooling worth writing

Three small scripts came out of this and all three earned their keep:

A plist-driven KeepAlive replacement — reads every plist declaring KeepAlive, checks each via pgrep -f, restarts what's down. Supports a dry-run, and refuses to start anything when free disk space is below a threshold. That last guard matters: the original outage was a full disk, and blindly restarting 93 jobs into a full disk reproduces it.

A disk guard — reclaims stale snapshots and temp trees older than three days, warns below a threshold. Runs hourly from cron, which is to say, outside the domain that can die.

A plist-to-crontab translator — converts scheduled jobs to cron lines. 73 of 74 convert; the one holdout runs every 30 seconds and cron's floor is one minute. It writes a proposed crontab and installs nothing, which is the right default for a tool that could double-start a fleet.

One parsing gotcha: Python's plistlib rejects -- inside XML comments, which several of these plists contain. plutil and launchd both accept them. Fall back to plutil -convert json rather than concluding the plist is malformed.

A machine that's up while everything on it is dead is a textbook on-site Rescue: diagnosis first, verdict in writing, then the fix.

Key takeaways

1. LaunchAgents live in a domain that dies with the GUI session. On a headless Mac, use /Library/LaunchDaemons from the start.
2. stat -f "%Su" /dev/console is the diagnostic. root means no session. Don't trust who.
3. "Domain does not support specified action" means the domain is gone, not that a job is missing.
4. You cannot bootstrap GUI-domain agents over SSH. Every launchd path fails; two of them fail silently.
5. nohup … & disown works, and macOS has no setsid — a test using it fails misleadingly and will convince you nothing is possible.
6. Move old plists away during a migration. Two active copies means every job runs twice on the next login.
7. Auto-login is incompatible with full-disk encryption and wouldn't fix a mid-uptime session death anyway.
8. A watchdog outside the failing domain is worth more than a better one inside it. Ours was the clearest signal we had.