Every GPU Out-of-Memory Error Was One sysctl

Three unrelated GPU allocation failures on a 256 GB M3 Ultra, all with RAM to spare. The ceiling: iogpu.wired_limit_mb, 75% of memory by default.

Summary

Our engine kept failing GPU command buffers with kIOGPUCommandBufferCallbackErrorOutOfMemory while running a 320B mixture-of-experts model on a 256 GB M3 Ultra workstation. Three different changes set it off: a smaller draft head for speculative decoding, a 64 MB scratch buffer from a kernel experiment, and larger prefill chunks. We gave each one its own explanation, but they had a single cause. Metal counts pinned weights, per-sequence state and every scratch buffer against one working-set ceiling. On Apple Silicon that ceiling comes from the sysctl iogpu.wired_limit_mb, which defaults to 196,608 MiB (75% of the machine) no matter how much RAM is free. The model pins about 185 GB of weights plus 6 GB of state before it allocates a single activation. Raising the sysctl to 245,760 fixed it until the next reboot, and a boot-time LaunchDaemon makes the change stick. The engine now prints its pinned and scratch totals against the limit at startup and warns within 10%.

Three failures that looked unrelated

They showed up one at a time, each tied to a different feature, and each got a local explanation.

The draft head. We ship a frequency-ranked draft head for speculative decoding, whose argmax runs over only the most frequent quarter of the vocabulary. On this model every decode with it enabled failed with the out-of-memory error, three runs out of three. The full head ran fine at 39.2–40.2 ms per pass. Our conclusion at the time was to leave this model out of the feature.

The kernel experiment. A split-K matmul tile needed a 64 MB scratch buffer. On a smaller model it simply lost on speed and was dropped. On this model, at 512-token prefill chunks, those 64 MB were enough to trigger a GPU out-of-memory. Our conclusion: the margin is thin at that chunk size.

Bigger chunks. 2,048-token prefill chunks were the obvious lever for this model's expert kernels. Its routing puts about 14 tokens on each expert in a 512-token chunk and 57 in a 2,048-token chunk. The first attempt failed with the same error, even after we removed a 4.8 GB attention copy that grew with the chunk. Our conclusion: find more scratch to cut.

That made three features and three workarounds. None was wrong as far as it went, but none of them asked which budget all three were overrunning.

RAM is not the budget the GPU checks

On unified memory it's natural to reason from the machine's total: 256 GB of memory, a model of about 185 GB, plenty left over. Free pages in vm_stat never entered into it.

Metal doesn't check against the machine's total. Every buffer the GPU uses is wired against the device's recommendedMaxWorkingSetSize: weights pinned with mlock and a residency set, per-sequence state and short-lived scratch alike. That value follows iogpu.wired_limit_mb, which defaults to 75% of physical memory. On this machine the default puts 68.7 GB (decimal) out of the GPU's reach, no matter how idle the rest of the system is.

The error doesn't tell you any of this. It names a command buffer, not a ceiling, and it reads the same whether the machine is truly out of memory or has tens of gigabytes free.

The units will mislead you

Three unit systems meet here, and we got caught between them:

Our first note on this said the model "sat at 191 of 192". The 191 was decimal gigabytes (185 GB of weights plus 6 GB of state, as the engine prints them), but the 192 was gibibytes. In the same units, the default ceiling is 206.2 GB. The diagnosis, a ceiling rather than RAM, still held, but the arithmetic didn't. You can't plan a chunk size against a margin computed in mixed units.

The fix, in three parts

1. Raise it now. The sysctl takes effect immediately, with no reboot:

sudo sysctl iogpu.wired_limit_mb=245760   # 240 GiB of 256; 16 GiB stays with macOS
sysctl iogpu.wired_limit_mb               # verify

2. Make it survive a reboot. The setting doesn't persist. A reboot quietly drops you back to 75%, and the first symptom is the same out-of-memory error on a configuration that worked yesterday. The persistent version has to be a system LaunchDaemon, because the sysctl needs root and shouldn't wait for anyone to log in:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>local.iogpu-wired-limit</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/sysctl</string>
    <string>iogpu.wired_limit_mb=245760</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>StandardErrorPath</key>
  <string>/var/log/iogpu-wired-limit.log</string>
</dict>
</plist>

To install it, copy the plist into /Library/LaunchDaemons/ with sudo cp, then run sudo launchctl bootstrap system /Library/LaunchDaemons/local.iogpu-wired-limit.plist. A user LaunchAgent is the wrong domain for this; four wrong diagnoses of a locked-out Mac covers why user agents aren't reboot-safe.

3. Print the budget at startup. Once weights are pinned and scratch is allocated, the engine prints one line with three decimal-GB figures: weights pinned, activation scratch, and the GPU working-set limit. If pinned plus scratch is above 90% of the limit, the line ends with a warning:

const double limit = device.recommendedMaxWorkingSetSize / 1e9;
const double used  = (pinned_bytes + scratch_bytes) / 1e9;
fprintf(stderr, "%.1f GB of weights pinned, %.1f GB of activation scratch; "
                "GPU working-set limit %.1f GB%s\n",
        pinned_bytes / 1e9, scratch_bytes / 1e9, limit,
        used > 0.9 * limit
            ? " (within 10%: raise iogpu.wired_limit_mb before growing the chunk)"
            : "");

The first configuration it warned about was one we actually shipped. After the scratch cut in the next post, 2,048-token chunks fit under the default limit: 185 GB pinned plus 8.8 GB of scratch against 206.2 GB, or 94%. It ran, it printed the warning, and the sysctl went up the same day.

The check has one blind spot. It adds up weights and scratch but not the per-sequence state or anything else the process wires, so a line with no warning doesn't prove you have headroom.

Didn't we tell you to leave this limit alone?

We did. A 192 GB model on a 256 GB box ends with: leave the wired memory limit alone, and if a model needs you to raise it, the model doesn't fit. That machine panicked, most likely because its limit had been hand-raised to 224 GB for a 192 GB download with nothing budgeted for anything else.

The rule we'd write now is narrower: raise the limit only for a process whose working set you can print before it runs. Our engine pins its weights so they never page out, allocates its activation scratch once at load, and reports both against the ceiling before the first token. Raising the limit to 240 GiB is still a bet that macOS and everything else on the box fit in the remaining 16 GiB. We panicked a machine twice in one day with long GPU-saturating jobs, so we don't treat that bet as free. The difference is that this time it's a bet with a printed budget.

The sysctl that was eating the GPU memory is the kind of trap the buyer's toolkit calls out before you buy.

Key takeaways

1. On Apple Silicon, a GPU out-of-memory error is measured against iogpu.wired_limit_mb, not RAM. The default is 75% of physical memory, and free pages don't enter into it.
2. Three failures in three features came from one ceiling. When the same error shows up across unrelated changes, look for the shared budget before you write the third workaround.
3. The sysctl resets at boot. A system-domain LaunchDaemon makes it stick. After any reboot, check the sysctl before you blame the engine.
4. Keep the units straight. The sysctl is in MiB, the spec sheet in GiB, and most logs in decimal GB. 192 GiB is 206.2 GB, and our first note mixed the two.
5. Print pinned plus scratch against recommendedMaxWorkingSetSize at startup, and warn within 10%. The first warning it raised was on a configuration we shipped.
6. Raise the limit only for a process whose working set you know before it runs. For anything else, the old rule from post 24 still applies.