A 192 GB Model on a 256 GB Box

Three independent blockers on a model download, and the one that ended in a kernel panic: raising the wired GPU memory limit to make room.

Summary

Someone tried to run a new ~180B-class mixture-of-experts model on a 256 GB Mac Studio by fetching the 8-bit quantization — 192.2 GB of weights. Three independent things were wrong with that plan, each sufficient on its own. The machine kernel-panicked a few hours later, and the most likely cause is the step taken to make room: hand-raising the wired GPU memory limit to 224 GB. Here's the whole failure, and what the right version looks like.

Three blockers, each fatal

1. Nothing was ever downloaded. The cache directory was 4.0K — an empty scaffold with zero snapshots, zero blobs, and not even a partial-download file. A fetch started and received no bytes.

Worth knowing: the shell history had no record of the command either, because a kernel panic loses unflushed history. When post-morteming a crash, don't treat an absent history entry as evidence something didn't happen.

2. The architecture was unsupported. The model declares a brand-new architecture type and ships its own 35 KB implementation file for remote-code loading.

The standard MLX language-model package does not do trust_remote_code. It dispatches on the declared model type to a class inside its own package. We checked all five virtual environments on the machine, across four package versions: zero copies of the required module.

The load would have failed with "model type not supported" — after downloading 192 GB.

There's a trap adjacent to this one. The package does ship a module for a similarly-named earlier architecture. They are different architectures. A quick grep for a familiar-looking filename would have produced a false positive on support.

3. The arithmetic doesn't work. 192.2 GB of weights on a 256 GB machine leaves nothing for KV cache, activations, or the operating system. And the model is multimodal, so the vision path is more memory again. (The general version of this arithmetic: how much RAM you actually need for local LLMs.)

The 4-bit build at roughly 96–112 GB is the only realistic option, and it always was.

One more signal we should have read: the 8-bit repository had zero downloads. Thirty-three files, uploaded the day before, nobody had ever fetched it. On a public model hub, being the first person to download a quantization is information — usually that it isn't the one people run.

The kernel panic

Timeline: the empty cache directory is stamped 08:45. The machine watchdog-panicked at approximately 13:26, with the GPU subsystem and MLX named in the panic body, then sat at the pre-boot encryption prompt until someone unlocked it at 15:52.

In between, the wired GPU memory limit had been hand-raised to 229,376 MB — 224 GB of 256.

Which is exactly what you'd do to make room for a 192 GB model.

Wired memory is not reclaimable. The kernel's memory-pressure mechanisms cannot evict it. Setting the wired limit to 87.5% of physical RAM removes the system's ability to respond to pressure at all, and overrunning it stalls the whole machine rather than killing one process.

On a discrete-GPU system, an oversized model gets you an allocation failure. On unified memory, the same mistake takes the operating system with it. The failure mode is qualitatively different and much worse.

A footnote that matters: a boot-time service on that machine re-sets the wired limit to a safe value at every boot. So the hand-raised value never persisted anyway — it bought a few hours of danger and no lasting capability. If your fleet has a policy daemon enforcing a limit, working around it manually gets you the risk without the benefit.

The right version of this plan

The follow-up research found the model is runnable on this hardware. The runtime was the blocker, not the machine.

The model is ~180B-class with 6B active — 512 experts, top-10 routed plus one shared, 48 layers, hybrid linear attention with a sparse-attention variant, native 262k context.

And the KV cache is tiny: 2 KV heads × 256 head dimension × 48 layers is about 98 KB per token, so 128k of context is roughly 13 GB. For a model this size that's remarkable, and it changes the memory budget completely — nearly all of it goes to weights.

Runtime options, checked rather than assumed:

• The standard MLX package: no. The pull request adding support was still open. Vendoring one file is a workaround; a reviewer had validated it on this exact chip at 111 GB peak and ~385 tok/s prefill.
• A community MLX server: yes, with first-class support for the architecture, continuous batching, tiered memory-and-SSD KV caching, and speculative decoding at 2.33–2.62×. OpenAI-compatible, so it drops in behind an existing gateway unchanged.
• A vision-language MLX package: yes, text and vision.
• llama.cpp: yes, support merged the same week, with perplexity within noise of the reference.

And the build to actually take, measured against the hub API rather than trusted from a README:

Two-bit builds exist at 68–80 GB if headroom ever becomes the binding constraint.

Do not convert locally. The bf16 base is 360 GB — a 256 GB machine cannot hold the conversion, let alone the result.

The order to do this in

The whole failure is an ordering problem. Every check that would have stopped it is cheaper than the step it would have stopped.

1. Settle architecture support first. Read the config's declared type, grep your runtime for it, and confirm it's the same architecture and not a similarly-named neighbor. Costs one minute, no download.
2. Do the memory arithmetic including KV, activations, and the OS. Weights are not the budget — the per-model-size numbers are written down in the buyer's toolkit and serving configs.
3. Check download counts and file listings on the hub before assuming a quantization is the standard one.
4. Pick the quantization that fits with headroom, not the one with the best quality number.
5. Leave the wired memory limit alone. If a model needs you to raise it, the model doesn't fit.

Key takeaways

1. Check runtime architecture support before downloading anything. Ours would have failed to load after 192 GB, and the check is a one-minute grep.
2. A similarly-named module is not support. Two architectures one character apart in a filename would have given a confident false positive.
3. Weights are not the memory budget. KV cache, activations, and the OS all need to fit; multimodal paths need more again.
4. Wired memory is not reclaimable, and unified memory means the OS shares the failure. Overrunning it panics the machine instead of failing one allocation.
5. Zero downloads on a public quantization is a signal. Being first is usually a sign you've picked the wrong build.
6. A panic loses unflushed shell history. Don't read an absent command as evidence it wasn't run.
7. Working around a policy daemon gets you the risk without the benefit. Ours reset the limit at boot regardless.