Can SSD Offload Fit a Model Twice Your RAM? The Arithmetic.

Streaming mixture-of-experts weights from NVMe, priced before writing code: at ~6 GB/s against ~800 GB/s of bandwidth, anything over the limit decodes at 1–5 tok/s.

Summary

A 256 GB M3 Ultra can give the GPU at most 240 GiB (the post on the wired limit covers why), and open mixture-of-experts models keep arriving with expert pools from about 290 GB to over 800 GB. The usual idea is to keep the experts on the internal NVMe drive and stream in whichever ones the router picks. We priced that before writing any code, so the decode and prefill figures here are estimates, not measurements. The SSD reads about 6 GB/s and decode streams resident weights at about 800 GB/s, so a byte read from disk costs as much time as 130 bytes read from memory. Holding 10 tok/s takes at least 91% of the expert pool resident, which means streaming only rescues a model that is already within about 10% of fitting. A model around twice the machine's memory comes out at 0.8–1.4 tok/s. What works instead is fewer bits per expert, and after that a second machine.

Update (2026-09-25): Measured since: the full checkpoint is 475 GiB. That is 276 GiB of experts with their scales and 189 GiB of Engram lookup tables, which the resident figures above leave out. Our plan reads them from the file on the CPU, about 12 KB per token per layer. The expert format that fits turned out to be a 2- and 3-bit K-quant mix at 2.94 bits, about 199.6 GB, rather than 2.5 bits. On the previous V4-Flash it cost +0.031 nats against native fp4, with decode no slower.

The two rates that decide it

• Memory bandwidth. We regressed decode time against streamed weight bytes across three quantizations of one model and got 798 GB/s, 99.8% of the chip's rated bandwidth. Decode is already at that ceiling.
• SSD. Loading 147 GB of experts takes 25–30 s, which is about 5–6 GB/s as one sequential stream. The drive is rated at 6–7 GB/s. We use 6 in the estimates below.

Unified memory adds one more constraint. There is no second memory tier to offload into, only one pool. With 240 GiB wired to the GPU, about 16 GiB is left for macOS and the page cache combined. Designs that use the OS page cache as the expert cache have almost no page cache to work with here. The only design that works is a set of explicit pinned slots filled by direct reads.

The model

Decode time per token = the resident pass + (the expert bytes that miss / the SSD rate).

The assumptions are all listed here so they can be challenged:

• Expert budget: 240 GB wired − ~9–20 GB of dense weights − ~10 GB of KV cache and scratch ≈ 215 GB.
• Uniform routing. DeepSeek-style load balancing and a shared expert both push routing toward uniform. A study of 20 MoE models found that shared experts reduce the routing locality an expert cache depends on.
• Prefill touches everything. A 2,048-token chunk hits every expert in every layer, so each chunk has to read in the whole non-resident pool.

The prefill column is where streaming breaks first. Even the best row that needs the SSD at all spends 12 seconds per 2,048-token chunk just reading, which caps prefill at roughly 170 tok/s.

The 91% line

Work backwards from 10 tok/s:

• 10 tok/s is 100 ms per token. The resident pass takes about 35 ms, which leaves the SSD about 65 ms.
• At 6 GB/s, 65 ms is about 0.4 GB of expert reads per token.
• V4.1-Flash at fp4 moves 4.5 GB of expert bytes per token, so the miss fraction can be at most 9%, which means at least 91% resident.
• 215 GB / 0.91 gives an expert pool of about 236 GB at most.

So SSD streaming only rescues a model within about 10% of fitting. That last 10% is cheaper to remove some other way, either with lower-bit experts or by dropping the draft model.

Why prefetching doesn't change the answer

Published expert-prefetch schemes guess the next layer's experts, either from the current layer or with a draft model. Their gains are real, but they were measured on PCIe or CPU-memory tiers at 20–60 GB/s, where compute and transfer are within 10× of each other. Here, at 74% residency, one layer reads about 30 MB: ~5 ms of I/O against ~0.7 ms of compute. A perfect predictor would hide the ~0.3 ms routing sync and start the reads ~0.7 ms early. That fixes latency, but the limit here is bandwidth, and every wrong guess spends the scarcest resource.

Two more factors make it worse:

• Batching doesn't spread the cost. A step of 4 tokens touches about 4× the experts.
• Speculative decoding multiplies reads. Verifying drafted tokens reads more experts per step. One Apple-Silicon streaming project found multi-token prediction purely compute-bound and limited drafts to one token.

An expert cache only helps if routing locality beats uniform by 5–10× on the largest model, and the literature says DeepSeek-class routing is the worst case for locality.

What others measured on Apple Silicon

Only a few projects publish numbers with the SSD as the expert tier:

oMLX and SwiftLM both ran models with small experts on machines with free page cache, and RotaryQuant's pool is about twice its memory budget. None of those conditions holds for a 500 GB model on a box with 240 GiB wired and 16 GiB left over.

Our engine already has a crude version of this, an LRU expert cache used as a fallback when a model doesn't fit alongside other services. With a 151 GB cache budget and mostly cache hits, a 284B MoE with its drafter ran at 229 ms per pass (4.5 tok/s) with 44 command buffers per token. Another model with a 9.5 GB cache took 485 ms per pass. The SSD isn't the main cost there. Wrapping anonymous pool pages for the GPU cost 97 ms per expert matmul, because the GPU re-faults pages it doesn't own. That's the same trap as the memory-mapped weights that our engine's first bring-up re-faulted on every token. The output is still byte-exact, so the fallback works for correctness runs but not for serving.

What actually fits bigger models

1. Fewer bits per expert. DeepSeek-V4.1-Flash with 2.5-bit experts takes about 186 GB resident (170 GB of experts and 9 GB of dense weights, plus 7 GB for the drafter), for an estimated 30–40 tok/s. This buys fit, not speed. Expert bytes per token drop from 4.5 GB to 2.65 GB, but the 8-bit dense share stays the same. Quality is unknown. Our earlier 2-bit GPTQ runs on dense MLPs left 15–29% relative output error per projection, so a 2-bit expert build gets scored against the full-precision model before it ships.

2. A second machine. Pipeline parallelism splits the layers across two boxes. Only one hidden state per token crosses between them, about 10–20 KB, which even 10 GbE handles easily. That doubles wired capacity to 2 × 240 GiB at single-box decode speed. It's enough for V4.1-Flash at native fp4 (~298 GB) with no 2-bit quality question, or Kimi-K2 at about 2.5 bits (~330 GB). V4-Pro still doesn't fit. MLX's distributed backend supports RDMA over Thunderbolt 5 from macOS 26.2. We estimate 2–3 weeks of work, and none of it is built yet.

The cheap experiment before writing any code

Two inputs to the table above are assumptions, and both can be measured in half a day with no kernel work:

1. Miss fraction versus residency on a DeepSeek-class router. The engine's LRU cache already counts hits and misses; print those counts at 90%, 75% and 50% residency, over a fixed corpus and a 2K-token prompt.
2. Random-read rate at expert granularity. The 5–6 GB/s figure comes from one sequential stream. Streaming means about 18 concurrent random reads per layer of 18.8 MB (fp4) or 11 MB (2-bit) records. Measure at queue depths 1, 8 and 24, with the other services running.

The decision rule: build real SSD streaming only if some target model then predicts at least 10 tok/s decode and 100 tok/s prefill. By the arithmetic above, every model on the list fails that test.

Whether SSD offload actually fits a model twice your RAM is arithmetic, not hope — the on-site setup works it out on your machine.

Key takeaways

1. On an M3 Ultra, reading a byte from NVMe costs about as much time as reading 130 bytes from memory: ~6 GB/s against ~800 GB/s. Every streamed expert pays that ratio.
2. Streaming holds 10 tok/s only at 91% residency or better. That means a model within ~10% of fitting, and cutting that 10% with fewer bits is cheaper.
3. Prefill fails first. Each 2,048-token chunk touches every expert, which is 12 s of reads per chunk at 74% residency.
4. Unified memory has no spare tier. At 240 GiB wired, macOS and the page cache share the remaining 16 GiB.
5. Prefetching fixes latency, not bandwidth. With 5 ms of I/O per layer against 0.7 ms of compute, there's almost no compute time to hide the I/O behind.
6. Fit bigger models with fewer bits first, then with a second machine, and measure miss rate and random-read rate before building any streaming path. The published Apple-Silicon results used small-expert models with free page cache, and they don't carry over to a 500 GB model.