./runbooks/apple-32gb — free runbook · apple silicon

32 GB: a 27B-class model, 4-bit, with room to breathe.

A free public runbook grounded in two source files — our internal benchmark log and our deployment configs. What fits in 32 GB, what doesn't, and the measured decode numbers behind the recommendation.

01 What fits — and what doesn't

02 Throughput — measured, but on a 256 GB Studio

These are the real decode figures for the 27B family, all on studio1 (256 GB M3 Ultra, 80-core GPU). They are not what a 32 GB Mac will produce — treat them as a ceiling, not a target:

Qwen3.8-27B decode — 256 GB Mac Studio M3 Ultra (80-core GPU), our deployment configs

  AEON 8-bit        dflash-mlx + DFlash2       prefill 397   decode 42.9   77% accept
  AEON 8-bit        mtplx MTP depth 3 (prod)   prefill 377   decode 34.3   2.1 tok/round
  AEON 4-bit (C)    dflash-mlx + DFlash2       prefill 400   decode 60.6   77% accept
  stock Qwen3.8 4b  dflash-mlx + DFlash2       prefill 398   decode 66.9   70–75%
  stock Qwen3.8 4b  plain mlx-lm               —             decode 40.6   —

32 GB Mac: no measured tok/s exists in our internal benchmark log or our deployment configs.     not covered

03 Memory headroom

14 GB of weights on a 32 GB box leaves roughly 14–18 GB for KV cache, macOS, and whatever else is running — comfortable, not infinite. Context is the variable that eats it: on the 125B MoE, KV cache ran ≈ 98 KB/token and 128k context ≈ 13 GB measured (our internal benchmark log) — a different model, so read it as a scale hint, not a 27B figure estimate.

And the same caution from our hardware notes applies at every tier: on the 256 GB Studio, wiring 224 GB (87.5% of RAM) to the GPU caused a kernel panic measured. Leave real headroom; don't wire the ceiling.

04 Recommended settings

The 4-bit requant is the 32 GB-appropriate build. Verbatim conversion result from our deployment configs:

[INFO] Using dtype: bfloat16
[INFO] Quantizing
[INFO] Quantized model with 4.501 bits per weight.
converted: 14G        # ~/dflash-eval/aeon-4bit  (4-bit, g64, 14 GB)

Serving — the real dflash serve command from production (shown 8-bit; swap in the 4-bit model for a 32 GB box, since the 8-bit ≈ 27 GB does not fit):

exec dflash serve \
  --model /Users/you/.mtplx/models/aeon-8bit-turbo-1 \
  --draft z-lab/Qwen3.8-27B-DFlash2 \
  --host 0.0.0.0 --port 8101 \
  --log-level INFO --diagnostics basic

Useful dflash flags recorded in the configs: --draft-quant (e.g. w4:gs64), --enable-thinking, --prompt-cache-size, --fastpath-max-tokens, --max-tokens, --chat-template. Offline env: HF_HUB_OFFLINE=1, HF_HUB_DISABLE_TELEMETRY=1. measured config.

05 What this page does not cover

SOURCES our internal benchmark log · our deployment configs · measured posts: benchmark the hardware first → · a 192GB model on a 256GB box →