Running a private LLM for a small law firm on a Mac Studio — what actually works

A field-notes account of sizing, model choice, document handling, and network isolation for a small law firm running a local model on Apple Silicon, including the parts that still do not work.

The short version of why a law firm ends up running a local model instead of a chat subscription is that confidentiality is a duty, not a preference. The ABA's Formal Opinion 512, issued July 29, 2024, walked the Model Rules across generative AI and concluded there is no special AI exemption: competence (Rule 1.1), confidentiality (Rule 1.6), communication (Rule 1.4), supervision (Rules 5.1 and 5.3), candor (Rules 3.1 and 3.3), and fees (Rule 1.5) all apply to a model the way they apply to a fax machine. Of the six, confidentiality decides which AI you can use, because it turns on where client information goes and who retains it. A model running on the firm's own hardware changes that question from "do we trust this vendor's terms" into "do we trust our own server room" — a duty the firm was already satisfying before AI existed.

Sizing: concurrency, not headcount

The mistake people make when sizing a shared box is starting from payroll. A twenty-person office is not twenty simultaneous users. Most of the day it is one or two, with a lunchtime spike. What sets the hardware is how many people hit the model at the same instant — concurrency, driven by workflow, not staffing. A paralegal drafting a letter, an associate extracting dates, a receptionist summarizing a file: short, bursty requests followed by a minute of reading.

Once you size for concurrency, the answer is modest. Total unified memory is the binding constraint, and the budget has three parts: model weights, per-user KV cache, and OS headroom. As a labeled-as-estimate guide: 32 GB runs a 7–8B model for a handful of users; 64 GB runs a 13–30B model with room for several concurrent users, the sweet spot for most small offices; 96 GB buys longer contexts and more simultaneous users; and 128–256 GB is where a single workstation replaces a server room.

Memory decides whether the model fits; bandwidth decides whether it feels instant, because decode speed is roughly bandwidth divided by bytes per token. The Mac Studio M3 Ultra moves 800 GB/s of memory bandwidth; NVIDIA's DGX Spark moves 273 GB/s on its GB10 chip. For a small office the M3 Ultra's headroom is usually overkill for the model size and exactly right for making a few concurrent users feel immediate.

Model choice: fit the quant to the memory, not to the spec sheet

The most common failure is picking a quantization because it has the best quality number, then finding it does not fit. A 7–8B model at 4-bit is roughly 4–5 GB of weights; a 30B-class model lands in the high teens to low twenties; a 70B-class at 4-bit is around 40 GB — rule-of-thumb estimates, not measurements. The measured points are sharper: a 27B-class model requantized to 4-bit came in at 14 GB on disk, and at 8-bit the same class is roughly 27 GB, which does not fit in a 32 GB machine once you reserve anything for the OS and KV cache.

The failure mode worsens as the model grows. A ~180B mixture-of-experts model has been documented hitting three independent blockers: nothing actually downloaded, the runtime did not support the declared architecture (which would have failed after a 192 GB fetch), and the 192.2 GB 8-bit build left nothing for KV cache, activations, or the OS. That attempt ended in a kernel panic, most likely because the wired GPU memory limit had been hand-raised to 224 GB of 256 to make room. On a discrete-GPU box an oversized model gets you an allocation failure; on Apple Silicon the same mistake takes the whole machine down. The rule that falls out is clean: if a model needs you to raise the wired limit, it does not fit. The 4-bit build at roughly 94.5–111.7 GB was the only realistic option all along.

For a law firm the practical answer is a 7–30B-class model at 4-bit, sized so weights plus KV cache for a realistic peak of two or three concurrent users leave real headroom. On a 256 GB Studio the measured decode for the 27B family runs from about 40 tokens per second on plain MLX to about 67 with a speculative-draft head — fast enough for the drafting, summarization, extraction, and classification work a firm actually does. The model is not the expensive part; you pay for the machine and the labor.

Retrieval over on-prem documents

This is the least glamorous part of the setup and it needs the most honesty. A firm's matter files are long, and two facts from the field notes matter more than any product feature. First, KV cache is the concurrency term: on the ~180B MoE it measured about 98 KB per token, so 128k tokens of context is roughly 13 GB of KV cache. Second, prefix caching only helps if the front of the prompt stays byte-identical between turns: the notes document a 47,000-token agent prompt getting 49.8% cache reuse because a JSON tool-catalog array mutated at token 1,067, invalidating everything upstream of a carefully designed stable prefix — about 27× of the time-to-first-token sat behind that one volatile span.

In law-firm terms, the naive approach of loading an entire matter file into context works until the file is long, at which point either the KV cache overflows the budget or the prompt changes every turn and the cache never warms. The workable pattern is to keep the context for what is stable — system prompt, instructions, tool definitions, in a fixed order — and retrieve only the passages a query needs, appended at the tail where their churn does bounded damage. That means a small retrieval index over the firm's own documents, run locally too, because the point is that no document leaves the machine. The notes are direct about the tradeoff: this is a project, not an appliance, and one request with a huge context can stall everyone else on a shared box — a scheduling and configuration problem, not a hardware one.

Network isolation

The cleanest way to satisfy the confidentiality duty is to make disclosure structurally impossible rather than contractually unlikely. A local model works with no internet connection at all, which matters in a courtroom-adjacent office or anywhere with a hostile network. The pieces are mundane: the serving stack runs against loopback or the LAN, not the public internet; model downloads happen once on a separate step with telemetry disabled — the recorded flags are HF_HUB_OFFLINE=1 and HF_HUB_DISABLE_TELEMETRY=1 — after which the machine never phones home to use the model; and access control, encryption at rest, and audit logging replace the vendor's terms as the thing you rely on.

This is the honest flip side of "local fixes the disclosure half." It fixes that half completely — no third party, no retention, no training on client data, and for tax work the IRC §7216 offshore-consent question collapses because nothing goes offshore. It fixes nothing else. Verification is still the lawyer's job: a hallucinated citation is the practitioner's error, not the model's, under both Opinion 512 and the IRS's Circular 230 guidance. Security becomes your job instead of the vendor's — patching, access control, backups, physical security. A local model trades a vendor-trust problem for an operations problem.

What does not work

A local model is not a better ChatGPT; it is a different thing. You give up the frontier model, which remains ahead of open-weight models on the hardest open-ended reasoning. You give up image generation and integrated web search. You give up effortless updates — a local model is frozen until someone downloads a new one, a reproducibility feature and an operations burden at once. A firm that needs the strongest general-purpose model for hard open-ended work should keep a cloud subscription for that job and run the local model for the confidential, regulated work: two tools with different failure modes, not competitors. On the hardware side, avoid buying too little memory (a box that swaps is worse than no box) and too much compute for a memory-bound workload. Before spending anything, run a three-question filter on each proposed use: does client information leave the firm, who verifies the output, and who is accountable when it is wrong. A local model makes the first question easy and does nothing for the second and third.

Notes: RobotNurse is a New York City business that installs and maintains private, on-site AI systems for regulated small organizations; the figures in this piece come from its published field notes and sizing guidance. https://robotnurse.io/