./posts — every field note, in one place
Every published note from the notebook, newest first. Dated, tested, and honest about what broke.
A conversion driver without an `if __name__ == "__main__"` guard, a worker pool that starts with spawn, and every worker re-ran the whole conversion. Three more converter bugs from the same day, none of which the test fixture could reach.
Our renderer wrote the previous generation's tool-call format for a model trained on a different one. The model copied the placeholder words from the instructions, or ended its turn with one token. The fix was to port the vendor's encoder step for step and test it byte for byte.
Our engine's copy of a model cannot finish its own tool-call tags; another engine running the same prompt can. Here is everything we ruled out, the one test that 'failed' by 10% without finding anything, and the suspect our reference cannot see. Still open.
A model whose hyper-connection streams peak around 2.0e6 on an ordinary prompt cannot store them in fp16, whose ceiling is 65,504: every logit becomes NaN. Our fixture never left fp16's range, so no test could see it. The fix is preventive, and it is not the fix for our open copy bug.
Three different ways of switching thinking off all left one model thinking at its maximum setting, because its chat template never reads any of them. It reads only reasoning_effort, and treats every value except two as 'Max'. A two-render check at startup finds this in any template.
Three agent conversations against two serving slots: time to first token grew from 16 to 61 seconds a turn. Against four slots it stayed flat at 1.6-1.9 seconds, and the prefix cache we had just switched on recorded zero hits.
A daily inference-server crash got two mitigations, both matched to the symptom by name, both 'validated' by a load test that passed 12 of 12. One mechanism was impossible, the other flag routed traffic into the failing path, and one of them cost 24% of decode.
Our calibrated quantizer had never once run on a mixture-of-experts model, because the dispatch guard looked at layer 0 and layer 0 is dense. The tell was a calibrated arm that tied its baseline to four decimal places.
A profile said a quarter of every decode pass was un-overlapped host work. An unprofiled control arm, which costs ninety seconds, said it was 1.6%. The 26% was the profiler's own synchronization.
A clean A/B, two rounds, token-identical arms, +23.3% — measured on a conversion with the model's speculative decoding, sparse attention, coprocessor offload and prefix cache all switched off. The other machine's copy had them all on.
+25% decode at one sequence, −27% at four. The draft head barely engages once sequences batch — 126 drafted passes solo, 1 batched — while still paying for itself. Which way you set it is a question about your traffic, not your engine.
118 tok/s at a 30-token prompt, 35.7 at 86,000. Decode halves for every 4× of context and prefill halves past 20k — and on this engine there is no configuration lever for it, because every cache compressor drops you off the fast path.
The real question isn't whether local AI is cool, it's whether it solves a workflow for your business. An honest decision guide: the two tensions, five signs it's right, five signs to wait, real cost ranges, and a realistic rollout.
A compact local-AI box promises an always-on private assistant in a small footprint. It delivers that — with real ceilings. An honest look at what a Spark-style device is for, the specs that matter, and where it falls short.
Ollama is the easiest way to run a local LLM, but it's a tool, not magic. A beginner-friendly, honest walkthrough: install, pick a first model, chat, add a document layer, tune, and fix the common failures.
Can you run local AI on a Mac? Yes — and it's more capable than most people expect. But it's not a drop-in replacement for a cloud frontier model. A plain-language guide to hardware, model size, install, and where local AI genuinely wins.
Private AI is a specific thing with a specific meaning, and it is not the same as a cloud provider's privacy promise. What a private AI setup really is, what it's built from, and how to honestly decide if it's right for you.
A field-notes account of sizing, model choice, document handling, and network isolation for a small law firm running a local model on Apple Silicon, including the parts that still do not work.
45 measured local-inference figures, transcribed verbatim with sources, whose headline is that two identical Mac Studios differed by 4.7x — and the cause was invisible until you measured it.
Three unrelated GPU allocation failures on a 256 GB M3 Ultra, all with RAM to spare. The ceiling: iogpu.wired_limit_mb, 75% of memory by default.
A sparse-attention MoE model was allocating dense-attention scratch it never touched. A per-allocation trace found it, and prefill jumped to 327–399 tok/s.
Streaming mixture-of-experts weights from NVMe, priced before writing code: at ~6 GB/s against ~800 GB/s of bandwidth, anything over the limit decodes at 1–5 tok/s.
Chained benchmark queues waited for their predecessors with `pgrep -f`, which also matched launch wrappers and the other waiters. None of them started.
An exact split-K tile, a cache-friendly tile order, resized expert tiles and a fast verify kernel, each with the number that killed it.
A frozen SHA-256 proved a quantizer refactor left the default path unchanged, on the machine that froze it. The CI runner's BLAS rounds differently.
The blocker isn't the model, it's the business associate agreement. What ChatGPT and cloud AI can and can't touch inside a covered entity, and why running the model on the practice's own hardware changes the question.
ABA Formal Opinion 512, IRC §7216, and the IRS's first Circular 230 AI guidance have moved AI from a policy choice to a professional-responsibility question. How local models map onto the confidentiality duty, and where they don't help.
Cloud AI is metered per token; on-prem is a fixed machine plus power and labor. Where the crossover sits depends on volume and data, and there are cases where cloud genuinely wins. The honest math, with the estimates labeled.
The model weights are mostly free. You pay for the machine and the labor of making it safe and useful. What a private-AI project looks like from audit to handoff, and what each price tier buys.
A local model is a real alternative to a ChatGPT subscription, but it is not a drop-in swap. The honest ledger: what you give up, what you get back, and who should and shouldn't make the switch.
Headcount is the wrong variable — concurrency is. How to size memory and throughput for an office that shares one machine, what fits in what, and where the real bottlenecks are.
Splitting each MLP's up projection between the GPU and the M3 Ultra's two Neural Engine instances took prefill from 425 to 508 tok/s with no fidelity change.
A scheduler change that removed a 14-second stall, plus the earlier version that was quadratic and undetectable by unit tests.
Six weeks building a Metal LLM runtime for Apple Silicon: the six fixes that took a 27B model from 0.9 to 41 tok/s, and the optimizations that lost.
A guard with a wall-clock cap and a free-memory floor didn't prevent the second kernel panic. A userspace watchdog is starved by exactly what it exists to detect.
A round of parallel coding agents on a real GPU codebase: all five died before committing, and the work was salvaged by hand — with the week's best speedup.
A microbenchmark reusing one weight matrix reported 850 GB/s; cycling distinct matrices reported 600. Every tuning call from the first number pointed the wrong way.
A Rust engine that beats our incumbent on prefill by 4-8% and loses decode by a third — plus the benchmark table everyone quotes, measured on swapping hardware.
After an engine swap failed to beat our incumbent, the decode win came from changing how tokens are drafted rather than how they're computed.
We spent a day evaluating a prefill engine on a machine at one-third clock speed. A power meter reporting zero watts under load cracked it.
On macOS, backgrounding a process from a non-interactive SSH shell silently deprioritizes it. Every number you compare against a system daemon is invalid.
A 47,000-token agent prompt with a carefully designed stable-first layout, getting 49.8% cache reuse. The culprit was upstream of the system prompt entirely.
A synthetic prefill benchmark said 115 tok/s. The production request log said 320. Two research agents had already built rankings on the wrong number.
98% of our request log was liveness probes, consuming 63% of prefill GPU time and creating the sessions that froze real requests for up to 520 seconds.
Porting a Metal prefill engine from a laptop chip to a workstation chip. The single biggest win was deleting a fusion — and two of our claims were wrong.
A rate-limit error that was a malformed request, relabeled by the proxy's retry logic — and the self-sustaining loop that kept re-triggering it.
compressed 70 to 70 msgs, ~85,485 to ~85,485 tokens — logged at success level, four times an hour, for weeks. Four independent causes, each sufficient on its own.
The file was overwritten by the process we restarted, a duplicate daemon crash-looping since May, and session state — after we'd edited the wrong file entirely.
Three independent blockers on a model download, and the one that ended in a kernel panic: raising the wired GPU memory limit to make room.
Two headless Macs went dark overnight: FileVault pre-boot, a launchd domain that never existed, a false lsof reading, and a missing screen-capture grant.
Every technical choice in the stack held up against 2026 practice. The gap was governance: no provenance marking, on a system whose entire output is synthetic speech of a real person.
A conservative memory cap, set once and forgotten, silently disabled the server's auto-sizing. Repeat prompts re-prefilled from scratch: 139 seconds instead of 2.3.
We blamed GPU contention, then reproduced the failure on demand on the machine that had none. The fix was to raise a timeout — the opposite of what it looks like.
A stray syllable at the start of generated speech. One ASR pass both invented it and missed it — prepending 15ms of silence changed the transcript.
Five hours of deliberate abuse produced zero failures. Ordinary traffic killed the same engine in about an hour. The difference was load shape, not load size.
Running a third-party eval framework against our own models. Four traps, one of which produces a complete, plausible results table built on zero model calls.
We fine-tuned a TTS model for weeks. The base model beat it by ear and ran 2x faster. The actual quality ceiling was a 7-second audio clip with a 5.2 kHz bandwidth.
An extraction job that reported 'no conversational material here' on every chunk. The model was spending its entire budget thinking and emitting an empty list as its answer.
Profiling a voice pipeline: the LLM was 5% of wall clock, three tuning knobs did literally nothing, and most of the work was generating a reference clip the code immediately discards.
The fabricated examples carried the highest sample weight in the run — and they were teaching the model to say a catchphrase instead of answering the question.
Batching made speaker embedding nine times slower for numerically identical output. That one bug made a multi-hundred-hour corpus look infeasible to process.
Requests decoding at 1 token/second on hardware that does 42. Not a roofline — a scheduler default and a gated-off config field behind it.
Three false findings in one session, all measurement artifacts — then we found the gateway silently rewriting temperature 0 to 1.0 on every eval we'd ever run.
frequency_penalty=0.1 sounds harmless. On code it compounds until the model can't emit a newline, and it quits with finish_reason=stop so nothing looks wrong.
Nine fixes measured against one hedging regression: four refuted, one dead on arrival, and the winner a single tunable constant — set it to zero.
Measured: 2.05x tokens median, 9.66x on trivial extraction, and a failure mode where the model bills you for 1,200 tokens and returns nothing at all.
A config knob that won't boot, a context sweep that moved throughput by 1.6%, and the vLLM histogram showing the effort was aimed at 13% of the problem.
A twenty-message alert storm, an auto-restart that threw away thirty in-flight requests, and why the obvious watchdog fix would have broken real failure handling.
A benchmark found a serious failure rate in our serving stack. We nearly built a gateway routing layer to fix it. Then we measured real traffic.
93 background jobs vanished while the host stayed up with unbroken uptime and working SSH — all in a launchd domain that only exists while someone is logged in.
Thirty days of gateway logs, one SQL sweep, and the number that retired a 29% throughput upgrade before we built it.
A colleague reported broken JSON and was told to enable JSON mode. That was backwards: 24/24 valid without it, 8/24 with it.
A directory one level too deep meant an entire agent skill library was invisible. Every workflow that assumed it was loaded had been running without it.
A checkpoint swap cost us 11 points on PubMedQA. The entire deficit lived in one answer bucket, and the mechanism was not the one we first wrote down.
If the notes convince you to stop reading and start running models on your own hardware, the next step is a setup audit.
start with the setup audit → see the service →
robotnurse