robotnurseLarge language models on hardware you own and can touch.
Field notes from building, breaking, and fixing local LLM setups: quantization, inference tuning, hardware that actually matters, and the honest benchmarks behind each claim. Written slow, tested twice, updated when something stops working. If you would rather have it set up than read about it, book a free 30-minute call.
- lived
- 11 months
- hardware
- gpu ×8 · apple silicon ×2
- entries
- 70 field notes
- hype
- off
The notebook is what we learned. This is what we do with it.
Private, on-site AI on machines you own. Your hardware. Your weights. No invoices from a cloud provider.
On-site in NYC. We install, quantize, and tune models on your own machines, then leave you running.
A short, paid look at your current stack: what is slow, what is wasted, what to fix first. Numbers, not opinions.
Fixed-scope bundles for individuals, small teams, and clinics that need private inference with no cloud dependency.
Latest entries
All 70 posts →The Tag Was 128 Tokens Back: An Open Bug, and a Test Too Sharp to Trust
Our engine's copy of a model cannot finish its own tool-call tags; another engine running the same prompt can. Here is everything we ruled out, the one test that 'failed' by 10% without finding anything, and the suspect our reference cannot see. Still open.
We Sent a V4 Model a V3.1 Tool Prompt
Our renderer wrote the previous generation's tool-call format for a model trained on a different one. The model copied the placeholder words from the instructions, or ended its turn with one token. The fix was to port the vendor's encoder step for step and test it byte for byte.
Our Model Conversion Ran Five Times at Once
A conversion driver without an `if __name__ == "__main__"` guard, a worker pool that starts with spawn, and every worker re-ran the whole conversion. Three more converter bugs from the same day, none of which the test fixture could reach.
We Published a 36% Speedup and Withdrew It the Next Day
A profile said a quarter of every decode pass was un-overlapped host work. An unprofiled control arm, which costs ninety seconds, said it was 1.6%. The 26% was the profiler's own synchronization.
Every Sub-4-Bit Number We Had Was Uncalibrated
Our calibrated quantizer had never once run on a mixture-of-experts model, because the dispatch guard looked at layer 0 and layer 0 is dense. The tell was a calibrated arm that tied its baseline to four decimal places.
Your Headline tok/s Is a Short-Prompt Number
118 tok/s at a 30-token prompt, 35.7 at 86,000. Decode halves for every 4× of context and prefill halves past 20k — and on this engine there is no configuration lever for it, because every cache compressor drops you off the fast path.
Tools
Free calculators we use before buying anything.
Buyers’ notes
- 01Best Mac Studio config for local LLMs — 2026
- 02Mac mini vs Mac Pro for local AI — which is worth it
- 03How much RAM do you need for 7B, 13B, and 30B models?
- 04amdgpu vs Mac for local inference — a fair fight
- 05Mac Studio vs GPU PC for local AI — an honest comparison
- 06Best local AI setup under $3,000 — three builds that work
- 07RAM vs VRAM: how much memory do local LLMs actually need?
- 08Turn an old gaming PC into a local AI box
More in all buyers’ notes.