Inference Engines

llama.cpp, vLLM, and friends side by side. Same model, same machine, different results — measured, not marketed.

1. JSON Mode Was Causing the Malformed JSON (2026-08-07)
2. We Spent a Week Tuning Prefill. The Bottleneck Was Decode. (2026-08-11)
3. One Slow Request Locks Out Everything Else (2026-08-17)
4. One Environment Variable Cost Us 60x on Cache Hits (2026-08-24)
5. Unfusing a Kernel Made It 1.63x Faster (2026-09-01)
6. Our Prefix Cache Died at Token 1,067 (2026-09-02)
7. We Evaluated a Faster Inference Engine and Rejected It (2026-09-03)
8. 2x Decode From Block-Diffusion Drafting (2026-09-03)
9. We Wrote Our Own Inference Engine. It Started at 0.9 tok/s. (2026-09-06)
10. Our Chunked-Prefill Fix Was 72x Slower. Every Test Passed. (2026-09-07)
11. We Put the Neural Engine to Work on Prefill. It Added 19%. (2026-09-11)
12. 17 GB of Attention Buffers Nobody Read (2026-09-12)
13. Five Kernel Ideas That Measured Neutral or Worse (2026-09-12)
14. Speculative Decoding Won Alone and Lost Batched (2026-09-18)
15. We Sent a V4 Model a V3.1 Tool Prompt (2026-09-25)
16. The Tag Was 128 Tokens Back: An Open Bug (2026-09-25)
17. fp16 Residual Streams and a Two-Million Activation (2026-09-25)