Quantization

Bits are not free. A practical tour of q4, q5, q8, and fp16 — what the numbers actually mean to your latency, your memory, and your output.

1. A 192 GB Model on a 256 GB Box (2026-08-27)
2. We Pinned a Hash of Floating-Point Output (2026-09-12)
3. Every Sub-4-Bit Number We Had Was Uncalibrated (2026-09-19)
4. Our Model Conversion Ran Five Times at Once (2026-09-25)