We Pinned a Hash of Floating-Point Output

A frozen SHA-256 proved a quantizer refactor left the default path unchanged, on the machine that froze it. The CI runner's BLAS rounds differently.

Summary

We added an asymmetric-calibration option (GPTAQ) to the GPTQ quantizer in our model converter. We needed proof that the default path, the one every existing converted model came from, hadn't changed. The first test froze the SHA-256 of the default path's output on a seeded input. It passed locally and failed on the CI runner, whose BLAS rounds a float64 Cholesky factorization differently. Neither machine was wrong: the test had made one machine's rounding part of the spec. The replacement asserts an identity on whichever machine runs it: the new path with its correction disabled must equal the default path bit for bit, and that holds under any BLAS. The same week, a chat-template test imported jinja2, which was in our virtualenv but not in the requirements file CI installs from. It was the same mistake twice: each test assumed the developer's machine was the environment.

The refactor

GPTQ quantizes a weight matrix one column at a time. It pushes each column's rounding error onto the columns not yet quantized, using the inverse Hessian of the layer's calibration inputs, X: what the float model feeds that layer. GPTAQ (arXiv 2504.02692) starts from the fact that at inference time the layer actually receives X_q, what the quantized model produces, including errors from upstream layers. It adds a residual term that targets ‖Q·X_q − W·X‖, scaled by a factor alpha (reference value 0.25).

The code adds optional X_q and alpha arguments to the existing function. With X_q=None it must be exactly the GPTQ it was before: our converted models, their measured quality and the engine's parity tests all descend from that output.

Test 1: freeze the bits

The test used a seeded input: a 48×192 weight matrix and 300 rows of calibration data with rank-8 structure, in four configurations (4-bit symmetric and 2-bit asymmetric, each with and without act-order). We hashed the outputs of the pre-GPTAQ code, pasted the digests into the test, and asserted that the new code reproduces them. It also checks X_q=None, alpha=7.0, to prove alpha is ignored when there's no X_q:

frozen = {
    (4, False, False): "bbc9021d7e96…",
    (4, False, True):  "f52f9a5f8a42…",
    (2, True,  False): "2c04a3086d0e…",
    (2, True,  True):  "7b91015af2fe…",
}
for (bits, asym, ao), want in frozen.items():
    for kw in ({}, {"X_q": None, "alpha": 7.0}):
        blob, recon = gptq_quantize_tensor(W, X, bits, asym, 64, act_order=ao, **kw)
        h = hashlib.sha256()
        h.update(blob["packed"].tobytes())   # ... scales, zeros, reconstruction
        assert h.hexdigest() == want

It passed locally. In a narrow sense it was a correct test: on that machine, the refactor really hadn't changed a bit.

Why CI disagreed

CI runs on a hosted macOS runner. It builds the engine, creates a fresh virtualenv from the requirements file and runs the integration suite. The hash test failed there.

GPTQ's error propagation goes through a float64 Cholesky factorization of the Hessian, computed by whatever linear-algebra library NumPy is linked against. The runner's BLAS rounds that factorization differently. A difference in the last bits of the factor carries into the updated weights and the reconstruction. SHA-256 has no notion of "close": flip one bit anywhere and the digest is unrelated.

Both outputs are correct GPTQ. The test hadn't detected a regression; it had detected a different computer. What the digest really encoded was "the default path is unchanged on this CPU, with this NumPy build, linked against this BLAS", and that second half was never part of the requirement. Re-freezing the digests on CI doesn't help either. It just moves the machine dependency to the runner image, and a developer's machine fails instead.

Test 2: an identity on the machine running it

The replacement asks a question whose answer can't depend on the arithmetic library. With X_q = X and alpha = 0, the GPTAQ path computes its residual from dX = X_q − X, which is exactly zero, so the correction it adds is exactly zero. Adding an exact zero leaves an IEEE-754 double unchanged bit for bit. (The one exception, −0.0 + 0.0 = +0.0, still compares equal.) Within one process, both paths make the same library calls on the same inputs, so they must agree on any machine:

for bits, asym, ao in [(4, False, False), (4, False, True), (2, True, False), (2, True, True)]:
    base, base_recon = gptq_quantize_tensor(W, X, bits, asym, 64, act_order=ao)
    same, same_recon = gptq_quantize_tensor(W, X, bits, asym, 64, act_order=ao, X_q=None, alpha=7.0)
    zero, zero_recon = gptq_quantize_tensor(W, X, bits, asym, 64, act_order=ao, X_q=X, alpha=0.0)
    for other, other_recon in ((same, same_recon), (zero, zero_recon)):
        assert np.array_equal(base["packed"], other["packed"])   # ... scales, zeros
        assert np.array_equal(base_recon, other_recon)

It's still a bit-exact test: array_equal, with no tolerance. What changed is what it compares. It no longer checks this machine's output against another machine's digest; it checks two computations that are equal by construction on the machine running them.

The identity does give something up. It proves the new term, when disabled, is an exact no-op. It can't tell you the shared GPTQ core matches last week's, because if the core changes, both sides change together. That's a job for property tests, and the same file already had them:

• on low-rank calibration data, GPTQ's output error must be under 0.3× that of round-to-nearest;
• GPTAQ must do no worse than GPTQ on the asymmetric objective;
• on the converted test fixture, error must rank GPTAQ < GPTQ < round-to-nearest;
• the engine must keep token parity with the NumPy reference.

Those hold on any machine that computes correctly.

The same week: a dependency only our machine had

A new integration test for Llama-3-class models checks that the server renders a checkpoint's own chat template exactly the way Jinja2 does, and it imports jinja2 at the top of the module. jinja2 was in the local virtualenv. It wasn't in requirements.txt, the only file CI builds its virtualenv from, so CI failed with ModuleNotFoundError. The fix was one line. The useful part is how the problem got past us:

• An import at module scope fails at collection time, and pytest's default on a collection error is to stop the session before running any test. So one missing package in one file takes down the whole integration step.
• A long-lived local virtualenv accumulates everything anyone ever installed, while CI's is rebuilt from the file on every run. The drift reaches the docs too: our setup notes still list four packages in a comment beside the install command, and the file now has six.
• pytest.importorskip("jinja2") would have turned CI green, and it would have been the wrong fix. A test that skips in CI is a test nobody runs. We had already rejected skip-registration elsewhere in the suite for exactly that reason: it would let a missing virtualenv skip silently in CI.

The check that catches this before a push is cheap: build a throwaway virtualenv from the requirements file alone, and run the suite in it.

The pattern

Both tests encoded something about the developer's machine that the requirement never said: its BLAS rounding in one case, its installed packages in the other. The rule we took from it: a test should compare things that are equal by construction on the machine running it, and the test environment is whatever CI installs.

Frozen golden outputs are still the right tool when the output is the spec, such as a tokenizer's ids, a template's rendered text, or a file format's bytes, because deterministic integer and text output comes out the same everywhere. Floating-point output from a linear-algebra library isn't a spec, it's an observation. Pin its hash and you've pinned the observation.

Determinism that depends on a hash of floating-point output is the kind of thing the setup audit checks before it breaks in production.

Key takeaways

1. A hash of floating-point output pins the machine, not the behavior. Ours failed on a runner whose BLAS rounds a float64 Cholesky factorization differently.
2. Test a refactor with an identity that holds on any machine. The new path with its correction off (X_q = X, alpha = 0) must equal the default path bit for bit, because adding an exact zero changes nothing.
3. Bit-exact within one process is fine; bit-exact across machines isn't.
4. An identity can't catch a change to the shared core. Pair it with property tests: error bounds, orderings between methods, and parity with a reference.
5. Freeze golden outputs only where the output is the spec, such as token ids, rendered text or file bytes.
6. The test environment is what CI installs. Before pushing, run the suite in a fresh virtualenv built from the requirements file.
7. Don't importorskip your way to green. A test that skips in CI is a test nobody runs.