Every Surprising Eval Result Was Our Infrastructure

Three false findings in one session, all measurement artifacts — then we found the gateway silently rewriting temperature 0 to 1.0 on every eval we'd ever run.

Summary

We built a harness to decide whether local models were good enough for real work. The models were fine. The harness kept lying to us — three separate false results in one session, every one an infrastructure artifact rather than a capability finding. Then we discovered our own gateway had been silently rewriting temperature=0 to 1.0 on every eval request, invalidating an unknown amount of prior work. Here's the hygiene list we ended up with.

Three false results in one session

Each of these looked like a real capability finding. Each was the measurement apparatus.

False result 1: "the model scores 49.5% on MedQA." Alarming — that's near chance on a 4-way multiple choice. Cause: the proxy was returning sporadic HTTP 500s under concurrent load, and the harness scored an error as a wrong answer. Real score with errors excluded: 91.5%.

False result 2: "this model can't retrieve from long context." It could. The model is a reasoning model, and our max_tokens was low enough that it spent the entire budget thinking and returned empty content. Empty was scored as failure.

False result 3: "adversarial robustness is 0.60." We'd run multiple eval jobs concurrently against the same stack. Rate-limit 429s were scored as failures. Real score: 0.98.

The pattern is identical in all three, and it's the single most important thing in this post:

A benchmark harness's default behavior is to score infrastructure failures as model failures. An exception, a timeout, a 429, a truncated response, an empty string — all silently turn into a zero in the denominator and produce a confident number.

The fixes, now permanent in our harness:

• Retry transient errors with backoff, and exclude genuine errors from the denominator rather than scoring them wrong.
• Give reasoning models adequate max_tokens, or empty responses masquerade as failures.
• Never run multiple jobs against the stack at once. Serialize.

That last one is unglamorous, and it eliminated an entire class of false result.

The one that invalidated old work

Worse than a bad run is a bad run you don't know about.

Our gateway had anti-loop protections installed, and one of them enforced a sampling floor: any request with temperature ≤ 0.3 was silently rewritten to temperature = 1.0.

For production traffic that's a defensible policy. For evaluation it is catastrophic, because every deterministic eval we'd run through that gateway had actually run at temperature 1.0 — and nothing in the response says so.

Two things came out of this:

A dedicated exempt key for eval traffic. All benchmark runs now use a key alias that bypasses the sampling floor, with the reasoning documented next to it.

A contamination check you can actually run. Grep the gateway logs for the floor's rescue lines; a line naming your key means that run is contaminated. We used it to identify and mark specific result files as invalid rather than quietly deleting them.

Related discovery: our load-balanced alias is not deterministic even at temperature 0. The two clusters produce different completions for identical inputs — cross-machine floating point. For any reproducible run you have to pin a specific cluster and record which one in the result metadata.

"Temperature 0 means deterministic" is false the moment more than one machine sits behind the endpoint.

What the harness looks like

Two halves, deliberately kept separate:

Authored suites — tasks we wrote and score automatically: coding (execute against unit tests), extraction, reasoning, agentic tool-calling, clinical, adversarial-clinical, and calibration (does the model abstain on unanswerable questions, or confabulate).

External benchmarks — items with official answer keys that we didn't write: GSM8K, HumanEval, MedQA, MMLU, MMLU-Pro, MedMCQA, MATH-500, GPQA-Diamond. Pulled from the public datasets API into local files.

The split matters because they answer different questions. Authored suites tell you about your workload. External benchmarks tell you whether your authored suites are lying to you.

Statistics: Wilson 95% confidence intervals on every proportion, and an exact McNemar test for paired comparisons — same items scored by two models. Two-independent-proportions is the wrong test when the items are shared, and it will report differences that aren't there.

The standing conclusion

Across 8 external benchmarks and 1,500+ items, comparing our two main local models:

Tied on the easy tests — GSM8K, HumanEval, MedQA, MMLU, MMLU-Pro, all McNemar p > 0.05.

Significantly different on the hardest three — MedMCQA 81.5 vs 73.5 (p = .02), MATH-500 95 vs 87 (p < .001), GPQA-Diamond 43.4 vs 33.3 (p = .005).

The pattern: equivalent on routine work, one model pulls ahead as reasoning difficulty rises.

That's a more useful finding than a single leaderboard number, and it's only visible because we ran benchmarks across a difficulty range and tested each one for significance. A single mid-difficulty benchmark would have shown a tie and we'd have picked on latency alone.

Both models: zero confabulation on the calibration suite, reliable retrieval to ~24k tokens, ~100% extraction accuracy on two independent real-text corpora.

Also measured, on the same machine at the same quantization: a mixture-of-experts 30B model runs at 116 tok/s against a dense 27B at 37 tok/s — about 3×. Apple Silicon favors MoE for speed and dense for quality per gigabyte.

Free-response grading needs a judge, and the judge needs care

One benchmark we wanted was only available ungated in free-response form. String and numeric matching cannot grade it: 261 TeV equals 2.6e5 GeV, algebraic rearrangements are equivalent, a matrix and a tuple can be the same answer. String-only scoring gave a bogus 16%.

The fix is an LLM equivalence judge, with two design choices that matter:

Fall back to the judge only when exact match fails. That structurally prevents false positives — the judge can only ever rescue an answer, never reject a correct one.

Use the same judge for both models. Absolute numbers become approximate; the comparison stays fair. That's usually the trade you want.

And the judge hit the same reasoning-model truncation bug as everything else: empty verdicts at low max_tokens. Give it 2048 and parse the last verdict token in the output, not the first.

Corpus-building traps

Key questions by unique text anchor, not line index. Our file-reading tool returns 1-based line numbers; using them as 0-based indices is a silent off-by-one that shifts every answer by one line. Caught by a build-time check that every gold answer literally appears in its source document — which is the kind of assertion worth writing even when you're sure.

Not every dataset is servable through the hosted API. One we wanted fails Arrow conversion. Streaming the raw CSV with a standard CSV reader works fine — it handles the multi-line quoted fields that break naive line-splitting — and you can stop early once you have enough items.

When an eval reports something impossible, the infrastructure is usually lying to you — reading the machine before the benchmark is what the setup audit starts with.

Key takeaways

1. Harnesses score infrastructure failures as model failures by default. Retry transient errors and exclude genuine ones from the denominator; don't score them as wrong.
2. Verify the measurement apparatus, not just the model. Every surprising result in our session was infrastructure. That's a reliable prior.
3. Serialize eval jobs. Concurrency produced 429s that scored as capability failures.
4. A gateway can silently rewrite your sampling parameters. Use an exempt key for eval traffic and keep a way to detect contaminated runs after the fact.
5. Temperature 0 is not deterministic behind a load balancer. Pin the backend and record it in your results metadata.
6. Use McNemar for paired comparisons, and Wilson intervals on proportions. Independent-proportions tests over-report differences on shared items.
7. Benchmark across a difficulty range. Our two models were tied on five tests and separated on three; either half alone gives the wrong answer.
8. An LLM judge should only ever rescue, never reject. Fall back to it after exact match fails, and use the same judge for every arm.