The Tag Was 128 Tokens Back: An Open Bug, and a Test Too Sharp to Trust

Our engine's copy of a model cannot finish its own tool-call tags; another engine running the same prompt can. Here is everything we ruled out, the one test that 'failed' by 10% without finding anything, and the suspect our reference cannot see. Still open.

Summary

This one is not solved, and the post is about how we narrowed it rather than how it ended. With the tool-call prompt fixed (previous post), our own engine's copy of DeepSeek-V4-Flash still breaks every tool call in the same place: it writes the first few pieces of the markup's opening tag, each at a log-probability of about zero — fully confident — and then emits end-of-sequence. With a prompt-cache hit or without, through the chat endpoint or the raw completions endpoint. At temperature 1 the tag comes out garbled instead (toolion, invol). The prose around it is fine. Along the way one of our own tests "failed" by about 10%, and the most useful thing we did that day was to not treat that as the bug.

What kind of failure this is

The model has to copy a tag it has seen before. The exact tag string sits in the tools block of the prompt, more than 128 tokens back. This model attends to its most recent 128 tokens directly, through a sliding window; anything older it can only reach through compressed rows selected by an indexer. So the tag is something the model must retrieve from the compressed long-range path, and it is failing at a copy.

That matches a failure we had already seen and not explained: in a long-context needle test of a quantized version of the same model family, the misses were misspelled passphrases — the model found the right needle and garbled the copy. Both arms of that test did it, including the unquantized one.

Ruled out

Not the model. A different engine on a different cluster, given the identical 318-token prompt, continues it correctly: it closes the tag and writes a valid call with its parameters. And this same checkpoint, served by that other engine until earlier this month, produced valid tool-call arrays of 23 objects for an agent in production.

Not the token's data. The embedding row for the tag's special token, and the hash-routing rows that token selects, match the published checkpoint exactly, or within 8-bit rounding.

Not the reference or the kernels, on anything we can build.

• The vendor's own model and kernel code, audited line by line against our NumPy reference: no discrepancy; the compressor is numerically equal to fp16 rounding.
• Engine against reference at 600 tokens, and at the real attention shapes (head dimension 512, window 128, the real indexer and block sizes): same tokens, logits within 2e-3.
• Restoring a cached prefix across three KV pages: bit-identical.
• The top-k selection path: already tested out to 32,768 compressed positions.
• The attention kernel keeps scores, softmax and accumulators in fp32.

The test that failed by 10% and found nothing

A random-weight fixture has soft, flat attention, which is forgiving of precision problems. So we built a "sharpened" fixture: the same tiny model with its key norms multiplied by 8, to push the attention scores up and make a precision problem show itself. It did diverge from the reference, by about 10%, and for a few minutes that looked like the answer.

It was the test. With scores around 180, a difference of one fp8 quantization step in a single compressed row moves the softmax enough to flip a token. The fixture had become ill-conditioned: small, legitimate rounding differences between two correct implementations were being amplified into disagreements. The check that settled it was layer 0 of the same sharpened fixture, where the inputs to both sides are bit-identical: it matches. The divergence appears only after rows that differ by representation, not by a defect.

The general point: a test you make harder in order to expose a bug can also become too sensitive to separate a bug from rounding. Before reading a divergence as a finding, find the input where both sides must agree exactly and confirm they do. If that holds, the divergence is measuring the test's conditioning.

The suspect our reference cannot see

Every check above compares our engine against a reference that reads our converted weights. That catches kernels that compute the wrong thing from the weights. It cannot catch weights that were converted into the wrong thing. And there is one untested candidate of exactly that kind: our converter stores the model's compressor and indexer weights at 8 bits, where the vendor runs the compressor in fp32 from bf16. Those are precisely the weights that decide which far-back rows the model can retrieve, and a far-back retrieval is precisely what fails.

That is a hypothesis, not a finding. It fits the shape of the failure and it would be invisible to every check we have run.

What is queued

• A plain passphrase copy test, 58 tokens back and 495 tokens back, through the engine cold. The other engine copies both. If ours copies the near one and garbles the far one, the window boundary is confirmed as the fault line.
• Arms on the 318-token prompt that switch off, one at a time: the fast kernels, the staged expert path, the Neural Engine offload, the fake-fp8 path — plus a per-layer trace.
• If the far copy fails: convert the compressor and indexer at full precision and rerun.

Verdict, so far

Open. What we can say: the model and the token data are fine, every component we can test against a reference matches it, and the one "failing" test was measuring its own conditioning. The remaining suspect is in the one place our reference shares our blind spot. When your reference implementation reads your converted artifacts, it verifies your arithmetic, not your conversion.

Related: our two machines ran different models under the same name and we pinned a hash of floating-point output.

Update, 2026-09-27: the suspect is cleared, and the bug is still open

We ran the test this post proposed, without the full-precision reconversion it called for. We built a reference that reads the converted model lazily, one tensor at a time, and swapped in the original checkpoint's tensors by name. With every non-expert tensor taken from the original, compressor and indexer included, the reference still wrote the wrong token at the same place: 858 where the prompt has 19795, and 19795 was not even in its top five. The routed experts, sampled against the original, were bit-for-bit identical. So the conversion is not the cause, and the 8-bit compressor was not the suspect we thought.

Two things we would have said differently. The other engine that "copies it correctly" was serving a different checkpoint of the same model family, so it is still possible that this checkpoint slips on this prompt by itself. And a reference that agrees with your engine proves less than it seems: on the next model in the family, our engine and our reference both applied a normalization the new model had dropped, agreed with each other perfectly, and were both wrong. Diffing against the vendor's code found it; parity never could.