We Sent a V4 Model a V3.1 Tool Prompt

Our renderer wrote the previous generation's tool-call format for a model trained on a different one. The model copied the placeholder words from the instructions, or ended its turn with one token. The fix was to port the vendor's encoder step for step and test it byte for byte.

Summary

Our engine served DeepSeek-V4-Flash with the tool-call prompt of DeepSeek-V3.1: V3.1's special tool tokens, its instructions, a generation prompt ending on a bare assistant header, and tool results wrapped in V3.1's output tokens with no assistant header after them. V4 and V4.1 were trained on a different format, an XML-like markup the vendor calls DSML. Their official encoder never writes a single one of V3.1's tool tokens; every assistant header is followed by an opening or closing think tag; and tool results go back inside the user turn. Given a prompt its training never produced, the model did two things. It wrote tool_call_name read_file {...} into plain content — copying the placeholder words from V3.1's instructions, because it cannot emit tokens it was never trained to emit. And twice it returned empty content with finish_reason: stop. The fix was not to patch our template; it was to port the vendor's own encoder function step for step and gate it byte for byte against the vendor's test vectors.

Two symptoms, one cause

The placeholder copy. V3.1's tool instructions describe the call format with literal placeholder words: a begin token, then tool_call_name, a separator token, then tool_call_arguments. A V3.1 model reads those as a template and fills them in. A V4 model has never seen the separator token used, so it has no trained path to produce it — but it can read English. It produced the placeholder words themselves, followed by the function name and a JSON blob, as ordinary text. A client parsing for tool calls sees a chatty reply with no call in it.

The empty reply. Our prompt left the model at a position its training never had: an assistant header with no think tag after it, or — after a tool result — no assistant header at all. From there, ending the turn immediately and emitting a <think>…</think> block with no answer after it are both likely continuations. A gateway that drops the reasoning field shows the second case as an empty reply too, so the two look identical from the client.

They are not identical in the request log: completion_tokens tells them apart. One completion token is the model emitting end-of-sequence right at the header. More than one, with empty content, is a think block that went nowhere. That one field saved us from chasing a "the model sometimes goes silent" theory.

The fix: the vendor's encoder, not our reading of it

The vendor ships its encoder as Python, with test vectors. We rewrote our renderer as a step-for-step port of that function rather than as a template we believed to be equivalent:

• tool results merged into the user turn and sorted into call order;
• thinking versus chat mode chosen the way the encoder chooses it, with earlier reasoning dropped unless tools are offered;
• the tools block placed on the last system message before the conversation starts, adding an empty one if there is none;
• the newer model (V4.1) told apart from V4 by its tokenizer, not by configuration: V4.1's encoder writes a system token that V4's vocabulary only holds a placeholder for.

The gate is byte equality. Every vendored test vector is rendered and compared byte for byte. Five pass; four are skipped, each with a stated reason (a developer-role turn our request reader maps differently, task tokens we do not support, images). The vendored cases all end on an assistant turn, so they never exercise a generation prompt — the exact place our bug lived. We generated eight more cases with the official encoder itself: generation prompts in both modes, parallel calls with every value type, unsorted tool results, and the reasoning-effort variants. Plus a replay check: turn two's prompt must begin with turn one's prompt plus its completion, byte for byte, or the prefix cache and the model's view of its own history both break.

Two honest limits went in with it. Tool-call arguments are not token-constrained in this format — they are validated against the schema after parsing — so a tool marked strict: true is refused with a 400 rather than silently accepted and not enforced. And a call whose text ends before its closing tag is reported as truncated, not parsed as a call.

Verdict

When a model family changes its tool format between versions, "the template for the previous version plus a few edits" is a different prompt, and the model will tell you so only indirectly — by writing your instructions' placeholder words into its reply, or by stopping. Port the vendor's encoder, test it byte for byte, and write your own vectors for the generation prompt, because the vendor's vectors end where the model's turn begins.

This did not fix everything. With the corrected prompts in, the served model still cannot finish its own tool-call tags. That is a separate failure, still open, and it has its own post.

Related: the 429 that was really a 400 and what thinking mode actually costs.