JSON Mode Was Causing the Malformed JSON

A colleague reported broken JSON and was told to enable JSON mode. That was backwards: 24/24 valid without it, 8/24 with it.

Summary

Someone reported our model producing malformed JSON — a duplicated opening brace — and was advised to enable JSON mode at the gateway to fix it. Measurement showed the opposite: the plain prompt produced valid JSON 24 times out of 24, and constrained JSON mode produced it 8 times out of 24. The cause was speculative decoding emitting several unconstrained tokens before the grammar engine engaged, and the fix was a two-file upstream backport.

The measurement

Same extraction prompt, temperature 0, through the gateway, 24 attempts each:

The corruption was consistent and specific — a duplicated opening brace:

{
{ "patient": ...

Two things stand out. The feature designed to guarantee valid JSON was the only thing producing invalid JSON. And the last row already names the interaction: it only happens with reasoning mode on.

Why the advice was backwards

"You're getting malformed JSON, so turn on JSON mode" is a completely reasonable suggestion. It's what the feature is for.

It's also the kind of suggestion that's hard to walk back, because when it doesn't work the natural next step is to enable it harder — switch from json_object to a full json_schema, add stricter validation. We can see from the table that would have helped slightly and not solved it.

The thing that broke the loop was measuring the control: how often is the JSON valid when we do nothing at all? Twenty-four times out of twenty-four. Once you have that number, the entire investigation redirects.

Always measure the do-nothing arm. It costs one run and it's the only thing that can tell you a feature is making things worse rather than insufficiently better.

The root cause

Two interacting mechanisms in the serving engine.

The grammar state machine derived its update window from the wrong counter. It used a placeholder count that is incorrect under asynchronous scheduling combined with speculative decoding when draft tokens are rejected. Consequence: the reasoning-block end marker was never observed, so the machine didn't know reasoning had finished.

And it deferred the state-machine advance for JSON, regex, and choice constraints — structural tags were the only same-step exception.

Now add speculative decoding. With a draft depth of 3, a single scheduler step emits up to four tokens. So several tokens escape unconstrained before the grammar engages. The model emits its opening brace during that window; then the grammar engages, doesn't know a brace has been emitted, and adds its own.

Hence {\n{.

This is a nice example of a bug that requires three features to co-occur — reasoning mode, speculative decoding, and constrained output. Each one is fine. Any two are fine. All three together produce a race, which is why it doesn't show up in anyone's test matrix.

The fix

An upstream pull request titled "Advance grammar across reasoning boundary," merged nine days after our container image was built. Right problem, already solved, just not in our build.

We backported it as two bind-mounted files rather than upgrading the image:

• The structured-output module: pass the actual new token IDs into the advance decision, and stop deferring.
• The scheduler: supply those token IDs at the call site.

Result: both json_object and json_schema at 24/24.

Bind-mounting two patched files instead of upgrading is worth defending. Our image is a specialized build carrying hardware-specific quantization and attention backends. An upgrade to pick up one fix would risk the entire tuned stack for reasons unrelated to the fix. Two files, one restart, fully reversible.

When you find your fix upstream, check whether you need the version or just the diff.

Two config levers that failed

Enabling grammar inside the reasoning block made it dramatically worse — 0 out of 12.

Obvious in hindsight: with the grammar applied from token zero, the model can never emit the reasoning-block end marker, because that marker isn't valid JSON. The parser then files the entire output as reasoning and content comes back empty.

That attempt also surfaced a genuinely dangerous configuration behavior: the flag replaces the entire structured-outputs configuration object rather than merging into it. Setting one field silently dropped our reasoning parser setting. Nothing warned us.

Any config flag that takes a JSON object is worth checking for replace-versus-merge semantics before you use it in production. The failure is silent and it removes settings you never mentioned.

Dropping the reasoning config entirely: no change. Both paths resolve to the same adapter.

Two honest notes

This was a regression we caused ourselves. Two days earlier we'd enabled reasoning mode as a server default, which bought 11 points of accuracy on negative judgments (our model stopped saying no) and broke structured output until this backport landed. Both costs were real, and the second one wasn't visible when we made the decision.

That's an argument for a broader post-change test sweep than "did the thing we were fixing get fixed."

We had a wrong hypothesis and the data refuted it. We were simultaneously chasing a blank-answer bug that also involved the reasoning-block end marker, and assumed the two shared a cause. They don't. The failing JSON cases had longer, fully-formed reasoning blocks — the boundary was fine, the grammar just wasn't told about it.

Two bugs touching the same token is a coincidence, not a mechanism. One query against the failing cases cost us essentially nothing; the unified theory would have cost us a day.

JSON mode that emits malformed JSON is a configuration smell, not a model problem — the kind of thing the setup audit catches on the first pass.

Key takeaways

1. Measure the do-nothing arm. 24/24 without the feature is what turned "the feature isn't working" into "the feature is the problem."
2. A constrained-output feature can be the source of invalid output. Speculative decoding emitting tokens before the grammar engages is a general hazard wherever both are enabled.
3. Bugs needing three features to co-occur don't show up in test matrices. Reasoning + speculative decoding + constrained output was fine in every pair.
4. Check whether you need the upstream version or just the diff. Two bind-mounted files beat an image upgrade that would have risked a tuned stack.
5. Config flags taking a JSON object may replace rather than merge. Ours silently dropped an unrelated setting.
6. Two bugs touching the same token are probably two bugs. One query against the failing cases refuted a unified theory we were about to spend a day on.
7. Test more than the thing you changed. Our accuracy fix broke structured output for two days and nothing connected them until a user complained.