Our Model Stopped Saying No

A checkpoint swap cost us 11 points on PubMedQA. The entire deficit lived in one answer bucket, and the mechanism was not the one we first wrote down.

Summary

After swapping to a new checkpoint of DeepSeek-V4-Flash, users reported the model "confirming things that weren't true." The complaints were real and reproducible: PubMedQA accuracy fell from 71.7% to 61.0%. But the cause was not the weights being worse, and it was not the affirmative bias we first diagnosed. The model had stopped committing to negative answers and started hedging instead — and that capability hadn't been lost, it had been moved behind the reasoning path.

The complaint

Users don't file good bug reports about model quality. They say it "feels worse," or "it's agreeing with me too much." That's not nothing — it's a signal with the mechanism stripped out — but you can't act on it, and the temptation is to dismiss it as vibes.

We had a hard date to anchor on: a checkpoint swap. So we ran the same eval on both sides of it.

PubMedQA, n=300, same corpus hash, seed 20260727, temperature 0, grammar-constrained to yes|no|maybe:

Three things fall out of that table immediately.

The regression is real. Eleven points, well outside the confidence intervals.

It is the weights, not our serving stack. Row two is the control: we loaded the old checkpoint onto the new image, on a drained cluster, and reproduced the old baseline exactly. That single row saved us from a week of chasing our own infrastructure.

The whole deficit is one bucket. The yes column doesn't move — 79%, 81%, 81%. The no column falls off a cliff: 75% → 39%. And turning reasoning on restores it exactly.

The mechanism we wrote down, and why it was wrong

Our first write-up said this was an affirmative bias: without reasoning, the model answers "yes" to questions whose answer is "no," which is why users experienced it as confirming false things.

That explanation is intuitive, it fits the user reports, and it is wrong.

It matters that it's wrong, because it points at the wrong fix. "Affirmative bias" tells you to put a logit penalty on yes. Here's what the model actually predicts on the 106 gold-no items with thinking off:

It does not confuse no with yes. It declines to commit. The prediction totals confirm it: with thinking off the model emits maybe 108 times against a gold count of 43, while yes stays correctly proportioned (148 predicted vs 151 gold).

The failure is over-hedging, not affirmation. The variable to move is the abstention rate.

Users weren't wrong about their experience — a model that answers "maybe, the evidence is mixed" to a question with a clear negative answer does read as failing to contradict you. But the fix for hedging and the fix for affirming are different fixes, and we had picked the wrong one.

What reasoning actually buys

Once you see it as a hedging problem, the role of reasoning changes.

With thinking on, the same gold-no items come back no 77.4% of the time and maybe only 19.4%. Reasoning isn't finding an analysis the no-think path missed. It's cutting the hedge rate.

We confirmed this a different way: a one-sentence prompt nudge does most of the same thing at 1× decode cost (no 67.9%, maybe 23.6%). Every single one of the nudge's 47 fixes was a maybe → yes/no transition. Nothing else moved.

This lines up with the literature rather than contradicting it. AbstentionBench finds that RLVR reasoning models are roughly 24% worse at genuine abstention than their base models. Reasoning training degrades your ability to say "I don't know." On PubMedQA, where only 14% of items are genuinely maybe, that commitment pressure happens to point the right way — which is exactly why turning reasoning on looks like it "restores accuracy." It restores accuracy on this corpus. It would hurt on a corpus where abstention is the correct answer more often.

It is new in this checkpoint

Same corpus, same harness, both with thinking off:

This table comes from a separate run from the breakdown above, so its counts differ from it by one or two.

yes is unchanged between checkpoints: 147 vs 149. The entire regression is no collapsing into maybe.

And here's the line that settled the diagnosis for us: the preview model's thinking-off hedge rate on gold-no items (18.9%) is essentially identical to the new checkpoint's thinking-on rate (19.4%).

The capability was not lost. It was moved behind the reasoning path.

That is consistent with what the vendor documents. The new checkpoint "keeps the same model architecture and size" and was "only re-post-trained" for "substantially enhanced agentic capabilities." Its published benchmarks are all reasoning-heavy — Terminal Bench went 61.8 → 82.7. Nobody was measuring the no-think path.

This is a known failure class, not a quirk of our deployment. Demystifying Hybrid Thinking (arXiv 2510.12680) finds hybrid LLMs achieve only partial mode separation, and that training-level fixes "reduce but cannot eliminate" a weak no-think mode. There's a documented RLVR reward-hacking problem specific to hybrid-mode training (arXiv 2601.04805). Path-Lock Expert (arXiv 2604.27201) proposes architecture-level separation precisely because training-level fixes are insufficient.

The model card for the new checkpoint documents nothing about non-think mode. No guidance, no accuracy statement, no system prompt, and its only sampling advice is for thinking scenarios. We were running a path the vendor neither optimized nor documents.

The claim is narrower than it sounds

"The new checkpoint lost 11 points" is true of PubMedQA and false as a general statement.

We ran the format matrix. On MedQA — 4-way multiple choice, thinking off — the new checkpoint scored 77.1% against the preview's 75.4%. Flat to better.

Multiple choice has no "no" option to under-select. The hedging has nothing to bite on.

The honest version: serving this checkpoint without reasoning costs you on negative and refutation judgments, and not on multiple-choice knowledge. Which is also why users described it as the model confirming things rather than as broad incompetence. They were describing the shape of the failure accurately, and we nearly generalized it into something false.

The regression also reproduces under grammar-constrained decoding, where the model emits exactly one token. So it lives in the next-token distribution — not in verbosity, formatting, or how the answer gets parsed.

Gotchas from the investigation

Most of the time on this went into instrumentation, not analysis. The findings that will save someone else a day:

The command: block in our compose file is a YAML folded scalar (- >). Newlines become spaces, so a # comment inside it swallows every flag after it on the joined line. Put rationale above command:, never inside it. We lost flags this way.

grep -q healthy matches (unhealthy). Poll for '(healthy)' with the parens.

Our original accuracy baseline was grammar-constrained, root ::= ("yes"|"no"|"maybe"), max_tokens 8. That protocol structurally cannot show rambling, runaway generation, or a wrong first token — which are the failure modes users actually report. Always run a free-form arm alongside the constrained one.

Check commit times, not just dates. Our first preview-vs-new comparison spanned an image migration that landed the same day, five hours after the baseline ran. The dates matched. The builds didn't.

Grammar-constrained evals break under thinking on. Eight grammar-forced tokens can never emit </think>, so every item returns content=null. Verified 3/3. Our nightly matrix now passes thinking-off explicitly for these arms.

reasoning_content is path-dependent. It comes back empty on the direct backend chat endpoint while the raw completions stream clearly shows the reasoning, and populated through the gateway. Production runs through the gateway, so diagnose against the gateway. This discrepancy made an ad-hoc probe and a 300-item eval look like they contradicted each other.

Use McNemar for paired arms. When the same items are scored twice, two-independent-proportions is the wrong test. We used it once and nearly reported a real difference where there wasn't one.

Key takeaways

1. A user complaint about "feel" can be a precise finding with the mechanism removed. "It's confirming things" was an accurate description of hedging. Don't dismiss it, and don't accept your first translation of it either.
2. Write the control run first. Loading the old checkpoint onto the new image took an afternoon and eliminated the entire infrastructure hypothesis.
3. Look at the per-class breakdown before you theorize. The overall number said "worse." The bucket breakdown said "worse at exactly one thing," which is a different and much more tractable problem.
4. Being confidently wrong about a mechanism costs more than having no mechanism. "Affirmative bias" sent us toward logit-biasing yes, when the variable that actually moves is the abstention rate.
5. Reasoning mode buys commitment, not analysis — at least on this workload. That reframing predicts which tasks it will and won't help, and it's why a one-sentence prompt change captures most of the benefit.
6. Scope your regression claim to the format you measured. Eleven points on yes/no/maybe, zero on multiple choice. The general claim would have been false.