You Can't Prompt Your Way to Both Commitment and Abstention

Nine fixes measured against one hedging regression: four refuted, one dead on arrival, and the winner a single tunable constant — set it to zero.

Summary

Our served model had stopped committing to negative answers, hedging into "maybe" instead. We measured nine candidate fixes on one frozen set of per-item probabilities, so every scoring rule was evaluated on identical model outputs with no re-query noise between arms. Four prompt fixes were refuted, contextual calibration was dead on arrival for a diagnostic reason, and the winner — subtracting a constant from the log-probability of "maybe" — beat reasoning mode, the prompt fix, and the older checkpoint, at 1× decode cost. Then cost-weighted scoring said to turn it off for clinical work. All of that is one finding.

This is part 2. Our model stopped saying no establishes the regression: the model over-hedges rather than over-affirms, and the deficit is entirely in the no bucket.

The setup that made this cheap

The single best decision here was capturing raw per-item probabilities once, then evaluating every scoring rule offline against that same file.

PubMedQA n=300, grammar-constrained, thinking off, temperature 0, seed 20260727, through the gateway. One probe run. After that, every arm below is a different function applied to identical model outputs. No re-query, no sampling noise between arms, no "did the cluster drift" question.

If you're going to sweep a decision rule, capture the distribution, not the decision.

The winner: penalize the hedge

decision(δ) = argmax( P(yes), P(no), P(maybe) · e^−δ )

One constant, subtracted from the log-probability of maybe.

At δ=3.30 — the value that matches the gold abstention rate — it beats the prompt nudge, beats thinking-on, and beats the previous checkpoint, at 1× decode. And unlike all three, it's a dial: the operating point is chosen per workload instead of imposed globally.

δ=5.55 scores higher but that's fitting the test set. δ has to be calibrated on a labelled dev set.

We expected this to fail, for a specific reason: if the median margin between maybe and no is large, then any δ big enough to matter should be an off-switch for abstention rather than a correction. That prediction was wrong, and the reason is instructive. The margin distribution is wide. A given δ flips the items the model was least confident about first, and leaves the confident ones alone. That's exactly the property you want, and it only shows up if you look at the distribution rather than its median.

Contextual calibration is dead here, and the way it died is the useful part

Contextual calibration (Zhao et al.) estimates a model's content-free prior by feeding it meaningless input, then divides it out. It moved us 60.0% → 60.3%. Nothing.

The reason is worth more than the result:

The method assumes a stable content-free prior. This model doesn't have one. It answers the content-free token itself — "N/A" reads as a negation, empty string reads as assent, "[MASK]" reads as uncertainty. The three average to roughly uniform (0.352 / 0.310 / 0.338), so dividing by the average is a no-op.

Don't retry this method on any model without first checking content-free stability. It's a three-request check and it tells you in advance whether the technique can possibly work.

The model is confidently wrong, not marginally wrong

This is the finding that killed an entire class of fix.

Of the 61 gold-no items answered maybe: no ranks second on 46 of them (75%). Encouraging — sounds like near-misses waiting to be reclaimed.

But the median P(maybe) − P(no) margin is 0.881, and only 1 of 61 sits under 0.20. Calibration recovered 0 of 61.

There are no near misses. Post-hoc renormalization cannot fix a distribution that is confidently wrong. Only changing the input (the prompt) or the computation (thinking) moves it — which is precisely why the nudge and reasoning work while calibration doesn't.

Five prompt fixes, four refuted

PubMedQA first n=150, thinking off, temperature 0, free-form, same items every arm. Compare arms only within this run — the first-150 subset is harder than the full set (plain scores 57.3% here vs 61.3% at n=300).

The balanced arm settles the important question. It explicitly licenses all three outcomes — the fair-minded phrasing you'd write if you wanted commitment and honest abstention. It produced identical abstention recall to the one-sided nudge (10.5%) while recovering no far worse (52.1% vs 75.0%).

You cannot prompt your way to both commitment and abstention. The collapse is intrinsic to forcing commitment, not an artifact of one-sided wording.

The other three negatives:

A bounded in-prompt rationale does not substitute for reasoning. 58.7% at 20× the tokens, and it damages the yes bucket badly (51.8% vs 77.1%). This matters beyond the arm itself: our build rejects thinking_token_budget, so "ask for one sentence of reasoning in the prompt" was the obvious workaround. It isn't one.

Few-shot negatives do nothing. 58.0% against a 57.3% control. Two exemplars including an explicit no moved the needle by noise.

The Mid-Think token-level trigger (arXiv 2601.07036) does not transfer. A leading "Okay" via assistant prefill with continue_final_message scored 17.8% — below chance — and emitted template artifacts like "# maybe". It made the model hedge more. Model-specific finding, not a general refutation of the paper.

Only two levers work: the prompt nudge (74–75%) and the δ threshold (75.7%).

They compose, and that's when it gets uncomfortable

We captured nudged-prompt logprobs for all 300 items and swept δ offline again.

Nudge + δ=3.45 → 80.7%. Against nudge alone at 74.7% and plain+threshold at 78.0%. That's +20.7 over the broken baseline, +12.4 over thinking-on, +9 over the older checkpoint, all at 1× decode.

The optimal δ drops from 5.70 to 3.45 because the nudge is already doing part of the work — the two levers are pulling the same rope.

But at that operating point the model predicts maybe 5–7 times out of 300, against a gold count of 43. By δ=4.5 it's zero.

It has stopped abstaining. We bought six points of accuracy by removing the model's ability to say "I don't know."

Cost-weighted scoring reverses the recommendation

Suppressing maybe does not distribute errors evenly. On items whose true answer is maybe:

Every abstention you remove becomes an affirmation about five times as often as a negation.

For a benchmark leaderboard that's a wash — accuracy is accuracy. For clinical use it is the worst available direction to fail in, and no harness in our fleet scored maybe → yes differently from maybe → no. The instrument was blind to the thing that mattered most.

So we built one that isn't. Pricing errors separately (a default matrix, explicitly not clinically validated — a clinician has to set it: no→yes 5, maybe→yes 4, yes→no 3, maybe→no 2, hedges 1):

Accuracy-optimal δ=3.30 → 80.7% accurate, cost 0.747. Cost-optimal δ=0.00 → 74.7% accurate, cost 0.643.

Accuracy tuning buys six points by manufacturing the most expensive error class. maybe → yes is the dominant cost cell even at δ=0.

Ship: nudge alone (δ=0) for clinical verdicts, nudge + δ≈3.45 for non-clinical throughput.

Validation, and the honest gap

Two-fold cross-validation within PubMedQA — fit δ on one disjoint half, score the other:

+13 to +21 points on unseen data. The curve is a broad plateau (74.0–77.3% across δ=3.0–4.2), so the exact value doesn't need to be right.

One thing we learned here: best-accuracy δ is stable across folds (6.00 / 5.70 / 5.70), but gold-rate-matched δ is not (4.05 / 2.25 / 3.30). Fit δ by accuracy on a dev set, not by matching an abstention rate. The rate-matched value is intuitive and unstable.

Cross-corpus generalization is untested and untestable with what we have locally. PubMedQA is the only yes/no/maybe corpus on the box. MedQA can't test this at all — 4-way multiple choice has no maybe to threshold. BioASQ yes/no is the obvious next corpus and we haven't run it. Everything above is one-corpus evidence with internal cross-validation, and should be read that way.

The final negative: the signal isn't there

We tried three independent ways to detect "this question is genuinely inconclusive" from the no-think path. All three are dead, and together they close the question.

1. The verbalized maybe token — broken by the re-post-training. It emits 109 against a gold count of 43, and they're the wrong ones.

2. Distributional margin. The yes/no margin is noise with respect to gold-maybe: recall ≤23% at any accuracy-preserving threshold, and the threshold is unstable across folds (0.55 / 0.80). This arm also turned up the tidiest result in the whole project: plain prompt + simply ignoring the maybe token scores 78.0%. The nudge, the δ threshold, and binary argmax are all the same fix — suppress broken abstention — and they converge at 78–81%.

3. Ask the model directly. A second pass: "is this conclusive or inconclusive?", grammar-constrained, with logprobs. AUC 0.577. Median P(inconclusive) is 0.998 on gold-maybe items versus 0.993 on everything else. The model calls everything inconclusive. The hedging generalizes to the meta-question.

(That script had a real bug — its combine table paired pass-1 and pass-2 rows by completion order rather than item order. The AUC and medians are computed per-row and are sound, and they alone kill the mechanism. Worth stating rather than quietly dropping the arm.)

Conclusion: genuine abstention is unrecoverable from the no-think path on this model. It requires reasoning mode. The ceiling for 1×-decode verdict accuracy is ~80.7%, and it is bought by suppressing abstention.

The clinical split — δ=0, or reasoning-on wherever "maybe" carries real meaning — is the optimum, not a compromise.

The other half of the story is boring, which is the point

MedQA n=300, grammar-constrained, thinking off, same session: 77.3% (95% CI 72.3–81.7%, zero errors). That reproduces the documented 77.1% and beats the older checkpoint's 75.4%.

Thinking-off ships today for extraction, summarization, multiple choice, and agentic work. The regression genuinely does not touch those formats. Only yes/no/maybe verdicts need the threshold or reasoning.

A regression that costs 11 points on one answer format and nothing on the others is a routing problem, not a rollback.

Key takeaways

1. Capture the distribution once, sweep decision rules offline. Nine arms on identical outputs, zero re-query noise. This is the cheapest experimental design available for anything logprob-shaped.
2. Check a method's assumption before running it. Contextual calibration needs a stable content-free prior. Three requests would have told us it wasn't there.
3. Confidently wrong beats marginally wrong at killing fixes. A median margin of 0.881 with 1 of 61 items near the boundary means no post-hoc rescoring can help. Look for the near-misses before you build the reclaimer.
4. A "balanced" prompt does not buy you both behaviors. Licensing all three outcomes gave identical abstention to the one-sided nudge and worse accuracy. The tradeoff is intrinsic.
5. Fit hyperparameters by the metric you care about, not the one that sounds principled. Accuracy-fitted δ was stable across folds; rate-matched δ wasn't.
6. Score errors by cost, or your harness will recommend the wrong thing. Six points of accuracy, bought entirely by manufacturing the most dangerous error class, looked like a clear win until we priced it.