What Thinking Mode Actually Costs

Measured: 2.05x tokens median, 9.66x on trivial extraction, and a failure mode where the model bills you for 1,200 tokens and returns nothing at all.

Summary

Reasoning mode is the largest single lever on a decode-bound serving stack, because decode scales with tokens generated and reasoning generates a lot more of them. We measured the multiplier per task type: 2.05× median, 9.66× on trivial extraction. We also found a failure mode that costs more than the multiplier — with reasoning on and a modest token budget, the model returns content: None, having spent the entire budget thinking. And the documented way to turn reasoning off doesn't work on this deployment.

Part 3 of a series. Our model stopped saying no is the accuracy regression that made us care about this flag; the abstention post is the nine fixes we measured against it.

Why this is the lever

Decode is 86.6% of inference time on our stack. Decode time scales with tokens generated. Reasoning mode generates a substantially larger number of tokens for the same answer.

That makes it the biggest knob available, and it's a boolean, which means it's also the easiest one to get wrong.

The multiplier, by task

Both arms given enough budget to finish naturally. This matters: our first attempt capped both arms at the same max_tokens, so the reasoning arm hit the ceiling and reported a meaningless "1.00×". A capped arm can only tell you about the cap.

Median 2.05× tokens, 1.75× latency.

The interesting column is the last one. Answer length was equal or longer with reasoning off in every single case. The speedup isn't bought by truncating the output. You get the same answer, faster.

And the tax scales inversely with task difficulty. Pulling a numbered list out of a document — a task with no reasoning content whatsoever — pays nearly 10× for a thought process that changes nothing about the answer. The tasks that need reasoning least are the ones charged most, in relative terms, because their baseline is so small.

This is the argument for routing rather than a global default. A stack serving mixed traffic with reasoning globally on is paying a 10× tax on its cheapest requests.

The failure that matters more than the multiplier

At max_tokens=1200 with reasoning on, the response came back:

content: None

1,200 tokens billed. 4,441 characters of unfinished reasoning in the trace. No answer. Reproduced 3/3.

The same prompt and budget with reasoning off returned 4,650 characters of answer.

Reasoning needs roughly 3,200 tokens before it yields an answer on these prompts. Any caller running reasoning-on with max_tokens below that risks getting nothing back — and getting it back as an empty string rather than an error, which is the worst possible shape for a bug. A caller that checks if response.error sails right past it.

Fleet-wide, about 70% of requests finish by length rather than by the model stopping. So the conditions for this were widespread.

Two syntax traps

The documented disable doesn't work. thinking: {"type": "disabled"} is what the docs say. On this deployment it behaves exactly like the default — reasoning stays on. So does reasoning_effort: "none".

What works: chat_template_kwargs: {"thinking": false}. Per-request, no restart needed, which is how every arm in this series was measured against a live stack.

vLLM returns the trace in reasoning, not reasoning_content. Read the wrong key and reasoning looks absent when it is fully present. This produced a confident, wrong reading of "the server dropped 1,200 tokens" before we caught it.

Two field names, one that isn't the documented one and one that isn't the obvious one. Between them they cost us more debugging time than the actual measurement.

Bounded thinking is not available, and you can't fake it

The obvious middle path is a token budget for reasoning: think, but only this much.

thinking_token_budget exists in the build — the file is right there in the source tree. The API rejects it:

not yet supported by the V2 model runner. Run vLLM with VLLM_USE_V2_MODEL_RUNNER=0.

Disabling the V2 runner would disturb a tuned stack — full-decode-only CUDA graphs, MTP speculative decoding, the FlashInfer backend. Not worth it for one knob.

The trap: passing thinking_token_budget inside vllm_xargs returns HTTP 200 and silently does nothing. It looks like it worked. We ran a full arm against it before noticing the numbers were identical to the control.

An unsupported parameter that 400s is a minor inconvenience. One that 200s is a wasted experiment and a false conclusion.

And the prompt-level workaround doesn't work either. Asking for a bounded rationale in the prompt — "one sentence of reasoning, then your verdict" — scored 58.7% against a 57.3% control, at 20× the tokens, while damaging the yes bucket badly (51.8% vs 77.1%). Detailed in the abstention post. In-prompt rationale is not a substitute for the reasoning path.

The routing rule we landed on

The accuracy risk from turning reasoning off is narrow, not general. Per our model stopped saying no, the regression is format-specific: yes/no/maybe verdicts only. Four-way multiple choice was 77.1% with reasoning off against 75.4% for the older checkpoint.

So:

• Default reasoning off. It's faster, the answers are the same length, and on most formats it's as accurate or better.
• Enable for negative and refutation judgments — the one format where it measurably matters.
• For the ambiguous middle, escalate on low verdict-token logprob, not on task type. You don't need to classify the request; you need to notice when the model is unsure and pay for reasoning only then.

That last one is the part worth generalizing. "Which tasks need reasoning?" is a hard routing problem you have to solve in advance. "Is this particular answer low-confidence?" is a cheap runtime signal you already have if you're requesting logprobs.

Key takeaways

1. Give both arms room to finish naturally. A capped comparison reports the cap. Our first measurement said 1.00× and meant nothing.
2. Reasoning-off answers were the same length or longer. The speedup isn't bought with truncation — worth verifying on your own workload before assuming a quality tradeoff exists.
3. The tax is worst where it helps least. Trivial extraction paid 9.66×. Mixed traffic with a global reasoning default is overpaying on its cheapest requests.
4. An empty string is a worse failure than an error. Reasoning-on plus a tight budget returns content: None with no error raised. Check for empty content explicitly.
5. A parameter that returns HTTP 200 and does nothing will cost you an experiment. Verify a config knob changed behavior before you trust an arm that depends on it.
6. Route on confidence, not on task type. Low verdict-token logprob is a runtime signal; "is this a reasoning task" is a classification problem you'd have to solve first.