We Published a 36% Speedup and Withdrew It the Next Day

A profile said a quarter of every decode pass was un-overlapped host work. An unprofiled control arm, which costs ninety seconds, said it was 1.6%. The 26% was the profiler's own synchronization.

Summary

We profiled decode on a mixture-of-experts model and found that 11.2 ms of a 42.4 ms pass was CPU time spent encoding GPU work, un-overlapped. A quarter of every pass, apparently free to reclaim. We wrote it into the engineering log as a finding with an estimated +36%, and it sat there for a day before someone ran the same prompt with the profiler switched off. The host is 0.4 ms. The profiler cost 40% of the throughput it was measuring, and it books its own synchronization as CPU time — so the instrument didn't just inflate the total, it inflated the exact quantity the conclusion rested on.

Both arms, same prompt, same 47 passes

Host encode is 1.6% of the pass, not 26%. Encoding is already effectively free.

The 40.2 tok/s also matches the number we had recorded for this model months earlier, which is the part that stings: the 24.1 tok/s that made the whole thing look like a mystery worth investigating was the mystery. We had a control arm in our own files and did not compare against it.

The method error, stated plainly

A single measurement, divided by an assumed cause, returns the cause you assumed.

We had one number — 11.2 ms of CPU time — and one candidate explanation — un-overlapped command encoding. The number is real. It is measured. It reproduces. And it is the profiler's synchronization overhead, which is also CPU time, and which nothing in the profiled arm distinguishes from encoding.

Our own documentation warned about this instrument in general terms: a 200 µs synchronization floor, dropped clocks, never trust its absolute numbers. That warning was written about kernel timings. It applies with more force to host timings, because the profiler's overhead is host time, so the contamination lands directly on the quantity being reasoned about rather than spreading evenly across kernels.

The rule we now follow: when a profile drives a conclusion about host time, run the unprofiled control first. Not afterwards as a sanity check — first, so the profile is read against it. It costs ninety seconds.

What survived the retraction

A retraction is more useful when it says what still stands.

Dispatch counts survived. 1,031 dispatches per pass is structural: both arms agree, and the count comes from the graph, not from timing. At the measured 2.34 µs launch floor on this chip, that is about 2.4 ms of a 25.4 ms pass, roughly 9.5%. So the original backlog estimate — cutting dispatches from 1,031 to 400 is worth about 0.64 ms — stands, and the "order of magnitude larger" correction in the withdrawn text does not.

One architectural conclusion survived because it rests on the ratio between two kernels measured in the same arm, which a uniform overhead does not invert.

Every per-kernel millisecond from the profiled run did not survive, including a claim that one layer type was 16% of decode kernel time. Those came off the inflated arm. The relative shares may well hold; they are not measured until someone re-runs them against unprofiled wall time.

And the lever it pointed at is dead. With host encode at 0.4 ms, pre-encoding the command stream — indirect command buffers, a static graph, all of it — has almost nothing to reclaim. That was the expensive thing this finding would have justified building.

Why it took a day

Because the finding was plausible, specific, and flattering. It had a mechanism, a number, a size and a fix. Findings shaped like that get written down and queued; findings shaped like "the number is 0.4 ms and there is nothing here" do not get written at all, which is exactly why the control arm never gets run.

Related: our kernel benchmark was measuring the system cache, benchmark the hardware before you benchmark the software, and our baseline was the three worst samples in the log.