fp16 Residual Streams and a Two-Million Activation

A model whose hyper-connection streams peak around 2.0e6 on an ordinary prompt cannot store them in fp16, whose ceiling is 65,504: every logit becomes NaN. Our fixture never left fp16's range, so no test could see it. The fix is preventive, and it is not the fix for our open copy bug.

Summary

A newer model family replaces the single residual stream with several parallel "hyper-connection" streams, mixed at every layer. The vendor keeps those streams in bf16, which has fp32's exponent range. Our engine stored them in fp16, which tops out at 65,504. On the newest model in the family, the streams reach about 2.0 million on an ordinary prompt — a figure from another project's issue report on the same checkpoint, which reached us before our own conversion had finished. Stored in fp16, the first overflow is layer 21's stream update, and from there every logit is NaN. Every test we had passed, because our synthetic fixture's streams never came near 65,504. We moved the streams to fp32 storage and built fixtures that deliberately exceed fp16's range. This is a preventive fix, made before the real model ran on our engine. It is not the fix for the copy failure we are chasing on the previous model, which is still open.

Why the math was right and the storage was wrong

The stream arithmetic was already in fp32: the mixing, the normalization, the head that collapses the streams back to one. Only storage and the kernels' load and store types were 16-bit. That is the usual pattern — accumulate wide, store narrow — and it is safe exactly as long as the stored values fit the narrow type. Residual-style streams accumulate: every layer adds its output in, and nothing subtracts it out. A model can be perfectly healthy with a stream value in the millions; the next layer's normalization divides it right back down. It is only the storage that cannot hold it.

So the change was type plumbing across the stream kernels — the stream blocks, the mixers, the collapse head, the per-layer embedding adds — and the scratch cost is small: the head collapse grows by 72 KiB per batch row at hidden size 4,096 with four streams.

One family member keeps fp16 streams on purpose: its learned low-rank mixer is fp16 end to end and is served as is. The fix is scoped to the families that need it.

The fixture was the blind spot

A random-weight fixture produces small activations. Ours never came near 65,504, so fp16 storage and fp32 storage gave the same answer, and "engine matches reference" was true for a reason that had nothing to do with the bug.

The new fixtures scale the embedding and every sub-block's output projection, so that each sub-block's output peaks around 3.2e4 — still inside fp16 — while the streams reach about 9.5e4 and the head collapse about 2.7e5, the way the real model's do. The tests first assert that the fixture really exceeds 65,504, then check for finite logits and for tokens and logits matching the reference. On the previous build, both fixtures decode all-NaN logits: tokens 0 0 0 ….

That first assertion is the part worth copying. A test that is meant to exercise a range has to prove it reached the range, or it silently degrades into a test of the easy case.

Two small things the change shook loose

The compiler contracted the same code two ways. Mixing an fp32 stream with an fp16 weight is no longer exact in fp32, and at six streams the Metal compiler fused a multiply-add in one kernel and not in its sibling — which broke our bit-identity gate. At four streams it didn't. The mix accumulation is now an explicit fma, so both kernels round the same way by construction rather than by the compiler's choice.

Three parity tests were sitting on near-ties. With fp32-rounded streams, the reference's own tokens changed on three single-prompt tests, because those prompts sat on the edge of the indexer's top-k selection or on a near-tie between two tokens. Nothing was wrong; those prompts had been fragile all along and happened to agree. They moved to prompts that are not on an edge. A test prompt whose answer flips under a change of rounding was never testing what you thought.

The trace that would have found it

The engine's per-layer trace printed an L2 norm and the first four values of each stream. A single massive activation — one channel in the thousands on one special token — hides inside an L2 norm across thousands of channels and is almost never among the first four values. The trace now prints the peak absolute value, its index, and a count of non-finite values. An fp16 overflow shows up as inf in that count on the layer where it happens.

Verdict

If a model family's reference keeps activations in bf16, check the peak magnitudes before storing them in fp16 — bf16 and fp16 are both 16 bits, and their ranges differ by many orders of magnitude. Make at least one fixture exceed the narrow type's range and assert that it does. And do not let a fix like this get credited with the bug you happen to be chasing at the same time: the previous model's copy failure produces fluent prose around the broken tag, which an overflow to NaN would not, so we are treating the two as separate until a measurement says otherwise.

Related: unfusing a kernel made it 1.63x faster and we pinned a hash of floating-point output.