Our Best Validation Loss Was Our Worst Model

A fine-tune beat production's validation loss by a wide margin (1.086 vs 1.427) and leaked Thai, Khmer and Chinese into English replies. Our harness scored it clean, our checkpoint sweep scored it 0 defects twice over, and four of mlx_lm's defaults meant something other than what our config said.

Summary

On the persona fine-tune behind this series' voice-clone posts, the two runs with the best validation loss of their generation were both the worst models of their generation. The first we have written about: its 1.227 was the best at the time because 22% of its training data had been generated by another model, and generated text is easier to predict. The second is this post. It scored 1.086 against production's 1.427 — a margin that looks like a clear win — and it emitted Thai, Khmer and Chinese fragments into English dialogue, and empty replies. At 48 samples per model under production conditions: production, 0 language leaks and 0 empty replies; the challenger, 3 leaks and 9 empty. Validation loss is a training-health signal. It is not permission to ship.

What makes this worth a post is how many of our instruments, on this run and the ones around it, agreed with the loss.

Every instrument said fine

The quality harness scored it clean. Zero bad citations, zero bad numbers, zero formatting violations. Every check measured something about a reply that was assumed to be a reply in English. None asked. The gate we added is one line — count alphabetic characters above U+02FF in text that should be Latin — and it catches the failure a citation checker structurally cannot see. So is the second gate: measure what is left of a reply after all the post-processing, because post-processing that strips tags and markup can reduce a reply to nothing, and it did.

A checkpoint sweep reported 0 of 32 defects for a model that failed in production. It used a short system prompt and explicitly switched thinking off. Production sent a 5,672-character persona prompt and left thinking on. A sweep that does not reproduce how the model is called measures a different model. Copy the system prompt, sampling settings, thinking mode and post-processing from the serving code, not from memory.

Another sweep scored eighteen checkpoints, from 250 to 12,852 iterations, all at exactly 0 of 16. The sweep loaded the base model once and swapped in each checkpoint's adapter weights with load_weights(..., strict=False), which silently does nothing when the keys don't match. It measured the base model eighteen times. Uniform results across inputs that genuinely differ mean the instrument is not reading them; the cheapest proof is to greedy-decode one prompt under an early and a late checkpoint. Identical output means nothing loaded.

A leak probe reported 0 of 15. It sent "model": "persona" to a local server that treats the model field as a repository id, tried to fetch a repository called persona, and returned 404 on every call. The harness caught each exception and counted it as a sample without a leak. Print attempts and successes separately.

Four instruments on one project, four clean results, and the models were not fine. A blind listen by a human had said the model was producing garbage. The coarse method was the honest one.

What the config said versus what it meant

Diagnosing why the run went wrong meant reading mlx_lm's source, and four of its semantics were not what our config assumed:

• scale is a raw multiplier. The LoRA output is y + scale * z, with no division by rank. In PEFT, the conventional alpha=32, r=16 is an effective multiplier of 2.0; writing scale: 32 in mlx_lm sets it to 32. The default is 20, and the adapter's initialization is tuned to it. Do not port an alpha across.
• num_layers means the last N layers. num_layers: 16 on a 36-layer model adapts layers 20-35 only — 44% of the model, all of it next to token selection.
• lr_schedule defaults to none: a constant learning rate from step 0, while the adapter's second matrix starts at zero and is maximally sensitive to the first updates.
• mask_prompt changes the loss denominator, so loss is not comparable across runs that differ in it, or in sequence length, or in validation set.

And one of ours: carrying an iters value across a change in dataset size silently changes the number of epochs. That alone made three of our runs non-comparable.

What we can and cannot claim here: the recipe was too aggressive, the first three of those settings were all implicated in it, and a raised learning rate with no warmup is our leading explanation; concentrating the change in the layers that choose the next token is a plausible mechanism for language leakage in particular, as opposed to generic damage. Neither is proven. The run changed several axes at once, so its result is not attributable to any one of them, and we are saying so rather than picking the one we like.

Verdict

Ship on measured behavior, not on the loss curve, and not on a single endpoint: save a checkpoint every ~500 iterations and sweep them behaviorally, because damage accumulates and an earlier checkpoint often holds the gain without the harm. Before you trust the sweep, prove it can fail — a checkpoint that must score differently, a probe that must reach the model. And read adapter_config.json, which records what actually ran, including every default you never set.

Related: every failure silently scored 0.5 and the fine-tune was dead weight.