One Environment Variable Cost Us 60x on Cache Hits

A conservative memory cap, set once and forgotten, silently disabled the server's auto-sizing. Repeat prompts re-prefilled from scratch: 139 seconds instead of 2.3.

Summary

Our MLX inference servers were re-prefilling identical prompts from scratch every time. A 39,623-token prompt took 139 seconds on the first request and 139 seconds on the second. The cause was a session-bank size cap set in the service definition — and the cap didn't just make the cache small, it disabled the server's automatic sizing entirely, because auto-sizing only engages when the variable is absent. Removing it took repeat prefill from 139 s to 2.3 s.

The symptom nobody reported

Large prompts were slow. Everyone knew large prompts were slow — prefill on this hardware runs a few hundred tokens/second, so a 40k-token prompt taking a couple of minutes was accepted as physics.

What nobody noticed was that the second identical request took just as long.

That's the tell. Slow cold prefill is physics. Slow warm prefill is a bug.

The cause

Two machines, two caps, both set at install time and never revisited:

Studio B's 2 GB was below the server's own 8 GB default. Someone had set a conservative value to be safe and made it substantially worse than doing nothing.

But the real problem is subtler than a small number. In this server, auto-sizing engages only when the variable is absent. Setting it to any value disables the automatic policy — "half of post-model RAM surplus" — and pins the cap at whatever you wrote.

So the variable does two things: it sets a value, and it turns off the thing that would have set a better value. The second effect isn't obvious from the variable's name, and it's the one that hurt.

This is a config-design pattern worth recognizing. Any setting that means "override the automatic behavior" as a side effect of being present will eventually be set once, for a good reason, by someone who doesn't realize what else they turned off. Then the automatic behavior improves in a later release and that machine never gets it.

The measurement

Same 39,623-token prompt, same machine, back to back:

About 60× on a warm prefix. Cold performance is unchanged, as you'd expect — nothing about this makes the first prefill faster.

The second machine, after being raised from 2 GB to 16 GB:

Zero percent to one hundred percent. There's no partial-credit regime here: either the bank is big enough to hold the prefix or the cache does nothing at all.

Why we set it to 16 GB and not unset

The two machines aren't in the same situation, and this is where blindly applying the fix would have been wrong.

Studio A: 108 GB free, zero swap, four listeners. Safe to unset and let auto-sizing take ~48 GB.

Studio B: fourteen model listeners and real memory pressure. We set an explicit 16 GB rather than unsetting, deliberately — above the server's 8 GB default, well below what auto-sizing would have grabbed.

The 60× number is exciting enough to make you want to apply it everywhere. On a memory-pressured box, unsetting the cap would have traded a prefill win for swap thrashing.

Operational notes

Each machine caches independently. A given prefix is cold once per box. With a round-robin load balancer, a repeat request can still land on the machine that hasn't seen the prompt and pay the full 118 seconds. Session-affinity routing for cache hits is what makes the win reliable rather than probabilistic — a 50% hit rate across two machines is a much less interesting result than 100%.

Reloading the service right after stopping it fails. The job is still tearing down, and you get an I/O error. Wait a few seconds and reload again. Permissions were never the problem, despite what the error suggests.

Confirm from the startup banner, not from the config file. Ours prints the effective budget and whether it's explicit or automatic:

session-bank budget: 48.0G total (auto: half of post-model RAM surplus)

versus the previous 6.0G total (explicit). That one line is the difference between the fix being applied and being merely written down. Read what the service says it's doing, not what you told it to do.

The one that didn't benefit, and why

Our coding agent gained far less than 60×. A run showed a 24,912-token prompt with only 2,048 tokens cached.

Something early in its prompt varies per run — a session identifier, a working directory, a timestamp — and breaks the shared prefix. Prefix caching is exact-match from the start of the prompt: one changed token near the beginning invalidates everything after it.

That's a separate and worthwhile optimization, and a general lesson about prompt construction. Put volatile content at the end. A timestamp in your system prompt costs you the entire cache; the same timestamp in the final user message costs nothing.

We hit a larger version of this problem elsewhere — several volatile spans scattered through a long system prompt, each independently destroying the cache below it. Worth auditing any long-lived prompt for content that changes between otherwise identical requests.

One env var, sixty times slower: config review is the cheapest intervention there is, and it's the core of the setup audit.

Key takeaways

1. Slow cold prefill is physics. Slow warm prefill is a bug. Measure the second identical request; that's where cache problems surface.
2. A setting can disable an automatic policy as a side effect of being present. Ours turned off auto-sizing merely by existing, which is not implied by its name.
3. A conservative value can be worse than no value. One of ours was set below the software's own default in the name of safety.
4. Prefix caching is all-or-nothing at these sizes. We went 0% → 100%, with no useful middle.
5. Don't apply the same fix to differently-loaded machines. Unsetting the cap was right on the idle box and would have caused swap thrashing on the busy one.
6. Verify from the service's startup banner. It reports what's actually in effect, which is the only thing that matters.
7. Put volatile content at the end of a prompt. One varying token near the start invalidates the entire cache below it.