Our Compression Logged Success and Compressed Nothing

compressed 70 to 70 msgs, ~85,485 to ~85,485 tokens — logged at success level, four times an hour, for weeks. Four independent causes, each sufficient on its own.

Summary

Our agent framework's context compression had been running on schedule, returning without error, and logging its result at success level. The result looked like this:

10:36  compressed  70 →  70 msgs,  ~85,485 →  ~85,485 tokens
12:05  compressed  52 →  52 msgs,  ~86,639 →  ~86,639 tokens
12:21  compressed 106 → 106 msgs, ~108,983 → ~108,983 tokens
13:03  compressed 142 → 142 msgs, ~110,996 → ~110,996 tokens

Identical in and out, every time. Four separate causes were each independently sufficient to produce that, and we found them one at a time over about a week. The worst of them let a single agent session reach 960,603 tokens and hold roughly 87% of a shared inference server's entire KV cache.

The log line is the whole problem

Nothing in that output is an error. The job ran, it finished, it reported numbers. The only signal that compression did nothing is that the two numbers on each line are equal, and no human reads a success line closely enough to subtract.

If you write a compaction, a cache eviction, a garbage collector, or anything else whose job is to make a number smaller: log the delta, and log at warning level when the delta is zero. A no-op is not a success. We eventually added exactly that, and it is the single highest-value line of code in this entire post.

Cause 1: the timeout was a no-progress fence, not a total timeout

The compression call had a 30-second default timeout. We read that as "give up if this takes more than 30 seconds," which for a summarization call seemed generous.

It is a no-progress fence: 30 seconds without a new streamed chunk. There is a separate total ceiling, defaulted to 600 seconds.

That distinction is fatal for a local model, because compression sends the entire session as one prompt. Nothing can stream until prefill finishes. Measured against our own trigger threshold, streaming:

57.1 is greater than 30, so the fence tripped before the first chunk arrived — on every single compression, deterministically. Once it started, it never once succeeded.

And here's why it went unnoticed for so long: the framework had originally used a small hosted model for compression, which returned its first token inside 30 seconds. The fence never tripped. Switching the compressor to our local fleet is what exposed a default that had been wrong all along, and the symptom presented as "the local gateway is slow" rather than "the fence measures the wrong thing." We spent real time on the first theory.

A no-progress timeout and a total timeout fail on completely different workloads. A long prefill with no streaming is invisible to one and fine for the other. Check which kind you have before tuning the number.

Cause 2: the setting was read from a different block than the one it appears in

The config has an auxiliary.compression: block, which is where the compression model and its fallback chain are configured. It is the obvious place to put a compression timeout, and it is not where the timeout is read from. The code reads it from the top-level compression: block.

Setting it in the natural location changes nothing and produces no warning. We had it right in the wrong block for a day.

There isn't a general lesson here beyond the specific one: when a setting appears to have no effect, grep the source for the key name before assuming your value is wrong. Two plausible config paths for the same concept is a very common shape, and neither one will tell you it's being ignored.

Cause 3: the protected window contained the bulk of the tokens

Compression protects the most recent N messages so the agent doesn't lose immediate context. Ours was set to 20 on one machine and 30 on another.

compressed  ~523,747 → ~523,556 tokens

Almost everything was inside the protected window. That is not bad luck — it's structural. Recency correlates with size, because the most recent turns are the ones carrying full tool outputs, and tool outputs are the largest messages in an agent session by an order of magnitude. Protecting "the last 20 messages" protects most of your token budget.

We dropped it to 6, and separately capped individual tool results at 20,000 bytes, down from 50,000. That second number is the arithmetic of the whole problem: 60 turns × 50 KB of tool output is how a session gets to half a million tokens without anyone doing anything unusual.

Cause 4: the context window was auto-detected from the server, and the server said a million

On one instance the context length was simply unset, so the framework auto-detected it by asking the inference server what it served. The server is started with a one-million-token maximum length, so it answered honestly. With a compression threshold of 0.5, that put the trigger at 500,000 tokens:

Preflight compression: ~960,603 tokens >= 500,000 threshold (model deepseek-v4-flash, ctx 1,000,000)

A session reached 960,603 tokens before anything tried to compress it.

This is a fleet problem, not a client problem. That server's entire KV cache is about 1.1M tokens. One session at 960k holds roughly 87% of the cache for every other tenant on the box, and co-tenant keys showed 553.8-second maximum latency in the same window. An unset client-side config field on one laptop degraded a shared inference cluster.

A model's advertised maximum context is a capability, not a budget. Auto-detecting it means your agent will happily use all of it, and the number a server advertises is chosen for the largest thing it can ever do, not for what any client should routinely do.

The fix that appeared to fail

We pinned the context length to 100,000, restarted, and watched live traffic get worse — average prompt tokens went from 126,826 to 311,831.

Config is pinned at session creation. Every running session keeps the settings it was born with, and all three active ones predated the restart. There is no command to force a running session to re-read config; you compress it by hand, start a new one, or wait for it to end.

And the log line we were checking for confirmation was stale. The newest line showing the old value was timestamped an hour before the gateway process had even started. We nearly concluded the fix hadn't landed.

Check the timestamp of the log line you are using as evidence against the start time of the process that would have written the new one. An absent new value and a stale old value look identical in a tail.

What it was worth

After all four:

One honest correction on that table, because it's the kind of thing that gets misattributed. Contexts remained above 100k afterwards — median 122,964, max 288,040 — and were fast anyway. So the throughput win is not from the context-length reduction. It's that the shared serving lane stopped being wedged, partly by us and partly by unrelated work. What our changes provably fixed is that compression fires at all and completes.

Before you re-tune a client's context budget chasing latency, measure whether the shared lane is wedged. We would have taken credit for the wrong fix.

Compression that reports success and compresses nothing is the sort of thing the on-site packages are scoped to catch.

Key takeaways

1. Log the delta and warn on zero. A compaction that reports equal input and output is a failure logged at success level, and nobody subtracts two numbers on a success line.
2. A no-progress fence is not a total timeout. Ours was 30 s against a 57.1 s time-to-first-token, so it tripped on every attempt, deterministically, forever.
3. Switching a component from hosted to local exposes latency defaults that were always wrong. The old provider was fast enough to hide it, so the bug arrived looking like "local is slow."
4. A setting can be read from a different config block than the one it obviously belongs in. Grep the source for the key before assuming the value is wrong.
5. Recency correlates with size in an agent session. Protecting the last 20 messages protects the tool outputs, which are most of your tokens.
6. A server's advertised context maximum is a capability, not a budget. Auto-detect it and your client will use all of it — and on a shared server, one session at 960k tokens holds 87% of everyone's cache.
7. Session config is pinned at creation. A correct fix produces no change in running sessions, and a stale log line looks exactly like a fix that didn't land. Compare log timestamps against process start time.