Token reduction is not cost reduction

The entire study ran in Claude Code, Anthropic's coding agent: three compression tools tested inside the agent, with unmodified Claude Code as the baseline. 2,908 paid sessions. Every cost was read from the provider's bill, and the measurement plan was locked in before a single session ran, so the results could not be cherry-picked. The harder a tool compressed, the more it cost.

The lesson is not that one product wins. It's that higher compression currently leads to higher cost.

Our research found that token cost attributes can be broken down as follows.

• 75% of cost is the framework's own system prompt and tool definitions, and another 19% is the model's hidden reasoning.
• Everything a compression tool can even touch adds up to 6.0%.

The framework baggage is re-sent and re-read on every turn of a session, before your task contributes a word. The bill is not driven by the prompt volume. It is driven, turn after turn, by everything the machinery has to remember and think in order to answer.

These are the measurements we concluded from our testing, which is based on simulated engineering sessions run on real software tasks and billed at actual provider prices; results can vary depending on the individual task.

Most cost compression tooling works on the parts of a session it can intercept: what the tools return and what the user prompts. Everything a tool can even touch adds up to 6.0%, and the share of it that a real compression layer can strip without breaking the run puts the working ceiling at about 5%.

The other 94% is locked: the agent framework's own instructions and tool definitions, and the model's hidden reasoning. No optimizer can compress what it never gets to see.

These statistics reflect real-life engineering scenarios as represented in the benchmark. Other use cases may differ. In research-heavy scenarios, for example, the share of addressable surface accumulates to well beyond 6%, and the saving potential grows from 5% toward 30%, a factor of six.

We built RTK-ML, an experimental layer on top of the open-source RTK, designed to keep only the context relevant to the engineering task at hand. It worked: it cut 38.4% of the text flowing to the agent. It should have saved money. It cost 6.8% more instead. And we were not the worst of it: Headroom, a third-party tool, cost 46.4% more on the same tasks.

Both numbers point the same way. Trimming what the agent can see changes how it works: it takes extra turns to recover what was removed, and every extra turn re-sends the entire history again. The savings come back as a bill.

Across everything we tested, how much text a tool removed turned out to be a weak and unstable predictor of what it cost: the correlation is close to zero and its confidence interval crosses zero. A token count in a lab says nothing about the bill, because the bill depends on how many turns the session takes, and that responds to what you delete.

Aggressive compression doesn't only fail to save; it can destroy the agent's ability to act. For example: to change a file, an agent has to quote the existing code back exactly, character for character. Compress that quote and the edit will not apply.

To test the effect of compression on performance and outcome quality, we benchmarked Claude Code with compression tools against the native version. For average engineering tasks, no degradation was observed.

For stress testing, we ran the same comparison against SWE, a benchmark of highly challenging debugging tasks, to see whether the patch-apply rate, the agent's ability to identify the problematic context behind a bug, was affected. In these extreme scenarios, a degradation was observed: 15 of the 40 challenges succeeded under compression, versus 27 of 40 without it.