The finding
Every token-reduction tool is sold on the same promise: fewer tokens, smaller bill. We measured that promise across 2,908 paid sessions of Claude Code, Anthropic's coding agent: 103 real software tasks, 7 repositories, 3 models, every task run under four setups from identical fresh working copies. Every cost figure was read from the provider's actual bill, not estimated from a token counter.
| Setup | Tokens removed | Cost per completed task | | --- | --- | --- | | Claude Code, unmodified | none, the baseline | the reference | | RTK, open-source compressor, unmodified | -1.3% | -2.9%, not statistically significant | | RTK-ML, our experimental build on RTK | -38.4% | +6.8% | | Headroom v0.27.0, third party | not measurable | +46.4% |
The only setup that came out ahead is the one that barely compressed anything, and even that saving is not statistically significant. The tool that cut the most text, RTK-ML, was not cutting blindly: it was built to keep only the context relevant to the engineering task at hand, and it worked, removing 38.4% of the text. It still lost money.
Why the bill went up
A compression tool is not editing a document. It is editing what a working agent can see, in the middle of the job. When context was removed, the agent noticed things were missing: it searched again, re-read files it had already read, and took more turns to reach the same place. Every extra turn incurs additional context, reasoning and output cost.
Across everything tested, how much text a tool removed was a weak and unstable predictor of what it cost: the correlation is close to zero and its confidence interval crosses zero.
The ceiling
Our research found that token cost attributes can be broken down as follows: 75% of cost is the framework's own system prompt and tool definitions, and another 19% is the model's hidden reasoning. Everything a compression tool can even touch adds up to 6.0%, and the practical ceiling for current compression tools is about 5%. Vendors in this category routinely advertise 60 to 90%.
These are the measurements we concluded from our testing, which is based on simulated engineering sessions run on real software tasks and billed at actual provider prices; results can vary depending on the individual task.
Read the research
- [The research hub](/AI-Research) with the 54-second summary video
- [The full paper on arXiv](https://arxiv.org/abs/2607.12161), free, no registration
- [The research summary and industry perspective](/AI-Research/guide-download) (10 pages)
- [The open-source AI Efficiency Benchmark](https://github.com/PointFiveLabs/ai-efficiency-benchmark), so any savings claim, including PointFive's own, can be run through it
Disclosure: this study was authored at PointFive and is not independent research. We ran RTK, the open-source compressor; RTK-ML, our own experimental build that cuts tokens more aggressively; and Headroom, a third-party tool. This research informs the design of PointFive's token optimization product, built to improve visibility, compliance, and governance. Method, raw per-run data, and limitations are in the paper.