Token Optimization Tools (2026): Approaches, Examples, and How to Evaluate Them

Token optimization is not one mechanism. Provider discounts, gateways, routers, caches, compression, observability, and agent-local optimization each act on a different part of the request path, and each saves money in a different way. Most teams end up combining several.

This guide explains each approach, what it reduces, where it falls short, and which tools document it today. It is written by PointFive, the maker of TokenShift. It is not an independent ranking or a performance benchmark: tools are listed as examples of an approach, in no order, and capabilities are taken from each vendor's own documentation.

TLDR

• Start with what your provider already offers. Prompt caching and batch processing are documented by Anthropic, OpenAI, Amazon Bedrock, Microsoft Foundry, and Google, and often cut the price of eligible tokens substantially with little engineering.
• Gateways (LiteLLM, Portkey, Cloudflare AI Gateway, Kong AI Gateway) centralize routing, budgets, rate limits, and caching for traffic that passes through them.
• Routers (OpenRouter, Not Diamond, Bedrock Intelligent Prompt Routing) choose a provider or model per request, trading cost against quality.
• Semantic caching and compression (GPTCache, LLMLingua, and gateway plugins) cut repeated or unnecessary context, with quality trade-offs you need to measure.
• Observability (Langfuse, Helicone) shows where tokens go; it does not change requests by itself.
• Agent-local optimization (TokenShift) acts where coding and production agents run, including traffic a gateway never sees.
• Evaluate on your own workload, using cost per completed task, not tokens removed.

Where token cost comes from

Before choosing a tool, know which part of the bill you are trying to reduce:

• Input tokens: prompts, system instructions, retrieved documents, tool definitions, and conversation history. In agents, input usually dominates, because context is re-sent on every step.
• Output tokens: typically priced higher per token than input.
• Cached tokens: cache reads are billed at a discount on most providers, and some providers now also charge for cache writes.
• Retries, failed tasks, and extra steps: tokens spent on work that did not succeed.
• Tool and platform charges: gateway fees, routing fees, storage for cached context, and subscriptions.

Each approach below reduces a different line. A tool that removes input tokens but causes extra retries can raise the bill.

1. Provider-native caching and batch discounts

The cheapest optimization is often already in your provider's API. As of October 7, 2026:

• Anthropic (Claude API): prompt caching bills cache reads at 0.1x the base input price for most models (lower for some newer models), with cache writes at 1.25x (5-minute TTL) or 2x (1-hour TTL). The Batch API reduces costs by 50%.
• OpenAI: prompt caching is on by default for supported models, with cached input discounted by up to 95%. For GPT-5.6 and later, cache writes cost 1.25x the uncached input rate. The Batch API offers a 50% discount with a 24-hour window.
• Amazon Bedrock: prompt caching bills cache reads at each model's cache-read rate (see the pricing page for the model you use), and it is not supported together with batch inference. Batch inference is priced 50% below on-demand for select models, and the Flex tier is 50% below Standard.
• Microsoft Foundry (Azure OpenAI): prompt caching is on by default for prompts of 1,024 tokens or more, with an input discount on Standard deployments and up to 100% on provisioned deployments. From GPT-5.6, cache writes can incur charges. Global Batch costs 50% less than Global Standard.
• Google Gemini API: implicit context caching is on by default for Gemini 2.5 and newer models. Google's optimization guide lists a 90% discount on cached tokens plus prorated storage, and Batch and Flex at 50% of Standard pricing. On Google Cloud, these services now sit under Gemini Enterprise Agent Platform (formerly Vertex AI).

What it reduces: the price of repeated input, and the price of work that can wait. Trade-offs: caching only helps when prompts share a long, stable prefix and arrive within the cache lifetime. Prefixes that change on every call (timestamps, reordered tool lists) defeat it. Batch only fits work that tolerates hours of latency.

For worked examples, see the Efficiency Hub entries on missing prompt caching in Amazon Bedrock and Azure OpenAI batch deployments.

2. AI gateways

A gateway sits between applications and model providers. It gives one place to track spend, enforce budgets and rate limits, route, fall back, and cache.

• LiteLLM: an open-source, self-hosted proxy with exact and semantic caching, budgets and rate limits per key, user, and team, and routing strategies including lowest-cost routing and fallbacks. In March 2026, two LiteLLM releases on PyPI (1.82.7 and 1.82.8) were compromised; pin versions and verify releases.
• Portkey: an AI gateway with open-source and managed options, fallbacks, simple caching on all plans and semantic caching on select Enterprise plans (docs), and budget limits on Enterprise and select Pro plans. Palo Alto Networks completed its acquisition of Portkey in May 2026.
• Cloudflare AI Gateway: a managed gateway with exact-match caching, rate limiting, and dynamic routing that can switch to a fallback when a budget limit is reached. Its cost figures are estimates; Cloudflare recommends checking the provider's dashboard for exact costs.
• Kong AI Gateway: AI plugins for Kong Gateway, including semantic caching, token- and cost-based rate limiting, prompt compression, and routing by cost or latency, each documented as part of the Enterprise offering.

What it reduces: uncontrolled spend, duplicate requests, and expensive models used where policy says they should not be. Trade-offs: a gateway only sees traffic routed through it. Coding agents on laptops, SaaS tools with embedded AI, and hosted agents often bypass it, so check each integration rather than assuming coverage. A gateway also adds a hop to the request path and becomes part of your security surface.

3. Model routing

Routers choose which provider or model handles each request. Because model prices differ by an order of magnitude or more, sending simple requests to smaller models is one of the largest savings available.

• OpenRouter: a managed API across many providers. Provider routing can prioritize price, and an Auto Router selects a model for each request. Requests that fall back are priced on the model actually used. Check the FAQ for platform fees.
• Not Diamond: a router that predicts which model will give the highest-quality response at the lowest cost, with a cost trade-off setting and custom routers trained on your own evaluation data. Pricing is per token routed.
• Amazon Bedrock Intelligent Prompt Routing: routes between two models in the same family. AWS's pricing page says it can reduce costs by up to 30 percent without compromising accuracy; the user guide still describes it as a preview that supports a limited set of models and is optimized for English prompts.

What it reduces: the price per request, by not paying frontier-model prices for simple work. Trade-offs: routing changes the model, which can change answer quality, format, and tool-calling behavior. Measure task success per route, not just cost. For a method, see cost per successful task.

4. Semantic and response caching

Provider caching discounts a repeated prefix. Response caching skips the model call entirely when a similar enough request has been answered before.

• GPTCache: an open-source semantic cache library. The repository is not archived, but its latest tagged release is from August 2024; check maintenance status before depending on it.
• Gateway caches: LiteLLM and Kong offer semantic caching, Portkey offers it on select Enterprise plans, Cloudflare AI Gateway caches exact matches, and Helicone offers response caching.

What it reduces: repeated calls for repeated questions, such as support answers, FAQ-style queries, and classification of recurring inputs. Trade-offs: a semantic match can return an answer to a question that is similar but not the same. It suits stable, low-risk answers and is a poor fit for personalized or rapidly changing content. Track hit rate and wrong-answer rate together.

5. Prompt compression

Compression shortens the context sent to the model by removing tokens that contribute little.

• LLMLingua: Microsoft's open-source compression methods (LLMLingua, LongLLMLingua, and LLMLingua-2). The project reports up to 20x compression with minimal performance loss on its benchmarks. Kong's prompt compressor plugin is built on LLMLingua-2.

What it reduces: input tokens on long prompts. Trade-offs: compression changes the prompt, which can break a cached prefix and change results. Measure on your own tasks. See the compression and caching evaluation guide for the cost categories and research limitations.

6. Observability and evaluation

You cannot optimize what you cannot attribute. Observability tools show which application, feature, user, or step spends the tokens, and evaluation tools show whether a cheaper configuration still does the job.

• Langfuse: tracing with token and cost tracking, including cached tokens, and evaluations such as LLM-as-a-judge, datasets, and experiments. Available as a cloud service or self-hosted. Langfuse joined ClickHouse in January 2026 and says it remains open source and self-hostable.
• Helicone: request logging, cost tracking, cost-based rate limits, and response caching. Helicone joined Mintlify in March 2026 and says its services will remain live in maintenance mode, so weigh that before adopting it.

What it reduces: nothing directly. Observing cost is different from changing the model or the request. Its value is in showing which of the other approaches to apply, and proving whether they worked. Trade-offs: determine which components collect data, which recommend a change, and which enforce it. A low token count or a short response is not a quality measure by itself; use traces and completed-task outcomes. For more options, see our guide to open-source LLM observability tools.

7. Agent-local optimization with TokenShift

Coding agents and many production agents make model calls from places a gateway does not see: developer workstations, containers, and hosted agent environments.

TokenShift provides visibility, governance, and optimization for supported AI agents on employee workstations, in production containers, through SDK integrations, and in hosted environments. It is part of PointFive and complements PointFive OS.

Analysis runs where the agent runs. Derived metadata and analysis results reach the control plane; prompt content does not. Model requests follow the selected model's execution path.

Context optimization reduces unnecessary context while preserving the selected model. Smart Routing is a separate opt-in module that can select a different model. Evaluate cost, latency, and task quality for each enabled mode. For the wider practice of profiling production agents and fixing their most expensive step, see production AI agent cost optimization.

Side-by-side comparison

How to choose

1. Turn on provider caching and batch where they fit. It requires the least change and applies to traffic you already have.
2. Measure before you optimize. Attribute spend by application, team, and task, so you know which line of the bill matters.
3. Find where your traffic actually flows. If most spend comes from coding agents and hosted agents, a gateway alone will miss it.
4. Add routing where tasks vary in difficulty, with an evaluation set that proves the cheaper route still succeeds.
5. Use semantic caching and compression selectively, on workloads where you can measure the quality impact.
6. Check vendor status. Several tools in this market changed ownership in 2026; review licensing, deployment model, and maintenance before standardizing.

Put spending statistics in context

Menlo Ventures estimated $37 billion in enterprise generative-AI spending in 2025. Its $7.3 billion departmental category included approximately $4 billion for coding. Coding spending is not the whole departmental total.

Gartner's May 2026 release forecast worldwide AI spending of $2.59 trillion in 2026, up 47%. This is a forecast with a broader scope than Menlo's enterprise generative-AI estimate; the totals should not be combined or compared as the same market.

Historical declines in selected inference prices do not establish what a particular team pays today. Consult the provider rate card and your contract, then measure the actual mix of models and usage.

Compare on a shared workload

Use the same representative tasks and success criteria. Record all billable token categories, tool charges, optimization overhead, retries, and subscription costs where applicable. Check deployment effort, data handling, supported agents, and policy exceptions alongside the result.

Frequently asked questions

What is token optimization?

Token optimization is reducing what you pay for the tokens AI models process, by sending fewer tokens, paying less per token, or avoiding calls entirely, without reducing the quality of completed work.

What is the easiest way to reduce token costs?

Use the prompt caching and batch processing your provider already offers. Both are documented by the major providers and need little engineering, as long as prompts share stable prefixes and the work can tolerate delay.

Do I need an AI gateway?

A gateway helps when many applications call models directly and you need one place for budgets, rate limits, routing, and caching. It only governs traffic that passes through it, so agents on workstations and hosted environments may need a different approach.

Does model routing hurt quality?

It can. A cheaper model may answer differently or fail more often. Route on an evaluation set, and compare cost per successful task, not cost per request.

Is fewer tokens always cheaper?

No. Removing context can cause retries, break cached prefixes, or lower success rates, which can raise the total bill. See our research on why token reduction is not cost reduction.

Methodology

Capabilities, prices, and ownership changes were taken from each vendor's official documentation, pricing pages, or announcements, linked above, and checked on October 7, 2026. Tools are listed as examples of each approach, not ranked. Prices, plans, and supported integrations change; confirm them with the vendor before deciding. This guide avoids fixed catalog counts and market-wide savings percentages that the cited sources do not support.