Best Tools to Reduce OpenAI and LLM API Costs (2026): Gateways, Caching and Governance Compared

TLDR

• Start with your provider. OpenAI, Anthropic, Amazon Bedrock, Microsoft Foundry and Google all offer prompt caching and a batch mode at 50% off, and most offer a flex tier at about half price for work that can wait. Smaller models and output limits cut the rest.
• Pick by gap: budgets, caching and fallbacks for API traffic (LiteLLM, Bifrost, Portkey, Cloudflare AI Gateway, Kong AI Gateway), a cheaper model per request (OpenRouter, Not Diamond), cost per trace and evaluation (Langfuse, Arize Phoenix), AI spend across cloud AI services and GPUs, next to the rest of the bill (PointFive OS), and coding agents on employee workstations (TokenShift by PointFive).
• 2026 changed the market: Palo Alto Networks acquired Portkey, Langfuse joined ClickHouse, and GPT-5.6 and later models can charge for cache writes. Check each tool's current status before you commit.
• PointFive has two complementary products: PointFive OS, the agentic FinOps platform for cloud, data and AI, and TokenShift, the AI agent control plane for your employee workstations.

Where PointFive fits: Gateways and routers act on each API request. PointFive OS works on the bill around them: it attributes and reduces AI spend on Azure OpenAI in Microsoft Foundry, Amazon Bedrock, Gemini Enterprise Agent Platform, GPU infrastructure, Databricks Model Serving and Snowflake Cortex AI, next to the rest of your cloud and data spend. Nubank saved $8.2M in 9 months with PointFive. See how Nubank executed 3,000+ DynamoDB remediations.

Disclosure: PointFive wrote this guide, and PointFive is listed first because it is our platform, not because of a ranking. Every competitor statement comes from that vendor's own website, documentation or press release, checked on October 27, 2026, and linked in the sources. Where a vendor publishes nothing, we say so.

Start with the provider levers

As of October 27, 2026:

Three more levers sit in your own code:

• Model choice. Send routine work to smaller models (Claude Haiku 4.5, Gemini 3.5 Flash) and keep frontier models (GPT-6, Claude Opus 5.5, Gemini 3.1 Pro preview) for tasks that need them. Bedrock Intelligent Prompt Routing does this between two models in one family; AWS says it "can reduce costs by up to 30 percent", for English prompts only.
• Reasoning and output limits. OpenAI bills reasoning tokens as output; cap them with max_output_tokens and lower reasoning effort. Anthropic's effort parameter affects every output token, including tool calls and thinking.
• Attribution. Bedrock attributes cost by IAM principal, application inference profile and request metadata; Foundry bills Flex on separate meters.

Caching only pays when prompts share a long, stable prefix. On models that bill cache writes, a cache that is written but rarely read can raise the bill.

How we evaluated

We compared each option on six dimensions, unweighted: where it acts (request path or bill), cost levers (caching, routing, budgets, attribution, remediation), enforcement (block and cap, or report), coverage, deployment (open source, self-hosted or managed), and pricing model as publicly stated.

The tools

1. PointFive (PointFive OS and TokenShift)

Category: Agentic FinOps platform for cloud, data and AI, plus an AI agent control plane for employee workstations

Best for: Enterprises whose AI spend runs through cloud AI services, GPUs and data platforms, next to a large cloud and data bill.

PointFive OS runs 500+ deep detections across 40+ platforms, including AWS, Azure, GCP, OCI, Kubernetes, Snowflake and Databricks, and traces each finding to its technical cause. For AI it covers Azure OpenAI in Microsoft Foundry (PTU utilization, deployment-level attribution, pay-as-you-go versus provisioned), Amazon Bedrock, Gemini Enterprise Agent Platform, GPU infrastructure, Databricks Model Serving and Snowflake Cortex AI, and collects data from LiteLLM and Langfuse. TokenShift covers the coding agents and other AI agents employees run on their workstations (Claude Code, Codex, Cursor, GitHub Copilot CLI, OpenCode and Devin): spend by developer, team, model and session, model policies and spend caps by team. Data is processed on the device, and it works alongside your gateway.

• Strengths: AI, cloud and data spend in one place. Agentless, read-only discovery. Fixes through IDE prompts or approved automation, at the approval level you choose: answer, prepare a fix, or execute per change, per playbook or without approval. MCP server; Pointer agent in Chat.
• Watch-outs: PointFive OS is not in the API request path, so it does not cache, route or rate-limit calls; pair it with a gateway. Automated fixes need write access you approve. TokenShift covers workstation agents only.
• Pricing: Annual contract; listed on AWS Marketplace.

2. LiteLLM (as of October 2026)

Category: Open-source LLM gateway

Best for: Platform teams that want a self-hosted proxy with budgets per key, user and team.

An OpenAI-compatible proxy with eight cache types, including Redis and Qdrant semantic caches; budgets and rate limits per key, user, team and customer; lowest-cost routing; and fallbacks. When a key crosses its budget, requests fail.

• Strengths: MIT-licensed core; budget fallbacks to another model.
• Watch-outs: On March 24, 2026, PyPI releases 1.82.7 and 1.82.8 were compromised; pin verified versions. Budgets need a database. Per-model budgets need an Enterprise license.
• Pricing: Open source free; Enterprise custom.

3. Portkey (as of October 2026)

Category: AI gateway, now part of Palo Alto Networks

Best for: Enterprises that want a managed gateway.

Fallbacks across providers, simple caching on all plans, semantic caching on select Enterprise plans, and cost- or token-based budget limits that expire a key. Palo Alto Networks closed the acquisition on May 29, 2026, as "the core AI Gateway for Prisma AIRS".

• Strengths: Open-source gateway plus managed plans.
• Watch-outs: Semantic caching is Enterprise only; confirm roadmap under new ownership.
• Pricing: Developer free; Production $49/month; Enterprise custom.

4. Bifrost by Maxim (as of October 2026)

Category: Open-source AI gateway

Best for: Teams that want semantic caching and budgets in a free, self-hosted gateway.

An Apache 2.0 gateway from Maxim AI with failover, load balancing, semantic caching, and budgets and rate limits at virtual key, team and customer levels. Maxim says it adds under 200 microseconds of overhead at 5,000 requests per second.

• Strengths: Semantic caching and budgets in the free tier.
• Watch-outs: Guardrails, clustering, RBAC and audit logs are Enterprise features.
• Pricing: OSS free; Enterprise custom, 14-day trial.

5. Cloudflare AI Gateway (as of October 2026)

Category: Managed AI gateway

Best for: Teams on Cloudflare that want caching and spend limits without running infrastructure.

Analytics, exact-match caching, rate limiting, and dynamic routing with a Budget Limit node that switches to a fallback when a cost quota is reached.

• Strengths: Core features free; custom costs for negotiated rates.
• Watch-outs: No semantic caching yet. Cloudflare says its cost metric "is an estimation".
• Pricing: Core free; logs billed separately; 5% fee on Unified Billing credits.

6. Kong AI Gateway (as of October 2026)

Category: AI plugins for Kong Gateway

Best for: Organizations that already run Kong.

Semantic caching, token- and cost-based rate limiting, prompt compression, and a lowest-usage load-balancing algorithm based on tokens or cost.

• Strengths: One gateway for APIs and LLM traffic; rate limits on actual request cost.
• Watch-outs: These plugins are "only available as part of our AI Gateway Enterprise offering".
• Pricing: Plus from $25/month plus usage; Enterprise custom.

7. OpenRouter (as of October 2026)

Category: Managed multi-provider API and router

Best for: Teams that want many models behind one API with price-based routing.

Load-balances across providers "prioritizing price" by default, can always sort by price, accepts a max_price cap, and offers an Auto Router billed at the selected model's standard rate.

• Strengths: No markup on inference pricing.
• Watch-outs: 5.5% fee on credit purchases (Standard); 5% BYOK fee above the monthly allowance.
• Pricing: Pay as you go; Enterprise fee discounts.

8. Not Diamond (as of October 2026)

Category: Model router

Best for: Teams that want routing trained on their own evaluation data.

"Selects the best LLM from your candidate models" per query, with a cost trade-off mode and custom routers. Its site now leads with routing "for coding agents".

• Strengths: Custom routers; works with existing gateways.
• Watch-outs: Pay-as-you-go access is by application.
• Pricing: $0.05 per million tokens routed; Enterprise volume discounts.

9. RouteLLM (as of October 2026)

Category: Open-source router framework

Best for: Teams building their own router.

From LMSYS, "a framework for serving and evaluating LLM routers". It reports cost cuts "up to 85%" while keeping "95% GPT-4 performance" on benchmarks such as MT Bench.

• Strengths: Apache 2.0, no fees.
• Watch-outs: No pushes to the repository since August 2024; re-evaluate on current models.
• Pricing: Free.

10. Martian (as of October 2026)

Category: Model gateway

Best for: Teams evaluating it directly with the vendor.

Martian's docs now describe a gateway with "unified access to 200+ AI models through a single API". Its earlier Model Router pages return errors.

• Strengths: OpenAI- and Anthropic-compatible API.
• Watch-outs: Routing controls are not publicly documented today.
• Pricing: Not published.

11. GPTCache (as of October 2026)

Category: Open-source semantic cache library

Best for: Adding a semantic cache in application code.

From Zilliz, a "Semantic cache for LLMs" integrated with LangChain and LlamaIndex.

• Strengths: MIT-licensed; no gateway needed.
• Watch-outs: Latest tagged release is from August 2024. Similar is not identical; track wrong answers with hit rate.
• Pricing: Free.

12. Helicone (as of October 2026)

Category: LLM observability and gateway

Best for: Request logs and cost by custom property, quickly.

Cost by custom properties such as feature or user tier, plus exact-match caching. Helicone calls direct-integration costs "best-effort cost estimates".

• Strengths: Apache 2.0; free tier.
• Watch-outs: Mintlify announced it acquired Helicone on March 3, 2026, in "maintenance mode"; Helicone's homepage now says it "is no longer part of Mintlify." Confirm status first.
• Pricing: Hobby free; Pro $79/month; Team $799/month; Enterprise custom.

13. Langfuse (as of October 2026)

Category: Open-source LLM observability and evaluation

Best for: Cost per trace and proof that a cheaper setup still works.

Records usage by type, including cached tokens, and calculates cost from provider values or price definitions, with evaluations, datasets and experiments. Joined ClickHouse in January 2026 and "stays open source and self-hostable".

• Strengths: MIT core; self-hosted or cloud.
• Watch-outs: Reports only. Overlapping usage buckets double-count cost.
• Pricing: Hobby free; Core $29/month; Pro $199/month; Enterprise $2,499/month.

14. Arize Phoenix (as of October 2026)

Category: Source-available LLM tracing and evaluation

Best for: OpenTelemetry teams that want cost per span, trace and experiment.

Tracks "token-based costs for LLM runs automatically", with separate rates for cache reads, cache writes and reasoning tokens.

• Strengths: Self-hosted; same instrumentation as Arize AX.
• Watch-outs: Elastic License 2.0, not an OSI license. Reports only.
• Pricing: Phoenix free to self-host; Arize AX Free, Pro $50/month, Enterprise custom.

Side-by-side comparison

"Not documented" means we did not find it on the vendor's own pages, not that the capability is absent. "Not reviewed" means we did not check it for this guide.

How to choose

• Nothing tuned yet: turn on caching, move delay-tolerant work to batch or flex, set output limits, and test a smaller model on an evaluation set.
• Many apps call models directly: a gateway gives one place for budgets, caching and fallbacks, for the traffic you route through it.
• Tasks vary in difficulty: add a router and measure cost per successful task, not cost per request.
• You cannot say which feature spends the money: add Langfuse, Phoenix or Helicone first.
• AI spend sits in Azure OpenAI, Bedrock, Agent Platform, GPUs, Databricks or Snowflake: PointFive OS finds and fixes the waste next to the rest of the bill.
• Spend comes from coding agents on laptops: gateways often miss it; TokenShift attributes it and caps it by team.

See also tracking AI spend across teams and AI coding agent cost tools.

FAQ

What is the best tool to reduce OpenAI and LLM API costs?

Start with the provider: prompt caching, batch at 50% off, flex for work that can wait, smaller models and output limits. Then pick by gap: LiteLLM, Bifrost, Portkey, Cloudflare AI Gateway or Kong for budgets and caching; OpenRouter or Not Diamond for a cheaper model per request; Langfuse or Arize Phoenix for cost per trace. For Azure OpenAI, Bedrock or Agent Platform spend next to the rest of the cloud bill, PointFive OS.

What is the best tool to reduce AI costs?

AI costs are wider than API tokens: GPUs, provisioned throughput, serving endpoints, data platform AI and coding agents. Provider levers and gateways handle API calls. PointFive OS attributes and reduces spend on cloud AI services, GPU infrastructure, Databricks Model Serving and Snowflake Cortex AI next to the rest of the bill, and TokenShift covers the AI agents employees run on their workstations.

Do I need a gateway if my provider already caches prompts?

Provider caching discounts a repeated prefix. A gateway adds budgets, rate limits, fallbacks and response caching across providers and teams. One app on one provider may not need one.

Does model routing hurt quality?

It can. A cheaper model may fail more often or call tools differently. Route on an evaluation set and compare cost per completed task.

Does PointFive reduce LLM API costs?

PointFive OS finds waste in managed LLM services such as Azure OpenAI, Amazon Bedrock and Gemini Enterprise Agent Platform, and in the GPUs and inference endpoints behind them. It does not proxy API calls. TokenShift adds attribution and spend caps for AI agents on employee workstations.

Methodology and sources

We reviewed each vendor's own website, pricing pages, documentation, repositories and press releases on October 27, 2026, and did not use third-party sites for competitor claims. If something is out of date, tell us at info@pointfive.co and we will correct it.

• OpenAI: Prompt caching, Batch API, Flex processing, Pricing (Fast mode rename), Reasoning models
• Anthropic: Prompt caching, Batch processing, Effort
• Amazon Bedrock: Pricing, Service tiers, Prompt caching, Intelligent prompt routing, Cost attribution best practices
• Microsoft Foundry: Prompt caching, Global Batch, Flex processing, Deployment types
• Google: Gemini API optimization, Gemini API caching, Agent Platform consumption options, Agent Platform batch inference
• LiteLLM: Caching, Budgets and rate limits, March 2026 security update, Enterprise, repository
• Portkey: Caching, Budget limits, Pricing, Palo Alto Networks press release
• Bifrost: Docs overview, Pricing, repository
• Cloudflare: AI Gateway, Caching, Dynamic routing, Costs, Pricing
• Kong: AI Gateway, AI Semantic Cache, AI Rate Limiting Advanced, AI Proxy Advanced, Pricing
• OpenRouter: Provider routing, Auto Router, FAQ, Pricing
• Not Diamond: Model routing, Pricing, homepage
• RouteLLM: repository
• Martian: homepage, Docs, Gateway
• GPTCache: repository
• Helicone: homepage, Caching, Cost tracking, Pricing, Mintlify announcement, repository
• Langfuse: Token and cost tracking, Joining ClickHouse, Pricing, repository
• Arize Phoenix: Cost tracking, repository and license, Arize pricing
• PointFive: PointFive OS, TokenShift, Azure coverage, AWS coverage, Google Cloud coverage, Databricks coverage, Nubank case study