AI model cost breaks down into pieces that rarely show up as one clean number: infrastructure, per-token API cost, and the token volume itself. Here's how to actually take it apart.
The three layers of AI model cost
Infrastructure cost. GPU instances, managed model-serving platforms like Bedrock, Vertex, or Azure OpenAI, and any self-hosted serving infrastructure. This is the layer most cloud cost tools already track well.
Per-token API cost. The price charged per input and output token, which varies by model and provider. This is where most published cost comparisons stop.
Token volume. How many tokens a given workflow actually consumes, which depends on context size, prompt design, and how much noise (build logs, raw HTML, repeated file reads) makes it into the request. This is the layer almost no tool tracks, and it's often the largest lever for cost reduction.
Why token volume is the layer worth focusing on
Infrastructure cost and per-token pricing are largely fixed once you've chosen a provider and model. Token volume is the one variable you actually control day to day, and it's also the one most teams have no visibility into. A session with a bloated context window and one with a lean, well-managed context can cost meaningfully different amounts for functionally the same task.
How to calculate cost per inference
Start with (input tokens plus output tokens) multiplied by the provider's per-token rate for that model. The hard part isn't the math, it's knowing your actual token count per request, since most teams only see aggregate monthly totals, not per-session detail. Endpoint-level tooling that classifies and logs token usage per session is what makes this calculation possible at a granular level instead of a rough monthly average.
Where token waste actually comes from
Based on TokenShift's own optimization data across beta customers, the most common sources of avoidable token volume are build logs pasted into context instead of a pass or fail summary, raw HTML included instead of cleaned text, files re-read repeatedly instead of cached, inefficient commands like unoptimized grep calls, and high-resolution screenshots included at full size instead of resized to what the model actually needs.
What a full cost breakdown should include
Infrastructure spend by provider and instance type, API spend by model and by team, and token volume by developer and session, ideally broken down further by which of the categories above is driving it. Most tools give you the first two. TokenShift is built specifically to fill in the third, running locally on the endpoint where that detail actually lives.
Frequently asked questions
How do I calculate cost per inference for our AI models?
Multiply total tokens (input plus output) by the model's per-token rate. The practical challenge is getting an accurate token count per request rather than a rough monthly average, which requires endpoint or API-level logging, not just a provider invoice.
What's the biggest lever for reducing AI model cost?
Token volume, not infrastructure or per-token pricing, since it's the variable most directly shaped by how prompts and context are constructed. Compression techniques applied at the endpoint, before a request reaches the model, typically show the fastest measurable reduction.
The bottom line
Most AI cost reporting stops at infrastructure and per-token pricing, the two layers that are already fixed once you've picked a provider. Token volume is the layer you can actually act on, and the one worth instrumenting first.
Methodology
This guide is based on public provider pricing and TokenShift's own product documentation. For corrections, reach out at pointfive.co/contact.