Why do input tokens drive cost while output tokens drive latency?

Input is cheaper per token but arrives in bulk, and prefill is heavily parallel. Output is priced several times higher and is produced by strictly sequential forward passes. On a 150,000-token RAG call with 500 output tokens at 5 dollars per million in and 25 out, input is 98.4% of the bill while those 500 output tokens account for essentially all of the wait. Halving your input barely moves perceived latency; halving your output nearly halves it.

When does LLM prompt caching actually pay off?

On the first reuse, for the short TTL. With a 5-minute cache write at 1.25x base input and reads at 0.1x, writing then reading n times costs 1.25 + 0.1n against 1 + n uncached, so break-even lands around n = 0.28. The 1-hour TTL at 2x breaks even near n = 1.11, which makes it a fit for gappy traffic rather than higher volume. For consistently one-shot calls, caching makes you slightly worse off.

Why is my prompt cache not working?

Usually the prefix is too short or not byte-stable. Minimum cacheable prefix length varies 8x across a single vendor lineup and does not track model tier, so short prompts routed to a cheap model can fall under the minimum and cache nothing, with no error returned. Unstable JSON key ordering in tool schemas, conditionally including tools per request, and a timestamp or per-user value in the system prompt each invalidate the prefix.

How should I order a prompt for maximum cache hits?

Most static to most dynamic: tool and function definitions, then system prompt and policy, then few-shot examples, then long static documents, then the cache breakpoint, and after it retrieved RAG chunks, conversation history and the current user turn. On Anthropic the prefix is tools then system then messages, and a change at any level invalidates that level and every level after it.

Are reasoning tokens billed differently from output tokens?

They are billed at the output rate, which is the most expensive line in the price list, and you pay for thought tokens you never see. Defaults are expensive and essentially nothing defaults to off, so an unset effort parameter means paying the vendor's quality-optimized guess on every request including trivial ones. Reasoning effort is also the largest latency lever that exists.

Input Tokens Drive Cost, Output Tokens Drive Latency: LLM Cost Engineering in 2026

Here is the decomposition that reorganizes everything else.

Time to first token is dominated by prefill: processing all your input tokens. It scales roughly linearly with input length, and it parallelizes across the sequence. Time per output token is dominated by decode: one forward pass per token, memory-bandwidth-bound, strictly sequential. End to end is TTFT + (TPOT × output_tokens).

Now price a realistic long-context RAG call. Claude Opus 5, 150,000 input tokens, 500 output tokens, at $5 per million in and $25 per million out.

Cost: input is 0.15 × $5 = $0.75. Output is 0.0005 × $25 = $0.0125. Input is 98.4% of the bill.

Latency: the 150k-token prefill is heavily parallel. The 500 output tokens are 500 strictly sequential forward passes. At roughly 54 tokens per second, that is about 9.3 seconds of pure decode.

So the two levers point in opposite directions. To cut cost on this workload you cache the prefix, prune retrieved context, and lower top-k. To cut latency you shorten the output. Halving your input barely moves perceived latency. Halving your output nearly halves it.

Conflating these is the most common analytical mistake I see. A cheaper model is not automatically a faster one. Optimize the right axis.

What follows is what I have verified about the current landscape, as of July 30, 2026. Prices move constantly, so treat every figure here as a starting point to re-verify rather than a constant to hardcode. That warning is not boilerplate: Claude Sonnet 5's introductory pricing ends August 31, 2026, taking it from $2/$10 to $3/$15 per million tokens on September 1. That is a 50% increase on both sides.

Prompt caching, precisely

Caching is the single largest lever on context-heavy workloads, and most teams implement it in a way that silently does not work.

The multipliers. On Anthropic: a 5-minute cache write costs 1.25x base input, a 1-hour write costs 2x, and a read costs 0.1x. OpenAI has converged on the same model for its current generation: reads at 10% of input (a 90% discount) and writes at 1.25x. This is a change worth flagging, because OpenAI's original 2024 announcement offered a 50% discount with no write charge. Anyone whose cost model dates from 2024 has both numbers wrong.

Google is different: implicit caching gives a 75% discount rather than 90%, and it is on by default for Gemini 2.5 and newer.

Break-even is one reuse. With a 5-minute write at 1.25x and reads at 0.1x, writing then reading n times costs 1.25 + 0.1n against 1 + n uncached. Break-even lands around n = 0.28. Caching pays on the first reuse. For the 1-hour TTL at 2x, break-even is about n = 1.11, so the long TTL is for gappy traffic rather than higher volume.

The corollary matters: for consistently one-shot calls, caching makes you slightly worse off. You pay the write premium and never amortize it.

Minimum prefix lengths vary 8x across a single vendor's lineup, and not monotonically with tier. On Anthropic, Opus 5 and Fable 5 cache from 512 tokens, Sonnet 5 and Opus 4.8 from 1,024, Opus 4.7 from 2,048, and Opus 4.6, Opus 4.5, and Haiku 4.5 from 4,096. Haiku 4.5 sits at the top of that range, tied with two Opus models and eight times the Opus 5 minimum. The cheapest model in the lineup has the strictest caching requirement.

Sit with that. Teams route short prompts to Haiku specifically to save money, land under the 4,096-token minimum, and get zero caching. No error is returned. Caching simply does not happen. On Gemini the minimums also went up with the newer models, from 2,048 on the 2.5 line to 4,096 on 3.x.

And clearing the minimum is necessary, not sufficient. A published third-party measurement on GPT-5.5, self-run and not peer-reviewed, found a roughly 1,300-token prompt got zero cache hits across 15 calls, while a 3,700-token prompt got 9 hits across 14. On a hit, 3,328 of 3,741 input tokens came from cache, cutting input cost by 80%, not the advertised 90%, because the variable tail never caches. Their conclusion is the one to take to your finance team: you cannot promise a flat 90% reduction.

The ordering rule

Caches match on an exact token-sequence prefix. So sort your content from most static to most dynamic:

1. Tool / function definitions      <- most static
2. System prompt, persona, policy
3. Few-shot examples
4. Long static documents
   ----- cache breakpoint -----
5. Retrieved RAG chunks
6. Conversation history
7. Current user turn                <- most dynamic

On Anthropic the prefix is explicitly [tools] -> [system] -> [messages], and a change at any level invalidates that level and every level after it.

Which makes tool definitions the highest-leverage and most commonly botched item in the whole system. Three consequences:

Serialize tool schemas deterministically. A JSON serializer with unstable key ordering silently destroys your entire cache, on every request, forever, with no error.

Do not conditionally include tools per request. Adding one tool for 5% of requests costs the other 95% a full re-prefill. If you need variation, use two or three fixed tool-set variants rather than assembling dynamically.

Watch the non-obvious invalidators. On Anthropic, changing tool definitions invalidates everything. Changing tool_choice invalidates only the messages cache, leaving tools and system intact. Toggling web search or citations invalidates system and messages, as does the speed setting, and changing output_config.effort always invalidates message blocks. That last one matters, because this post is about to tell you to tune effort.

This is also, incidentally, why the MCP spec now says servers should return tools/list in deterministic order, and says so explicitly to improve prompt cache hit rates. If your tool list reshuffles between calls, you are paying for it. I wrote about what else changed in that revision if you run MCP servers.

The most common silent failure is a timestamp or request ID in the system prompt. Cache writes happen only at breakpoints, so if the block containing the breakpoint changes, the prefix hash differs and you get nothing. Lookback will not rescue stable content sitting behind a poisoned breakpoint.

The second most common is per-user personalization in the system prompt. Injecting a username or tenant ID gives every user a distinct prefix and drops your effective hit rate to roughly 1/N of your user count. Move personalization after the breakpoint. If it genuinely must be early, bucket it by locale or tier rather than individuating it.

RAG chunks, and an inversion worth considering

Retrieved chunks are dynamic by construction, so they cannot live in a shared cached prefix. The default answer is correct for most systems: put retrieval last, cache everything static, pay full price only on the chunks.

But there is a second pattern that has quietly become viable. If your corpus fits in context, skip retrieval and cache the corpus. At cache-read rates as of this writing, a cached 500k-token corpus costs about $0.25 per query on a frontier model, or $0.10 on the mid tier (rising to $0.15 when Sonnet 5's introductory pricing ends). That is frequently cheaper than running an embedding query plus reranking plus paying full input price on twenty chunks, and it is strictly better on recall.

This trade flipped as 1M contexts became standard and cache reads landed at 10% of input. If your corpus is under roughly half a million tokens, run the arithmetic before you build a retrieval pipeline. Sometimes the pipeline is the expensive part.

Google bills for cache storage by time, and it changes the math

Google is the only major provider that charges for cache storage per token-hour. On a Pro model at the Priority tier, caching a 200k-token prefix costs 0.2 × $8.10 = $1.62 per hour in storage alone, roughly $39 a day, whether or not you read it. On Standard the rate is $4.50, so $0.90 an hour.

A 200k-token uncached read on that model costs $0.72, so the storage charge equals roughly 2.25 full uncached reads per hour. Each cached read saves you about $0.648, which puts break-even at $1.62 / $0.648 = 2.5 reads per hour. So explicit Gemini caching only pays if you read the cache more than about two and a half times an hour, every hour. That ratio holds on both the Standard and Priority tiers. That is a genuinely different cost structure from Anthropic and OpenAI, where an unused cache costs nothing after the write, and it deserves its own line item in your model.

Reasoning tokens are the fastest-moving line on your bill

All three providers bill reasoning tokens at the output rate. Google's docs say the quiet part directly:

"Pricing is based on the full thought tokens the model needs to generate, despite only the summary being output from the API."

You are billed for tokens you never see, at the most expensive rate in the price list.

Defaults are expensive. Gemini 3.1 Pro defaults to high. OpenAI defaults to medium. Essentially nothing defaults to off. If you have never set an effort parameter, you are paying for the vendor's quality-optimized guess on every request, including the trivial ones.

The controls, currently: Anthropic uses output_config: {effort: ...} with five levels, low through max, on Opus 4.5 and later, defaulting to high, or budget_tokens on older models. Note that adaptive is a thinking mode, not an effort value, and the two are easy to conflate. OpenAI's reasoning.effort has seven levels, none through max. Google uses thinking_level.

OpenAI's effort: "none" exists specifically for latency-critical tasks without reasoning benefits, and it is the cheapest single config change available for a classification or extraction endpoint.

Two traps. Anthropic notes that budget_tokens is a target, not a strict cap, so do not treat it as a cost guarantee, and that budgets above 32k should go through batch processing to avoid timeouts. And on streaming, Anthropic's thinking-token breakdown appears only on the final message_delta event. If your instrumentation closes the span on the last content delta, you lose reasoning-token accounting entirely, which means you are silently under-reporting the most expensive tokens in every request.

Reasoning effort is also the largest latency lever that exists. Measured time-to-first-token at maximum reasoning effort runs into the tens of seconds, sometimes past two minutes, while small non-reasoning models come in around 0.3 to 0.4 seconds. That is a span of two to three orders of magnitude, and nearly all of it is reasoning-token generation rather than model size or context length. Neither Anthropic nor OpenAI publishes a quantitative effort-versus-quality curve, which means measuring it on your own workload is the highest-value experiment on this entire list.

One more cost fact that is invisible in every price table. Anthropic's model migration guide notes that Claude 4.7 and later produce roughly 30% more tokens than earlier models for the same text. The tokenizer changed. Migrating between generations at an identical headline price is a ~30% cost increase on identical inputs. If you are estimating a migration using token counts from an older model, you will underestimate by about a third.

Batch, and the discount that stacks

All three majors offer 50% off for batch. The differences are operational.

	Anthropic	OpenAI	Google
Discount	50% in and out	50%	50%
SLA	most within 1 hour, all within 24	24h window only	targets 24h
Max requests	100,000	50,000	not stated
Max size	256 MB	200 MB	2 GB per file
Retention	29 days	not stated	6 weeks

Two things people miss.

Batch stacks with caching. Anthropic's docs say so explicitly, and the arithmetic is good: 50% batch times a 90% cache-read discount puts cache reads at 5% of list input price. The caveat is that batch cache hits are best-effort, with the docs citing hit rates anywhere from 30% to 98% depending on traffic pattern, so model the floor rather than the ceiling.

OpenAI batch draws from a separate rate-limit pool. For teams that are throughput-constrained rather than cost-constrained, batch is a capacity release valve, not just a discount.

Batch qualifies for evals and benchmark runs, bulk classification and extraction, embedding backfills, synthetic data generation, moderation sweeps, nightly summarization, and offline index construction. It disqualifies anything with a human waiting, anything with multi-turn dependencies inside the batch, and anything needing streaming.

There is also a middle tier worth knowing: Google's Flex is priced identically to batch but synchronous, and OpenAI's flex processing is analogous with a default 10-minute timeout that usually needs extending. One nice property of OpenAI flex: you are not charged when a 429 Resource Unavailable occurs.

Routing: the number everyone quotes and the number they omit

The canonical published router result is LMSYS's RouteLLM work. The headline is that the best router hit 95% of GPT-4 performance using only 14% of GPT-4 calls, 75% cheaper than the random baseline, on MT Bench.

Here is the part that gets left out. On MMLU, the same technique needed 54% of GPT-4 calls for the same 95% performance, which was only 14% cheaper than random.

Same approach, different benchmark, and in fact two different routers: matrix factorization on MT Bench, a causal LLM router on MMLU. 75% savings on one, 14% on the other.

The explanation is not about the router. MT Bench is open-ended conversation with wide difficulty variance, so there are lots of easy queries a small model handles fine. MMLU is uniformly hard knowledge questions, with little variance to exploit.

Routing gains are a function of your query difficulty distribution, not of your router. A team whose traffic is uniformly hard gets close to nothing from routing regardless of implementation quality. Measure your difficulty distribution before you build a router. That measurement is cheap, and it will frequently tell you not to build the router.

If you do route, the cascade break-even is simple. With cheap-model cost c, expensive cost e, and escalation rate r, cascading wins when r < 1 - c/e. For a 5x price ratio, cascading wins as long as you escalate less than 80% of the time. For a 15x ratio, less than 93%. Those are forgiving thresholds, which means the real risks are not token arithmetic. They are the latency penalty on escalated requests, since you pay both models serially, and the cost of the verification step itself.

One tension that is under-discussed: cross-provider routing costs you prompt-cache locality. Every provider switch is a cache miss and a full-price prefill. Route within a family to preserve cache, and route across families only for workloads that do not cache well anyway.

Also worth noting that the providers have absorbed part of this. Adaptive thinking lets the model decide how much to reason per request, and OpenAI's docs describe models reasoning adaptively and using fewer tokens for simpler tasks. That is intra-model difficulty routing, capturing some of what external routers used to provide.

The other levers, ranked by how much they actually move

Cut output tokens first. Output is priced at 5 to 6 times input across essentially every provider, and it is the only thing that costs sequential wall-clock time. Structured outputs eliminate preamble and trailing commentary, which on short responses can be a large fraction of total output. Stop sequences halt generation at a known terminator. Prompting for explicit length is a cost control. And do not ask a reasoning model to show its reasoning in the output, because you are then paying twice for the same work at the same rate.

Set max_tokens to your real ceiling, but understand it is a truncation guard rather than a cost guarantee. You pay for what was generated before truncation, and a truncated response usually gets regenerated, which costs more.

Tune top-k, because chunks are the part you cannot cache. At 500 tokens per chunk on a $5/MTok model, k=20 costs $0.05 per query uncached and k=5 costs $0.0125. At a million queries a month that difference is $37,500. Recall improvements are typically sublinear in k while cost is exactly linear, so there is a knee. Find it against your eval set instead of defaulting to 20.

There is a Gemini-specific wrinkle here: on Pro models, crossing 200k input tokens doubles the input price and raises output price by 50% for the entire request, not just the marginal tokens. A pipeline that drifts from 190k to 210k sees a step function, not a slope. Dropping back under the cliff is worth far more than the proportional token saving.

Compact agent context, carefully. Anthropic published a genuinely striking result: in a 100-turn web search evaluation, context editing let agents complete workflows that would otherwise fail from context exhaustion while reducing token consumption by 84%, and the memory tool plus context editing improved benchmark performance by 39% over baseline. That is the rare optimization that also improves quality, because context exhaustion and attention dilution were the binding constraints rather than model capability.

But the caching interaction is a trap. Clearing tool results invalidates the cached prefix. You pay a cache write every time clearing fires. Naive compaction can increase your costs: clear 5k tokens, invalidate a 100k cached prefix, then pay 100k times 1.25 to rewrite it. Tune the trigger and the minimum-clear amount so each clearing event removes enough to justify breaking the cache. Clearing thinking blocks, by contrast, preserves the cache when thinking is kept.

If you are building long-running agents, this is the same problem as engineering context across time, just viewed through the billing system.

Try fine-tuning last, not fourth. The usual argument is that fine-tuning removes tokens from every request by baking instructions and few-shot examples into weights. True. But prompt caching already removes about 90% of the cost of those same tokens, and costs nothing up front.

And on some providers, fine-tuned inference is priced above the equivalent base model, so check your provider's table before you model this. Where that holds, you are paying a per-token premium for the fine-tune and must recoup it purely on token-count reduction, on top of amortizing the training. That is a meaningfully harder bar than the usual framing suggests.

The order to try things: prompt engineering, then caching, then few-shot with caching, then distillation to a smaller model, then fine-tuning. Most teams jump three steps ahead.

Embeddings are never the problem. At current rates, embedding a 10-million-token corpus costs cents. Embeddings run 50 to 600 times cheaper than the cheapest generation input token. If embedding cost shows up in your budget, you have a duplicate-job bug, not an economics problem.

Self-hosting break-even is a utilization question

Run the arithmetic honestly. Cost per million output tokens is hourly_rate / (tok_per_sec × 3600) × 1,000,000.

Take a 9B model on a single H100 at roughly $5.49/hour, measured at about 145 tokens per second single-stream, or about 424 with speculative decoding.

145 tok/s: $10.52 per million output tokens
424 tok/s: $3.60 per million

Both at 100% utilization, which nobody achieves. At a realistic 30%, those become $35.06 and $11.98.

Compare against serverless for a comparable model at $0.25 per million output tokens, or $0.08 on the fastest inference hosts.

Single-stream self-hosting of a 9B model runs 14 times more expensive than serverless at 100% utilization, and 140 times at 30%. The entire economic case rests on batching: those throughput figures are single-stream, and a production vLLM deployment serves dozens of concurrent requests from the same weights, with aggregate throughput scaling well past single-stream until memory bandwidth saturates.

So the break-even is a utilization question, not a price question. You need sustained, high-concurrency, predictable load. The real reasons teams self-host are data residency, latency floors, custom weights, and capacity guarantees. When cost is the stated reason, the analysis has usually assumed a utilization rate that never materializes.

While we are here: speculative decoding gave that roughly 2.9x throughput improvement, and the metric that determines whether it works is acceptance rate. 60 to 80% is the sweet spot yielding 2 to 3x. Below 50% means a poorly aligned speculator or a workload too creative to predict. It trades a slight TTFT regression for a large TPOT win, so it is a clear gain on long outputs and can be a net loss on very short ones.

Instrument it, or you are guessing

The OpenTelemetry GenAI semantic conventions moved out of the main semantic-conventions repo into a dedicated one. The old opentelemetry.io/docs/specs/semconv/gen-ai/* paths now serve a redirect notice and are unmaintained. Any tooling or blog post pointing at the old path is stale.

The attributes worth capturing: gen_ai.provider.name and gen_ai.operation.name (both required), gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and critically gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens, and gen_ai.usage.reasoning.output_tokens. Metrics include gen_ai.client.operation.time_to_first_chunk and time_per_output_chunk, with server-side counterparts, plus a new gen_ai.workflow.duration for agent-level spans.

All of it is at Development stability. Expect churn.

Two things to know before you build on it. There is no cost attribute in the spec. Dollar attribution is entirely on you or your vendor. And gen_ai.usage.input_tokens should include all input tokens, including cached ones, which means computing spend requires subtracting the cache attributes rather than adding them. Getting this backwards double-counts cached tokens at full price and inflates your reported spend. I have seen this bug in the wild more than once.

Provider semantics differ in exactly the way that produces this bug. Anthropic's usage.input_tokens means tokens after the last cache breakpoint, so uncached only. OpenAI's prompt_tokens is the total, with cached broken out underneath. A cross-provider cost calculator that treats them identically will misprice one of them. Write the per-provider mapping explicitly, then reconcile it against an actual invoice.

Also: Anthropic's token counting endpoint is free, has independent rate limits, and accepts the same structured inputs as message creation. That is the right primitive for pre-flight budget enforcement. Count before you send, reject or compact if over budget.

The metrics that survive a conversation with finance are not token counts. They are cost per resolved ticket or completed task, where the denominator must be a successful outcome so failed retries do not look free. Cost per session and per active user per month, compared directly against ARPU. Cache hit rate as cache_read / (cache_read + cache_creation + input), tracked per prompt version and alerted on regression, because one deploy that perturbs a system prompt can silently multiply your input bill. Reasoning-token share of output. And p50/p95/p99 for TTFT and TPOT tracked separately, never blended.

Propagate session_id, user_id, feature, prompt_version, and model as span attributes, then compute cost at query time from a versioned price table. Prices change. A hardcoded price constant silently misreports your entire history the day a vendor adjusts a rate.

The failure modes that actually blow up bills

A widely circulated practitioner postmortem describes an autonomous agent that entered a retry loop on a Friday night and ran until Monday morning. Sixty-three hours, roughly 4,800 iterations per hour, about 302,000 total iterations, and $4,200. It is a self-published account with an anonymized customer and reconstructed logs, so treat the figure as illustrative, but the mechanism is exactly right and I have watched smaller versions of it happen.

The root cause was one instruction: keep trying until it works. On HTTP 429 rate-limit errors. With no cost awareness, no timeout, and no escalation path.

Retry-on-error plus an unbounded agent loop is a money pump. A 429 means you are going too fast. An agent that responds by retrying immediately, forever, converts a rate limit into a billing event. The postmortem's own assessment is the right one: the model was never the bottleneck. The failure was architectural. A $50 ceiling would have killed the loop and paged a human, turning a $4,200 incident into a $50 one.

Retries on long contexts are the multiplier nobody models. A 200k-token request costs $1.00 in input. Three retries cost $4.00. A retry storm across a thousand concurrent requests during a provider degradation costs $4,000 in minutes, for zero successful responses. And retries are usually cache-cold if they land on a different backend, so you pay full prefill each time. Note also that some SDKs retry timeouts twice by default, which means the true cost of a request can be up to 3x your naive estimate.

Cap retries. Use jittered exponential backoff. Never retry on 4xx other than 429. Make retry budgets per-request rather than per-call-site. And enforce token, dollar, iteration, and wall-clock budgets at the framework level, not by asking for them in the prompt. A prompt is not a control plane.

Cache thrash is the highest-frequency silent failure, because nothing errors. You just quietly pay ten times more on input, forever, until someone reads a bill closely.