Token Optimization for AI Agents: What to Fix First

A customer writes a support ticket, and your AI agent starts work on it. AI agent token optimization is lowering the cost of each completed task by making re-read tokens cheaper, sending fewer of them, and making each remaining model call cheaper. In the simplified model this note builds, the ticket is 800 tokens, the agent needs twelve model calls to resolve it, and by the twelfth call it is sending 48,000 tokens. Where did they all come from? In the same simplified model, the run uses 364,800 input tokens, and about two thirds of its cost goes on re-reading text that was already in an earlier call of the same run.

An agent's bill is mostly re-reading. Make the re-read cheap, then make it smaller, then make each remaining call cheaper, and measure the cost of each completed task, because the tactics interfere with each other.

If you read only this:

• Cache the stable prefix first. It is the first and largest step as ordered here, and where caching is on by default, most of the work is putting stable content first.
• Shrink what you append, not what you already sent. Editing history that is already cached breaks the cache from that point on.
• Route whole tasks, set reasoning per task, and batch only work nobody is waiting for. Then judge every change by cost per completed task, not cost per call.

Open the diagram full size:

Where agent tokens go: the re-read

A chat assistant answers once. An agent loops. On each turn it sends its instructions, its tool definitions, the task and everything that has happened so far, then gets back one tool call or one answer. The tool's result is appended, and the whole prompt goes back for the next turn. Microsoft's guide to Azure OpenAI reasoning models describes the same mechanism in one line: these models "resend a growing conversation on every turn".

So the question to ask of any agent is not "how long is my prompt?" It is "how many times does each token get sent?" In a twelve-turn run, the system prompt is sent twelve times and the first tool result eleven times. Output tokens are expensive, but in this model each is billed once. Input tokens are cheaper, but the loop multiplies them.

The worked model: one support ticket

To make the tactics comparable, we cost one running example: an agent that resolves a customer support ticket by looking up the customer, their orders and the refund policy, then drafting a reply. This is a deliberately simplified model. Every token count is an assumption, chosen to be plausible rather than measured, and printed here so you can replace it with your own. The arithmetic is in code you can run, in "Run the arithmetic yourself" below. Prices are list prices for Gemini 3.5 Flash on the Global endpoint of Google's Gemini Enterprise Agent Platform, taken from the Agent Platform pricing page as of 29 October 2026. Google describes the platform as an evolution of Vertex AI, and its release notes refer to it as Agent Platform (formerly Vertex AI). Gemini 3.5 Flash is the priced reference for this model's arithmetic, not the newest Gemini model. Its Agent Platform model page shows it still generally available, with retirement listed as 19 May 2027 or later, though the Gemini Developer API has deprecated it and routes its requests to Gemini 3.6 Flash, as its Gemini API model page explains.

The prices in that table are Google's list prices as of 29 October 2026. They change often, so check the provider's page before relying on them.

In the simplified model, the baseline with no optimization comes to $0.73 per ticket. In this model, input is 75% of the baseline cost, and 87% of the input tokens were already in an earlier call of the same run, either sent to the model or written by it. At 20,000 tickets a month, the baseline in this model costs about $14,600 a month.

1. Prompt caching: make the re-read cheap

What it fixes. The repeated prefix. When the start of a prompt is byte-identical to the start of an earlier one, the platform can reuse its already-processed state for that prefix and bill those tokens at a discount. Caching changes the price and the latency. OpenAI's prompt caching guide says it can reduce the time spent processing input before the response starts, and Anthropic's prompt caching documentation that it cuts latency as well as cost for long documents. It does not change the answer. OpenAI's guide says plainly that caching "does not change how the model generates output tokens". The platform reuses a prefix; the model does not remember anything.

How. Put everything stable first: instructions, then tool definitions, then reference material. Put everything that changes last: timestamps, request IDs, the newest tool result. Keep the conversation append-only. Then check the platform's rules, because they differ in ways that change the economics.

Prices and caching rules are as of 29 October 2026. Both change often, so check each provider's page before you rely on them.

Two details in that table change the economics. First, where a cache write costs more than ordinary input (1.25 times on Anthropic, on OpenAI's newest models and on some Bedrock models), caching a prefix that is never read again is a surcharge, not a saving. Second, a prefix below the minimum is silently not cached. There is no error, only a zero in the usage field.

When not to. When the first thousand tokens differ on every request, there is nothing to reuse. When requests sharing a prefix arrive further apart than the cache lifetime, you pay the write and rarely the read. Implicit caching also comes with no promise: Google's caching documentation offers ways "to increase the likelihood of an implicit cache hit", and AWS says an eligible request is not assured of a cache hit. So a hit rate is something you measure, not something you assume.

Effect on the ticket. The prefix in the simplified model is 12,000 tokens, comfortably above Gemini's 4,096 minimum. If 80% of each turn's reusable prefix is served from cache, cost per ticket in the simplified model falls from $0.73 to $0.38, a 49% saving. In the same model, a 50% cache hit rate gives $0.51 per ticket and a 95% hit rate gives $0.31. The spread is the reason to measure your own.

2. Send less: compact tool results and load tools on demand

What it fixes. The size of what gets re-read. Once caching is on, every token you append is paid in full once and then at the cached rate on every later turn, so the cheapest token is the one you never append.

How. Two places carry most of the weight in an agent. Tool results: an order-lookup API that returns 3,000 tokens of JSON when the agent needs six fields should return the six fields. Do the extraction in the tool, in code, before the result reaches the model. Tool definitions: thirty tools described in full on every turn is a fixed tax. Load only the tools a task needs, or let the model search for them. Anthropic's tool search documentation, for example, describes a five-server setup whose definitions take about 55,000 tokens before the model does any work.

When not to. Do not trim by editing history the agent has already sent. A cache is a prefix, so changing turn three invalidates everything after turn three. Anthropic's context editing documentation says this directly: clearing tool results "invalidates cached prompt prefixes", and it suggests clearing enough at once to make the new cache write worthwhile. Compact at the source, once, before you append.

Effect on the ticket. In the simplified model, we cut tool results from 3,000 to 800 tokens and tool definitions from 9,000 to 2,500, so the fixed prefix becomes 5,500 tokens, still above Gemini 3.5 Flash's 4,096-token cache minimum (newer Flash models need 6,144). With compacted tool results and tools loaded on demand, cost per ticket in this model falls from $0.38 to $0.26.

Here is the interaction the model makes visible. Suppose that instead of compacting, you keep caching on and drop all but the last two raw tool results on every turn. In the simplified model, that per-turn trim removes most of the tool output yet saves only 4% to 7% over caching alone, because each edit breaks the cache from the edit point onward. The range depends on whether your platform can still match the unchanged tokens before the edit. Applied on top of compacted results in this model, the same per-turn trim lands between a 2% saving and a 2% loss. Compacting at the source saved 32% in this model. Savings from separate tactics do not simply add up.

3. Reasoning effort: set the thinking level per task

What it fixes. Thinking tokens. Reasoning models think before they answer, and the thinking is generally priced like output tokens where it is metered separately. Anthropic bills thinking as output tokens, per its thinking documentation. OpenAI bills reasoning tokens as output, as its reasoning models guide explains. Google's price list labels its output rate "response and reasoning". In this model, output costs six times as much as input, so 1,500 thinking tokens a turn is the largest single output line.

How. Use the control your model exposes, and treat it as model-specific. Google's thinking documentation says Gemini 3 and later models take a thinking_level. The older thinking_budget is deprecated for them and returns an error on Gemini 4. Current Claude models take an effort setting, and Anthropic's extended thinking documentation says Claude 4.7 and later reject the older fixed thinking budget. OpenAI's reasoning models take reasoning.effort. Pick the level per task type: a refund lookup does not need the depth of a disputed-charge investigation. Fewer thinking tokens also mean a faster answer, which is how Google frames the control: a way to optimize between latency and reasoning depth.

When not to. Change the level at the start of a task, not between turns. Both Anthropic's cache documentation and OpenAI's list effort among the settings that rewrite the prefix, so changing the request's effort setting mid-run invalidates the cache. On supported models, OpenAI documents a configuration update that changes effort without rewriting the earlier prefix. On supported models, Anthropic's per-message effort change (in beta) also keeps the cache, but it takes effect only from the next user turn with new input, not after a tool result. Also watch the output cap. If thinking uses up max_output_tokens, OpenAI and Microsoft both warn that you pay for the input and the reasoning and receive no answer.

Effect on the ticket. If this task type holds its quality at a lower level and thinking falls from 1,500 to 600 tokens a turn, cost per ticket in the simplified model falls from $0.26 to $0.16. Whether a task type holds its quality at a lower thinking level is the whole tactic. Lower thinking that causes a second attempt costs more than it saves, so validate it against your own evals.

4. Model routing: route the task, not the turn

What it fixes. Paying a capable model's price for work a smaller model does just as well. At the list prices used in this model, Gemini 3.5 Flash-Lite costs a fifth of Gemini 3.5 Flash's input price.

How. Classify the task once, at the start, with a cheap call. Run simple tasks end to end on the smaller model, and send the rest to the larger one. Check the smaller model's result, and when it fails, run the task again on the larger model. Routing whole tasks keeps each run on one model, so each run keeps one warm cache. OpenAI's caching guide lists the model itself among the settings that affect the cached prefix, so a run that switches model mid-way reaches the new model without the earlier model's cache. Managed routers exist on Bedrock (intelligent prompt routing) and Foundry (model router). They make the same choice for you, per request.

When not to. Before you build a cascade, measure the simpler alternative. Anthropic's guide to optimizing for cost and intelligence recommends sweeping effort levels on one model first and calls that curve "the baseline any multi-model strategy must beat". It also explains why routing savings disappoint: "a failed task still bills its tokens, then the retry".

Effect on the ticket. Assume, in the simplified model, that 60% of tickets are simple and go to Flash-Lite, 10% of the simple tickets fail and are re-run on Flash, and the classifier reads 1,200 tokens. With whole-ticket routing, cost per completed ticket in the simplified model falls from $0.16 to $0.10, and the failed attempts are paid for inside that number. If escalations triple to 30%, cost per completed ticket in this model is $0.12. If only 30% of tickets are simple, cost per completed ticket in this model is $0.13.

5. Batch and Flex: for work nobody is waiting on

What it fixes. Paying real-time prices for work that can wait. Discounts are roughly 50% for eligible workloads on all five platforms, in exchange for slower or deferred results.

Discounts and terms are as of 29 October 2026. They move too, so check the provider's page before relying on them.

When not to. A customer waiting on a reply is not batch work. A multi-turn tool loop pushed through an asynchronous batch waits a batch cycle per turn. The natural fits around an agent are the jobs around it: evaluation replays before each release, backfills, nightly re-classification and offline research.

Effect on the ticket. The live ticket loop stays on standard pricing. In the simplified model, replaying 500 tickets through the optimized agent before each release costs about $79 at standard rates and about $40 on Flex.

The receipt, before and after

Open the chart full size:

Read the percentages as the shape of a result, not a promise. The order changes how much each step appears to save, and your token profile changes everything. The order still matters for a practical reason: each step makes the next one easier to measure. Caching tells you what your real re-read costs. Compaction shows what the model actually needs. Only then is a routing experiment comparing like with like.

Open the diagram full size:

Run the arithmetic yourself

Every number marked "in the simplified model" comes from the script below. Paste the three blocks into one file and run it with Python 3. The output matches the figures in this note. The blocks run in order: the cost model, then the five steps and the receipt, then the interaction checks. Change any assumption in the first block and every figure moves with it.

# Prices in USD per 1M tokens: input, cached input, output (response and reasoning).
# Gemini 3.5 Flash and Flash-Lite, Global endpoint, Standard PayGo, as of 29 Oct 2026.
FLASH, LITE = (1.50, 0.15, 9.00), (0.30, 0.03, 2.50)
FLEX = (0.75, 0.075, 4.50)  # Flex PayGo: 50% off Flash
MIN_CACHE = 4096            # assumed implicit-cache minimum (Gemini 3)
# Every token count is an assumption, not a measurement.
BASE = dict(turns=12, system=3000, tools=9000, ticket=800, result=3000,
            out=200, think=1500, hit=0.0)  # hit: share of reusable prefix served from cache

def ticket(p, px, keep=None, reuse="longest"):
    """USD, input, re-read and output tokens for one ticket.
    keep=N drops tool results older than the last N (naive trimming); reuse says
    whether the cache can match tokens after an edit ("longest") or not ("fixed")."""
    fixed, usd, tin, reread, tout = p["system"] + p["tools"], 0, 0, 0, 0
    prev = fixed if fixed >= MIN_CACHE else 0  # prefix warmed by other tickets
    for step in range(p["turns"]):             # step = tool results already appended
        kept = step if keep is None else min(step, keep)
        prompt = fixed + p["ticket"] + step * p["out"] + kept * p["result"]
        same = prompt - (p["result"] if step else 0)  # unchanged lead of the last prompt
        if kept < step:                               # history edited: match shrinks
            same = fixed + p["ticket"] + ((step - kept) * p["out"] if reuse == "longest" else 0)
        cached = p["hit"] * (min(prev, same) if prev >= MIN_CACHE else 0)
        out = p["out"] + p["think"]
        usd += ((prompt - cached) * px[0] + cached * px[1] + out * px[2]) / 1e6
        tin, tout, prev = tin + prompt, tout + out, prompt
        reread += prompt - p["result"] if step else 0
    return usd, tin, reread, tout

def routed(p, simple=0.6, escalate=0.10, clf_in=1200, clf_out=20):
    """Classify once; simple tickets run on Lite, and a failed Lite run is redone on Flash."""
    clf = (clf_in * LITE[0] + clf_out * LITE[2]) / 1e6  # assumed classifier call
    flash, lite = ticket(p, FLASH)[0], ticket(p, LITE)[0]
    return clf + simple * (lite + escalate * flash) + (1 - simple) * flash

p1 = dict(BASE, hit=0.8)             # 1 cache the stable prefix
p2 = dict(p1, result=800, tools=2500)  # 2 compact results, load tools on demand
p3 = dict(p2, think=600)             # 3 lower thinking level for this task type
usd, tin, reread, tout = ticket(BASE, FLASH)
costs = [usd] + [ticket(p, FLASH)[0] for p in (p1, p2, p3)] + [routed(p3)]
for n, c in enumerate(costs):
    print(f"step {n}: ${c:.4f}/ticket  ${c * 20000:,.0f}/month  cumulative saving {1 - c / usd:.1%}")
print(f"input tokens {tin:,.0f}; input is {tin * FLASH[0] / 1e6 / usd:.1%} of cost; "
      f"re-read is {reread / tin:.1%} of input and {reread * FLASH[0] / 1e6 / usd:.1%} of cost")
print(f"turn-12 prompt {BASE['system'] + BASE['tools'] + BASE['ticket'] + 11 * (BASE['out'] + BASE['result']):,}")

for h in (0.5, 0.95):                # sensitivity to the cache hit rate
    print(f"caching alone at {h:.0%} hit rate: ${ticket(dict(BASE, hit=h), FLASH)[0]:.4f}")
for reuse in ("longest", "fixed"):   # naive trim: keep only the last 2 raw results
    for name, p in (("cached", p1), ("compacted", p2)):
        trimmed = ticket(p, FLASH, keep=2, reuse=reuse)[0]
        print(f"trim on top of {name} ({reuse}): {1 - trimmed / ticket(p, FLASH)[0]:+.1%} saving")
print(f"compacting vs caching alone: {1 - costs[2] / costs[1]:.1%} saving")
for simple, escalate in ((0.6, 0.3), (0.3, 0.1)):
    print(f"routing {simple:.0%} simple, {escalate:.0%} escalate: ${routed(p3, simple, escalate):.4f}")
print(f"500-ticket replay after step 3: standard ${500 * costs[3]:.2f}, Flex ${500 * ticket(p3, FLEX)[0]:.2f}")

The output prints the values used above: $0.7308, $0.3757, $0.2558, $0.1586 and $0.0964 per ticket, monthly totals from $14,616 to $1,929, 364,800 input tokens, a 48,000-token twelfth prompt, 74.9% of cost in input and 65.5% in re-reading, and $79.31 against $39.65 for the replay. The two trim variants give the 4% to 7% saving on raw results and the 2% saving to 2% loss on compacted ones, quoted in step 2. The prices are the 29 October 2026 list prices from the text above, and the Flex line assumes half the Standard rate, as the Flex row in the batch table states.

Measure cost per completed task

Every tactic above can lower cost per call while raising cost per completed task, by adding turns, retries or escalations. The unit that cannot fool you is spend divided by tasks completed. The code below computes it from the counts every platform returns. Prices are placeholders, so substitute your own.

# Prices are placeholders, USD per 1M tokens.
# Priced reference: Gemini 3.5 Flash, Global, Standard PayGo (as of 29 Oct 2026),
# not the newest model; the current example model is gemini-3.8-flash.
PRICE = {"input": 1.50, "cached": 0.15, "cache_write": 1.50, "output": 9.00}

def call_cost(uncached, cached, cache_write, output_incl_thinking, p=PRICE):
    return (uncached * p["input"] + cached * p["cached"]
            + cache_write * p["cache_write"]
            + output_incl_thinking * p["output"]) / 1_000_000

def cost_per_completed_task(tasks):
    """tasks: [{"calls": [dict of the four counts, ...], "completed": bool}]"""
    spend = sum(call_cost(**c) for t in tasks for c in t["calls"])
    done = sum(1 for t in tasks if t["completed"])
    return spend / done if done else float("inf")

The four counts come from different fields on each platform. Read them from every response and keep them with the task ID.

Mind the totals. Anthropic's input_tokens already excludes cached and written tokens. On Gemini, OpenAI and Foundry, the prompt total includes them, so subtract them to get the uncached count. Check the same on Bedrock for your model.

Next to those, record turns per task and escalations per task. A drop in tokens per call alongside a rise in turns per task is the commonest way a saving turns out not to be one.

What this model does not tell you

• It is a model, not a measurement. Every token count is an assumption. Run the same arithmetic on a week of your own usage records before you quote any saving.
• Tokens are not portable. Each vendor uses its own tokenizer, so the same ticket is a different number of tokens on each platform. Compare vendors on cost per completed task, never on price per million tokens alone.
• Hit rates are yours to discover. Implicit caching is best effort, cache lifetimes vary by model, and traffic that splits across projects, workspaces or regions splits the cache too.
• Compaction can remove what the agent needed. If a trimmed tool result forces a second lookup, you have traded tokens for turns. Watch turns per task.
• Some platforms re-send earlier reasoning. On Claude models that keep thinking from all turns, and by default on GPT-5.6 models, earlier reasoning stays in context and is billed as input on later turns. This model does not re-send thinking, so on those platforms the re-read share is higher.
• Lower thinking and smaller models move quality. Our routing line assumes the smaller model uses the same number of turns. Often it does not.
• Prices move, and some are promotional. Several models on the Agent Platform list carry introductory or promotional pricing with end dates. Regional endpoints cost more: 10% more on the non-global rows of the Agent Platform prices used here, and a 1.1 times multiplier for US-only inference on Anthropic's API. Re-price before you decide.

Takeaways

• The loop multiplies input. In the simplified model, about two thirds of the cost was re-reading text already sent.
• Caching is the first move, and ordering is most of the work. Stable first, volatile last, append-only.
• Shrink at the source. Edit what you append, not what is already cached.
• Set reasoning and model per task, not per turn, so the cache survives the decision.
• Batch the work around the agent, such as evaluations and backfills, rather than the live loop.
• Judge everything on cost per completed task, with turns and escalations counted.

Reread-me checklist

• Nothing that changes per request (timestamps, IDs, user names) appears before the stable prefix.
• The stable prefix is above your model's cache minimum, and cached reads are non-zero from the second call onward.
• Cache writes are followed by reads; a prefix written and never read is a surcharge.
• Tool results are reduced to the fields the agent uses before they are appended.
• Tools are loaded per task, or searched for, rather than all sent on every turn.
• The reasoning level is set per task type and is not changed mid-run.
• Output caps leave room for thinking, and incomplete responses are handled.
• Routing happens once per task; failed small-model runs are re-run and counted.
• Evaluation replays and backfills run on Batch or Flex.
• Spend, turns and escalations are recorded per task ID.

What's next

Token tactics lower the price of each task. Deciding which tasks are worth running, who pays for them, and how budgets and alerts behave in practice is a separate discipline. We cover it in FinOps for AI agents. The other half of "send less", deciding what belongs in the context window for quality rather than cost, is context engineering.

In AELSHA™ ASTRA, the framework we use to deliver every agent engagement, cost is a release gate: an agent leaves the Harden phase only with a security and FinOps sign-off, and every trace that breaches a cost threshold is kept rather than sampled away.

One question to take back to your own logs: in your busiest agent, what share of last week's input tokens were already sent once in the same run? Send us that number and your turns per task, and we will tell you which of the five tactics we would try first.

Further reading

• Prompt caching, OpenAI: The clearest table of which request settings rewrite the cached prefix, the model and reasoning effort among them.
• Prompt caching, Anthropic: Breakpoints, lifetimes, minimums and an invalidation table worth reading before you place your first cache marker.
• Context caching overview, Google Cloud: How implicit and explicit caching differ on Agent Platform, with the minimum prefix sizes and when storage is billed.
• Optimizing for cost and intelligence, Anthropic: A short case for sweeping effort levels on one model before you build a router, and for why failed tasks shrink routing savings.
• Estimate and cap LLM costs before an AI agent runs, AELSHA™ NEHEMVIX: A low, likely and high cost range before anything is spent, and a spending cap enforced with ledger reservations.

Prices, model names, cache rules and preview features change often, so check each provider's own page before you rely on the numbers here.