Context Engineering for AI Agents: What the Model Reads

Picture an agent working through a disputed invoice. Context engineering is the work of deciding, for every model call an AI agent makes, what goes into its context window, in what form, and in what order. On the agent's third turn the customer writes one clear sentence: credit the amount against next month's invoice, and do not refund to the card on file, because that card is closed. Thirty-seven tool calls later, the agent refunds the closed card. Its instructions did not change and the model did not get worse. What changed is what it was reading. In the simplified model later in this note, by then about 125,000 tokens of invoice records and old API responses have piled up after that one sentence.

Failures like this look like reasoning failures. Often they are reading failures: the model answered the material it was handed, and the material had drifted away from the case.

An agent does not remember your case. It rereads the pack you assemble, every turn.

If you read only this:

• Treat the context window as something you assemble for each call, from six sources: instructions, tool definitions, retrieved documents, memory and notes, tool results, and conversation history.
• More context is not free quality. Published studies show models using long inputs less reliably than short ones, so the goal is the smallest set of material that does the job.
• In long runs, reach for the lightest tool first: clear old tool output, then summarise older turns, keep durable facts in notes placed near the end, and give heavy reading to a sub-agent.

What context engineering is

Anthropic, in Effective context engineering for AI agents, describes context engineering as "the set of strategies for curating and maintaining the optimal set of tokens (information)" a model works with, and states the aim as "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome." Prompt engineering is about writing good instructions. Context engineering covers everything else in the window too, across every turn of a long run.

The reason it matters is plain once stated. Anthropic's Messages API documentation says it directly: "The Messages API is stateless, which means that you always send the full conversational history to the API." Some providers can hold history for you. As OpenAI's conversation state guide explains, the Responses API chains calls with previous_response_id, and the Conversations API stores a conversation as a durable object, so your code need not resend earlier turns. Prompt caching, separately, lets a call resume from a stored prefix instead of reprocessing it, as Anthropic's prompt caching documentation describes. Neither changes what the model reads. OpenAI's guide also notes that all previous input tokens in a chain are still billed as input tokens, which is the plain sign that on every call the model works from the whole window, whether your code, your framework or the provider assembles it. Google's engineering write-up on context in its Agent Development Kit (ADK) puts the same idea as a design principle: "Context is a compiled view over a richer stateful system." Sessions, memory and files are the sources; the window is what you compile from them for one call.

A useful picture is the briefing pack a chief of staff prepares before a board meeting. The board member reads only the pack. If the decisive page is buried in an appendix behind last quarter's raw spreadsheets, the decision suffers, however able the reader. The difference with an agent is the twist: it rereads the entire pack, from the first page, before every step it takes.

Open the diagram full size:

Why more context can make an agent worse

Long windows tempt teams to include everything. The research says that has a price in quality.

• Position matters. Liu and colleagues' Lost in the Middle study (Transactions of the ACL, 2024) found that performance "is often highest when relevant information occurs at the beginning or end of the input context", and drops when it sits in the middle, "even for explicitly long-context models."
• Length matters even when the needle is findable. The NoLiMa benchmark (ICML 2025) tested 13 models that claim at least 128,000 tokens of context on questions with minimal wording in common with the passage that answers them, so the model has to infer the link. At 32,000 tokens, 11 of the 13 fell below half of their short-context scores.
• Similar-looking material distracts. Chroma's technical report on what it calls context rot (July 2025) found performance degrading consistently as input length grew, even on simple tasks, with distracting passages making it worse.
• Several facts at once are harder than one. Google's Gemini long-context documentation warns that when you look for multiple specific pieces of information, "the model does not perform with the same accuracy."

Anthropic's context-engineering article offers the mechanism in plain terms: models have an "attention budget", and a transformer relates every token to every other, giving "n² pairwise relationships for n tokens." That is the simplified picture for standard dense attention; some models use variants that limit it, such as the sliding-window attention described in the Mistral 7B paper. The practical point holds either way: each token you add draws on the budget and adds to the cost of the call.

These studies measured the models of their day on benchmark tasks. Newer models may do better, and your task is not their benchmark. What carries over is the direction: a bigger window is a bigger pack, not a more careful reader. Validate the size and shape of your context against your own evaluations.

Six sources, and the question to ask of each

Every token in the window comes from one of six sources. For each, the useful questions are the same: what goes wrong, what to do, when not to bother, and what it changes for the invoice agent.

Instructions

What goes wrong: instructions either hard-code brittle rules for every case or stay so vague they assume knowledge the model lacks. Anthropic calls the target the "right altitude" between those two failures.

What to do: state how to decide, not every decision. Use a few diverse, canonical examples rather than a list of edge cases; Anthropic's phrase is that examples are "the 'pictures' worth a thousand words."

When not to: do not paste fast-changing policy into instructions. Retrieve it, so it can change without a prompt release.

The invoice agent: instructions say how to choose between a credit and a refund and when to escalate. The full credit policy lives elsewhere.

Tool definitions

What goes wrong: every loaded tool definition is reread on every call. Anthropic's tool search documentation gives an example multi-server setup that uses about 55,000 tokens of definitions before any work starts, and says Claude's ability to pick the right tool degrades once more than 30 to 50 tools are available.

What to do: load the few tools a task always needs, and let the agent search for the rest. Anthropic's tool search marks the others with defer_loading so they enter the window only when found. If tools arrive through MCP servers, each server you connect adds definitions; our note on MCP and A2A covers how those servers are scoped.

When not to: the same documentation says standard tool calling fits better below about ten tools, or when every tool is used in every request.

The invoice agent: five billing and CRM tools stay loaded; seven rarely used ones wait until searched for.

Retrieved documents

What goes wrong: loading whole documents up front, just in case, fills the window with pages the current step does not need.

What to do: retrieve just in time. Anthropic describes agents that keep "lightweight identifiers (file paths, stored queries, web links, etc.)" and load the content when a step needs it. ADK does the same with artifacts: by default an agent sees "a lightweight reference (a name and summary)" and loads the full file on request.

When not to: if every step needs the same document, loading it once at the start is simpler and keeps the start of the window stable.

The invoice agent: one policy section, on credits for closed cards, is fetched when the case reaches that question. Three whole policy documents are not loaded at turn one.

Memory and notes

What goes wrong: facts that must hold for the whole task live only in the conversation, where they drift towards the middle of the window or are lost when history is summarised.

What to do: keep durable facts in a short notes block outside the running history, and have the harness add it near the end of the window, just before the current step. Anthropic calls this structured note-taking: notes "persisted to memory outside of the context window" and pulled back in later. Across sessions, managed services extract facts for you. Google's Gemini Enterprise Agent Platform (Google describes it as an evolution of Vertex AI) offers Memory Bank. Amazon Bedrock offers AgentCore Memory, and Microsoft offers memory in Foundry Agent Service (preview). The context question is not where memory is stored but which memories enter this call.

When not to: memory written by a model can be wrong, stale or planted. Treat it as data to check, not as instruction (see the limits section).

The invoice agent: "credit, not refund; card closed" goes into the case notes the moment the customer says it.

Tool results

What goes wrong: in a long run, raw tool output becomes most of the window. In the simplified model below, a billing API returning 3,000 tokens per call makes spent tool output three quarters of the window by turn 40.

What to do: shape outputs at the source. Anthropic's guidance on writing tools for agents recommends returning meaningful fields, preferring names to opaque IDs, and "some combination of pagination, range selection, filtering, and/or truncation" with sensible defaults. It also suggests a response_format parameter so the agent can ask for concise or detailed output. Then clear what is spent: Anthropic's context editing (beta) replaces the oldest tool results with placeholder text once the window passes a threshold, keeping the most recent ones.

When not to: do not clear results the agent will need again, or it will simply fetch them again. Exclude those tools from clearing. Clearing also changes the start of the window, which interacts with prompt caching; the cost side of that is covered in our note on token optimization.

The invoice agent: the invoice tool returns the six fields the decision needs. Results older than the last three calls are cleared.

Conversation history

What goes wrong: history grows by every turn, and old turns compete with new ones for attention.

What to do: compact. Summarise older turns, keep the most recent ones word for word, and tell the summariser what it must preserve. Anthropic describes how Claude Code does it: the summary keeps "architectural decisions, unresolved bugs, and implementation details", drops redundant tool output, and the agent continues with the summary plus the five most recently accessed files.

When not to: short tasks do not need it, and every summary loses something. Compact before the window is full, not after.

The invoice agent: older turns are summarised; the customer's constraint survives because it is also in the case notes, and because the summarisation instructions name it.

Order: what goes where

Order changes results as well as choice, and four primary sources agree on the direction. Liu and colleagues found models use the beginning and end of the input best. Google's Gemini documentation says performance is better "if you put your query / question at the end of the prompt." Anthropic's prompting best practices put long documents near the top and the query at the end. Google's ADK write-up keeps "frequently reused segments stable at the front" and pushes dynamic content toward the end.

In a briefing pack the cover note sits on top. In a context window the equivalent place is the end, next to the question. So stable material goes first, reference material next, history after that, and the facts that must hold go last, beside the current step.

Open the diagram full size:

Long runs: clear, compact, take notes, split

When a run is long enough that the window approaches your budget, apply the lightest lever that fixes the problem. Anthropic calls tool-result clearing "one of the safest lightest touch forms of compaction." Summarising history is heavier, because it rewrites what the model sees. Handing a subtask to a sub-agent is heaviest, and also the most effective for reading-heavy work: the sub-agent explores in its own window and, in Anthropic's description, returns "only a condensed, distilled summary of its work (often 1,000-2,000 tokens)."

Open the diagram full size:

On the Claude API, tool-result clearing is a request setting under the context-management-2025-06-27 beta. This example fetches the case notes with a get_case_notes tool, which the agent calls just before each decision, so the newest copy lands next to the question; the tool is excluded from clearing so its results are never removed.

import anthropic
client = anthropic.Anthropic()
response = client.beta.messages.create(
    model="claude-model-id",  # placeholder: use your current Claude model ID
    max_tokens=2048,
    betas=["context-management-2025-06-27"],
    tools=billing_tools,  # your tool definitions
    messages=history,
    context_management={
        "edits": [{
            "type": "clear_tool_uses_20250919",
            "trigger": {"type": "input_tokens", "value": 60_000},
            "keep": {"type": "tool_uses", "value": 3},
            "exclude_tools": ["get_case_notes"],
        }]
    },
)

In Google's ADK for Python, compaction is one setting on the app, described in the ADK compaction guide: summarise older events once the session passes a token threshold, and keep the latest events as they are. ADK 2.13.0 marks the feature experimental.

from google.adk.agents import LlmAgent
from google.adk.apps.app import App, EventsCompactionConfig
dispute_agent = LlmAgent(
    name="invoice_dispute_agent",
    model="gemini-model-id",  # placeholder; current example: gemini-3.8-flash
    instruction="Resolve invoice disputes within the credit policy.",
    tools=[],  # your billing and CRM tools
)
app = App(
    name="disputes",
    root_agent=dispute_agent,
    events_compaction_config=EventsCompactionConfig(
        token_threshold=60_000,    # summarise older events past this size
        event_retention_size=6,    # keep the latest events word for word
    ),
)

The other platforms offer the same levers under different names.

The invoice agent, before and after

The invoice agent from the opening scene can be compared before and after context engineering in a deliberately simplified model, not a measurement: every input is an assumption, printed beside the result, and the arithmetic was run in code. A turn is one model call that ends in one tool call and its result.

• Both versions: 2,500 tokens of instructions; 300 tokens of conversation text per turn; the constraint stated in turn 3; the window inspected after turn 40.
• Naive: 12 tool definitions of 400 tokens each; 3 policy documents of 3,000 tokens loaded up front; 3,000 tokens per tool result, all kept.
• Engineered: 5 tool definitions loaded; one 1,500-token policy section fetched when needed; 800-token filtered tool results; results older than the last three cleared to a 10-token placeholder; a 400-token case-notes block placed just before the current step. History is not compacted here, so the model leaves that lever out.

In this model, the engineered window is about one seventh the size of the naive one, but the other two rows matter more. In the naive window of this model, three quarters of what the model rereads is spent tool output, and more than 125,000 tokens separate the deciding sentence from the question. In the engineered window of this model, a copy of that sentence sits a few hundred tokens from the question. None of this proves the agent decides correctly. It shows the second agent faces a much easier reading task.

What each recommendation rests on

Not all of the advice above has the same kind of evidence behind it. This table says which kind, so you can weigh it. No new experiments were run for this note.

How to know it worked

Context changes are easy to make and hard to judge by eye. Measure them.

• Plant your own needles. Build evaluation cases where a fact stated early decides a step late, and phrase the late question differently from the early fact, as NoLiMa does. Score whether the agent honours it at turns 10, 20 and 40.
• Log the window by source. Record tokens per call for each of the six sources in your traces. A rising share of old tool output is an early warning.
• Compare focused with full. Run the same cases with only the relevant material and with everything. Chroma's report found focused prompts scored higher than full ones across all the models it tested; check whether yours does.
• Re-run on every change. A new model, a new summarisation prompt or a new tool output shape can each move these numbers.

Honest limits

• Compaction is lossy. The summary decides what survives. Anthropic's on-demand compaction documentation notes that images, documents and fetched URLs inside summarised messages are gone once the summary replaces them. Write your own summarisation instructions that name what must be kept, and test them.
• Some compaction cannot be inspected. OpenAI describes its compaction item as encrypted, "opaque and not intended to be human-interpretable." Where you must show what the model was told, a readable summary you produce yourself is easier to audit, at the cost of more code.
• Memory and notes are attack surface. The OWASP Top 10 for Agentic Applications (2026) lists Memory and Context Poisoning (ASI06), and Microsoft's memory documentation warns of prompt injection and memory corruption. Anything written to notes or memory is data, never instruction. Our agent security work covers it.
• Just-in-time retrieval has costs. It trades a smaller window for more tool calls and more latency, and an agent can fail to fetch what it needs.
• The evidence is benchmark evidence. It sets the direction; it does not predict your agent.
• Several features are young. Anthropic's compaction and context editing are beta, Foundry memory is in preview, and ADK compaction is marked experimental. Names and defaults may change; pin versions and re-read the docs.

Takeaways

• The window is rebuilt for every model call. What the model reads is your decision, every time.
• More context is not more quality. Choose the smallest set that does the job.
• Shape tool output at the source; in long runs it is usually the largest part of the window.
• Put stable material first and the facts that must hold last, beside the question.
• Use the lightest lever first: clear, then compact, then split. Keep durable facts in notes either way.

Reread-me checklist

• Can you say how many tokens each of the six sources contributes at the last call of your longest run?
• Are fewer than about ten tools loaded up front, with the rest searchable?
• Do tools return only the fields a decision needs, and are old results cleared, with exclusions for those that must stay?
• Is there a case-notes block, and does it sit near the end of the window?
• Do your summarisation instructions name the facts that must survive?
• Do your evaluations include facts stated early and needed late, phrased differently?

What's next

Two questions follow: what all this rereading costs, and when the loop itself should stop, retry or ask a person. Each has a note of its own. For where people should approve an agent's actions, see human approval and autonomy tiers.

Take your agent's longest failed run, break the window at the failing call down by the six sources, and see which one was largest. If the answer surprises you, we would like to hear it: send us the breakdown. Tracing and evaluation are part of AELSHA™ ASTRA, the framework we use to deliver every agent engagement.

Further reading

• Effective context engineering for AI agents, Anthropic: The clearest single account of the discipline, from the attention budget to note-taking, compaction and sub-agents.
• Architecting efficient context-aware multi-agent framework for production, Google: How ADK treats the window as a compiled view over sessions, memory and artifacts, with the ordering rules spelled out.
• Lost in the Middle: How Language Models Use Long Contexts, Liu et al., Transactions of the ACL: The study behind the advice to keep decisive material at the start or the end of the window.
• Writing effective tools for agents — with agents, Anthropic: Practical guidance on shaping tool output at the source, which in long runs is usually the largest part of the window.
• How AI reads a large repository: map, reduce, write, AELSHA™ NEHEMVIX: Context engineering applied to code, with the real token counts of a pipeline that reads a whole repository.

Context features, beta flags and SDK defaults change often, so check each vendor's own documentation before you rely on them.