Harness Engineering: Why the Same Model Scores Differently

Take three AI models of comparable frontier capability, and give them the same 100 software-engineering tasks in three different setups. Harness engineering is the deliberate design, testing and versioning of everything an AI agent runs inside except the model: its tools, sandbox, permissions, state, and the hooks and guardrails that check each action. In a 2026 preprint study by Zhang and colleagues, Stop Comparing LLM Agents Without Disclosing the Harness, which kept each model fixed, changing the setup moved one model's pass rate by 13 points and the other two by 8.5 points each, while switching models within a setup moved it by 5 points at most. What changed was everything around the model: how its context was built, how its tools were described, whether failed steps were retried and whether its work was checked.

A caution on the evidence. The research on harnesses is early. That result comes from one small experiment (100 tasks, two runs for each model and setup) in a preprint position paper, and like all three studies cited in this note, it measures coding agents on coding benchmarks. Treat it as a signal of direction, and test it on your own agent before you plan around it.

The model sets what an agent could do. The harness decides what it actually does, what it may touch, and what happens when it is wrong. You rarely choose the weights; you always own the harness.

The model sets what an agent could do. The harness decides what it actually does, and you own the harness.

Our notes use one stack for agent systems: the harness is everything around the model inside one agent run (tools, sandbox, permissions, state and hooks) and hosts the inner agent loop; the graph orchestrates steps and runs; and loop engineering is the outer layer that starts repeated runs, proves "done" and stops them.

If you read only this:

• Harness engineering is the design of everything between the model and the task: tools, the execution environment, permissions, state, and the hooks and guardrails that check each action.
• It moves results about as much as model choice in the published comparisons, which are early, small and mostly about coding agents.
• Each control covers a path, not an outcome. A hook on the file-write tool does not see a shell command that writes the same file. Put the rules that must hold at a boundary every path crosses: the sandbox, the credentials, the network.
• Harness parts expire. Each one encodes an assumption about what the model cannot do. Re-measure on every model upgrade and remove what no longer earns its place.

Open the chart full size:

What harness engineering means

The word is used at two widths, sometimes by the same company. Anthropic calls its Claude Agent SDK, with its tools, permissions, sessions and hooks, a general-purpose agent harness. Its Managed Agents design uses the word narrowly: there the harness is "the loop that calls Claude and routes Claude's tool calls to the relevant infrastructure", and the session log and the sandbox are separate parts beside it. Amazon Bedrock AgentCore now ships a service named Harness, described as a managed agent loop.

Research papers use it broadly. One 2026 paper, Lin and colleagues' Agentic Harness Engineering study (more on it below), calls the harness the collection of "model-external, editable components" around an agent. Another, Lewis's Same Model, Different Harness, says the harness "decides what the model sees, which tools it can use, and how the work continues". OpenAI's February 2026 post Harness engineering: leveraging Codex in an agent-first world does not define the word, but its practice is the broad one. The engineers' job became to "design environments, specify intent, and build feedback loops", and when the agent failed they asked what capability was missing and how to make it "both legible and enforceable for the agent".

This note uses the broad meaning, because that is where the engineering decisions sit. Harness engineering is the deliberate design, testing and versioning of everything an agent runs inside, except the model. Five parts matter most in production:

1. Tools and their definitions
2. The execution environment and sandbox
3. Permissions
4. State and memory stores
5. Hooks and guardrails

Three neighbours are left out on purpose: the loop algorithm (plan, act, observe, stop) and the runs repeated around it, the orchestration graph that orders steps and runs, and the choice of what enters the context window each turn. The harness hosts the agent loop, but these three are separate design decisions, and each has a note of its own.

Open the diagram full size:

Harness versus model: who decides what

The model chooses. The harness decides what the choices are, where they run, whether they run, what survives them and whether they were right.

Two findings sharpen the split. In the Agentic Harness Engineering paper (Lin and colleagues, 2026), an automated process edited a coding agent's harness over ten rounds while the model stayed fixed. The coding agent's pass rate on the 89-task Terminal-Bench 2 rose from 69.7% to 77.0%. The ablations locate the gain in "tools, middleware, and long-term memory rather than the system prompt". Rewording instructions was the weakest lever. Changing what the agent can do and remember was the strongest.

The second finding is about time. Anthropic's engineers write in Harness design for long-running application development that "every component in a harness encodes an assumption about what the model can't do on its own". Their Managed Agents write-up gives an example: one model wrapped up tasks early as its context window filled, so they added context resets to the harness. With a later model the behaviour was gone, and "the resets had become dead weight". A part that once helped can later cost tokens, latency and new failure modes for nothing.

The five parts, on one running example

Picture a hypothetical agent that we will follow through every section. It receives a failed continuous-integration (CI) run on a repository, finds the cause, fixes the code and opens a pull request for a person to review.

One failure to design against is making the build green by editing the test instead of the code. The work Anthropic describes in Effective harnesses for long-running agents met a close cousin of this problem and handled it in the prompt: its agents were told, in strong words, that removing or editing the tests in their feature list was unacceptable. Here the harness will make editing the tests impossible, not merely unacceptable.

Each part below follows the same shape: what it fixes, how, when not to, and what it does for this agent.

Tools and their definitions

What it fixes. An agent acts only through its tools, and it chooses among them by reading their names and descriptions. A vague or overlapping set produces wrong calls that no prompt fully corrects.

How. Build a few tools shaped around the job, not one per API endpoint. Anthropic's guide to writing effective tools for agents says "More tools don't always lead to better outcomes". It recommends clear namespaces, unambiguous parameter names, responses that carry meaning instead of raw IDs, and error messages that tell the model what to do next. In the Model Context Protocol (MCP) tools specification, a tool definition is a name, a description and a JSON Schema for its inputs. The specification adds that clients "MUST consider tool annotations to be untrusted unless they come from trusted servers". A tool's claim to be read-only is a label, not a control.

When not to. Do not hand-craft tools for one-off exploration. A general tool such as a shell is acceptable inside a disposable sandbox with nothing to lose. The failure comes from a general tool in a sensitive environment.

For our agent. Four tools: read_file, write_file, run_tests (which runs the project's fixed test command, not an arbitrary one) and open_pull_request. There is no general shell and no merge tool. That narrowness matters below: when there is one way to write, one check can see every write.

The execution environment and sandbox

What it fixes. Agent-written code and commands run somewhere. If that place holds credentials or can reach production, one prompt injection becomes an incident. Anthropic describes this in its first Managed Agents design: generated code ran in the same container as credentials, so an injection only had to get the model to read its own environment.

How. Run each task in an isolated, disposable environment, and keep secrets out of it. Anthropic's fix keeps tokens unreachable from the sandbox. The repository token is used at setup to clone, and tool credentials sit in a vault behind a proxy, so "The harness is never made aware of any credentials." OpenAI's Codex runs locally in one of three sandbox modes, read-only, workspace-write and danger-full-access, enforced by the operating system (Seatbelt on macOS, bubblewrap on Linux, a native sandbox on Windows). The managed platforms sell the same idea as a service, listed in the platform table below.

When not to. An agent that only answers questions from a fixed index needs controls on its data access, not a code sandbox.

For our agent. A fresh container for each CI failure, with the repository checked out and tests/ mounted read-only. The network reaches only the package mirror. There are no deploy keys and no cloud credentials, and the container is destroyed once the pull request is open.

Permissions

What it fixes. At some point the model will propose an action it should not take, because it misread the task or because injected text told it to. Permissions decide whether that proposal runs.

How. Deny by default, allow narrowly, and ask a person about actions that are irreversible or reach outside. According to its permissions guide, the Claude Agent SDK checks every tool request in a fixed order: hooks, deny rules, ask rules, the permission mode, allow rules, then a runtime callback. A deny rule holds even in its bypassPermissions mode, but an allow list does not limit that mode, which approves nearly everything that no deny rule, ask rule or hook stops. Codex pairs its sandbox modes with approval policies; on-request asks when the agent needs to go beyond the sandbox. OWASP's Top 10 for Agentic Applications introduces the principle as least agency. Which actions need a person, and how to make that approval a real control, is covered in our note on AI agent autonomy tiers.

When not to. Approval prompts on every read train people to click yes. Ask about what matters, and deny the rest outright.

For our agent. Writes are allowed under src/ only. tests/, .github/workflows/ and .env are read-only. open_pull_request is allowed, and because no merge tool exists, merging needs no rule. The agent's identity can open pull requests on this one repository and nothing else.

State and memory stores

What it fixes. Context windows end, processes crash and long tasks span sessions. Without durable state each new session starts blind, repeats work or declares the job done too early. Anthropic's long-running harness work names both failures: running out of context mid-change, and a later session seeing progress and calling the job finished.

How. Keep state outside the model, in plain stores a person can inspect. Anthropic's harness uses a progress file, a JSON feature list with every feature first marked failing, a script that starts the app, and git commits. It notes the model was "less likely to inappropriately change or overwrite JSON files compared to Markdown files". OpenAI's Codex team made the repository the system of record: a short AGENTS.md of roughly 100 lines serves as "the table of contents" to versioned plans and design documents, because "anything it can't access in-context while running effectively doesn't exist".

When not to. Memory that persists across users or tasks is also where a poisoned instruction can persist; OWASP lists memory and context poisoning as ASI06. Store what the task needs and let the rest expire.

For our agent. The session log, a short notes file in the working branch recording what was tried, and one git commit per attempt so a failed fix can be reverted. When the run ends, only the pull request remains. What goes into the window each turn is context engineering; this part is only about where state lives and how it survives.

Hooks and guardrails

What it fixes. A rule written in a prompt is a request. A rule enforced in code is a fact. Hooks are code the harness runs at fixed points, such as before and after each tool call. Guardrails are checks, often classifiers, on inputs, tool calls, tool results and outputs.

How. Put each rule that must hold in code that runs whether the model cooperates or not, and make its message useful to the model. OpenAI's Codex team enforced its architecture with custom linters and wrote "the error messages to inject remediation instructions into agent context". In Google's Agent Development Kit (ADK), a before_tool_callback runs before every tool call, as the ADK callbacks documentation describes. If it returns a value, ADK skips the tool and hands that value to the model as the tool's result.

import os
from google.adk.agents import Agent

REPO = os.path.realpath("/workspace/repo")  # the checked-out repository
READ_ONLY = [os.path.join(REPO, p) for p in ("tests", ".github/workflows", ".env")]

def guard_writes(tool, args, tool_context):
    # Runs before every tool call. Returning a dict skips the tool
    # and hands that dict back to the model as the tool's result.
    if tool.name != "write_file":
        return None
    # Resolve "..", absolute paths and symlinks before comparing.
    path = os.path.realpath(os.path.join(REPO, args.get("path", "")))
    if os.path.commonpath([REPO, path]) != REPO or any(
            os.path.commonpath([ro, path]) == ro for ro in READ_ONLY):
        return {"error": f"Policy: {args.get('path')} is read-only for this agent."}
    return None  # let the call run

agent = Agent(
    name="ci_fixer",
    model="gemini-model-id",  # placeholder; current example: gemini-3.8-flash
    instruction="Find why CI failed, fix the code, and open a pull request.",
    tools=[read_file, write_file, run_tests, open_pull_request],  # your functions
    before_tool_callback=guard_writes,
)

When not to. A hook that calls a second model on every step adds latency and a new way to fail. Use deterministic checks for rules you can state precisely, and keep classifiers for what you cannot.

For our agent. The callback turns "do not edit the tests" from an instruction into a fact. After each write, an after-tool callback runs the linter and returns its findings with the result. CI logs are untrusted input, since they can carry text written by whoever opened the failing pull request, so the harness passes them to the model as quoted data.

Open the diagram full size:

Where each control stops

Every control covers the calls it is wired to, not the outcome you care about. The vendors say so in their own documentation:

These are not weak products. Each documents its boundary plainly, and Microsoft's documentation says it outright: "Content safety guardrails don't determine whether an agent is authorized to call a tool or perform an action."

The lesson is where to put the rules that must hold. Put them at a boundary every path crosses: the sandbox's filesystem and network rules, the credentials the agent can reach, and the scopes on its identity. Layer hooks and guardrails on top for early blocks and useful error messages, not as the only wall. In our example the ADK callback sees every write only because write_file is the only way to write and the callback resolves each path before checking it. Add a general shell tool and the callback becomes partial. The read-only mount on tests/ does not.

The same harness on five platforms

Google Cloud's column is Gemini Enterprise Agent Platform, which Google's Agent Platform overview describes as an evolution of Vertex AI. Names follow each vendor's own documentation.

Each platform offers something in each row, though the mechanisms differ: Foundry's guardrails are classifiers, for example, while ADK callbacks and Claude hooks are your own code. None of them makes the five decisions for you.

Treat the harness as a versioned artifact

• Version it with the code. Prompts, tool definitions, hook code, permission rules and sandbox images live in one repository and are reviewed like code.
• Evaluate the pair. Zhang and colleagues argue that an agent's score means little without the harness that produced it, and propose disclosing seven layers: execution, tool, context, scheduling, observability, verification and governance. Record the harness version beside the model version in every trace.
• Strip on every model upgrade. Re-run your evaluation set with each harness part switched off in turn, and keep only what still earns its cost. Anthropic notes it became "difficult to tell which pieces of the harness design were actually load-bearing", which is the reason to test them one at a time.
• Change one thing at a time, with a prediction. The Agentic Harness Engineering process attaches a predicted effect to every harness edit and checks it against the next round's results. The same discipline works by hand.

Honest limits

• The evidence is young. The studies cited here are preprints, not peer-reviewed, all on coding-agent benchmarks, with small samples. Zhang's experiment used 100 tasks, two runs per cell and three models, and the authors write that they do not claim its headline ratio of harness variance to model variance is universal. At that size, gaps of 2.5 to 5 points between models are within noise. That reading comes from a deliberately simplified model: treating each cell's 200 attempts (100 tasks, two runs) as independent gives a standard error of about 5 points for the difference between two cells, which understates the true noise. The harness gaps were larger, but their size will not transfer to your workload.
• More harness is not free. Anthropic reports that its full planner, generator and evaluator harness ran for hours and cost many times more than a single-agent run, for visibly better output. Every part you add costs tokens and latency, and is worth it only if your evaluation says so.
• A harness can hide the model. Retries and checks can absorb a model regression until the checks become the bottleneck. Trace the failures the harness caught, not only the final results.
• Classifiers are not authorisation. A content guardrail can block harmful text. It cannot decide whether this agent may change this record.
• Harnesses do not port cleanly. Evaluation order, hook coverage and guardrail scope differ by platform. A harness designed around one SDK's rules needs re-testing on another.

Takeaways

• The harness is everything except the model: tools, sandbox, permissions, state, and hooks and guardrails.
• It moves results about as much as model choice in early, small studies. Evaluate the model and harness as one system.
• Tools, middleware and memory beat prompt wording as levers in one published ablation.
• Enforce must-hold rules at a boundary every path crosses, then add hooks and guardrails for speed and clarity.
• Every harness part is an assumption that expires. Re-measure on each model upgrade.

Reread-me checklist for your agent's harness:

• Every tool is listed with its description, and no two tools overlap.
• Agent code runs in an isolated environment that holds no credentials.
• Deny is the default, and the rules that must hold are enforced at the sandbox, credential or identity layer.
• State lives in stores a person can read, and nothing persists that the task does not need.
• Each hook and guardrail lists the paths it does not see.
• The harness is versioned, recorded in traces and re-tested on every model change.

What comes next

This note covered what surrounds the model inside one agent run. The outer loop that starts and stops runs, the graph that orchestrates steps and runs, and the context the model sees each turn are separate design decisions, and each has a note of its own, linked above.

One request. Write your agent's harness on a single page: every tool, where it runs, what it may do without asking, where state lives and which check guards which path. Then look for one action that reaches the outside world by a path no control sees. If you find one, tell us what kind of path it was; those are the cases worth writing up. If you would like a second pair of eyes on that page, our agent security work starts from the same map, and an AI agent readiness assessment is the place to start if the agent is not built yet.

Further reading

• Harness engineering: leveraging Codex in an agent-first world, OpenAI: OpenAI's account of the practice, with the repository as system of record and linters that teach the agent.
• Effective harnesses for long-running agents, Anthropic: A concrete account of state that survives sessions: progress files, a JSON feature list and git.
• Harness design for long-running application development, Anthropic: Why every harness part encodes an assumption about the model, and how to tell which parts still carry load.
• Stop Comparing LLM Agents Without Disclosing the Harness, Zhang et al., arXiv: The experiment behind the opening, plus a seven-layer checklist for reporting the harness beside any score.
• What one agent's harness allows, AELSHA™ NEHEMVIX: One agent's guardrails: what it may do and what it may not.

Agent SDKs, managed platforms and their guardrail features change quickly, so check each vendor's current documentation before you build on them.