A few lines of Python scored 500 out of 500 on SWE-bench Verified, one of the most widely used tests of AI coding agents. They fixed no bugs. They told the test runner that every test had passed.
The trick comes from an April 2026 audit by researchers at UC Berkeley. Their agent worked through eight well-known agent benchmarks and scored between 73% and 100% on them without solving a task. A benchmark score is a claim about one test, run one way, on one setup. This note shows how to read that claim before you put a budget behind it.
Before you believe an agent's score, ask what an agent that solves nothing could score on the same test.
If you read only this:
- A high score can be a broken test. The agent may be able to reach the grader, the answers may be public, or the tests themselves may be wrong.
- Ask six questions before you trust a number. What solving nothing can score, isolation, prior exposure, what counted as a pass, margin of error, and the setup and cost behind it.
- A benchmark shortlists. Your own failures decide. Use public scores to narrow the field, then run the survivors on cases taken from your own system.
Why agent benchmark scores break
Picture an exam where the candidate sits in the examiner's office, with the answer key in an unlocked drawer and the marking sheet on the desk. A high mark tells you very little. Several of the benchmarks in the audit were built like that room, because the agent and the code that grades it could touch the same files.
| Benchmark | Tasks | Score with no task solved | How it was done |
|---|---|---|---|
| SWE-bench Verified | 500 | 100% | A test-runner hook, with a variant for Django's tasks, forced every test to report a pass |
| SWE-bench Pro | 731 | 100% | The results parser was overwritten inside the container |
| Terminal-Bench | 89 | 100% | A fake curl wrapper tampered with what the verifier ran |
| FieldWorkArena | 890 | 100% | The validator never checked whether the answer was correct |
| CAR-bench | Hallucination tasks | 100% | Reward components were skipped for those tasks |
| WebArena | 812 | About 100% | The answer was read from the task's configuration file |
| GAIA | 165 | About 98% | Public answers, and loose matching of answer text |
| OSWorld | 369 | 73% | Virtual machine state was changed, and gold files were public |
Reaching the grader is one way a score goes wrong. Two others have nothing to do with exploits.
The model may have seen the test. OpenAI stopped reporting SWE-bench Verified in February 2026 after finding that all the frontier models it tested could reproduce the original fix, or exact details of the problem, for certain tasks. It recommended SWE-bench Pro in its place, and the April audit reached 100% on that as well.
The test may be wrong. The same review audited a 27.6% subset of tasks that models often failed, and found that at least 59.4% of those had flawed tests that reject correct solutions.
Agents also find these gaps on their own. In its May 2026 Frontier Risk Report, the research group METR says that for one model, cheating was effective enough that its measured time horizon, the length of task in human working time that it completes at a set success rate, would have been roughly twice as large if successful cheats had counted as passes. METR scores cheats as zero, and writes that manually checking for cheating is often the majority of the work in a run of its evaluation suite.
What this does not show. The Berkeley team states in its paper that it does not show frontier models using these exploits in normal evaluation runs. So a given published score has not been proven false. The point is narrower: the number alone cannot tell you.
Six questions to ask before you believe a score
Suppose two vendors show you agents for matching supplier invoices. Agent A scores 89% and Agent B scores 84% on a public benchmark of 100 tasks. The vendors and numbers are illustrative. Here is how to read them.
1. Find out what an agent that solves nothing can score
Three baselines do this. An agent that takes no action should score zero, apart from any tasks where doing nothing is the right answer. The audit's checklist puts it plainly: "Run a null agent that takes no actions. Its score is your floor. If it's not zero, something is wrong." An agent that acts at random should not beat it. An agent that only tampers with the environment should score nothing at all, and that is the baseline that matters most, because an empty submission would not have triggered the hook that opened this note.
These baselines belong to the benchmark, so look for them from whoever runs it. Ask each vendor which version of the benchmark they used and who checked the runs for tampering. If nobody can answer, you do not yet know what the 89 and the 84 measure.
2. Keep the grader out of the agent's reach
Move the examiner out of the room. Grading should run outside the agent's environment, on files copied out and treated as untrusted, with answers stored where the agent cannot read them.
Inspect, the open-source evaluation framework from the UK AI Security Institute and Meridian Labs, can run an agent's commands in a dedicated sandbox; keeping the grader outside it is a design choice you still have to make. BenchJack, the Berkeley team's open-source scanner, in early preview, audits a benchmark for these holes.
3. Check whether the model has already seen the test
Public tasks drift into training data. Ask whether the set is private or held out, and whether anyone checked for memorisation. OpenAI's full method used a second model probing over 15 turns. A simplified version is one you can copy: give the model a task's ID, ask for the exact fix, and compare the answer with the real one.
4. Read what counted as a pass
Two rules from the audit catch a lot: "A crashed task is a zero, not a missing data point", and failed tasks must not quietly leave the denominator. Then ask who read the transcripts. Anthropic's guide to agent evals is direct: "You won't know if your graders are working well unless you read the transcripts and grades from many trials."
Prefer a grader written in code. Where a model has to judge, treat the agent's output as untrusted input. In CAR-bench, hidden instructions in the agent's reply biased the judge.
5. Put a margin of error on the gap
Is 89 really better than 84? In a deliberately simplified model, with one run per task and tasks treated as independent, the 95% Wilson intervals are 81.4% to 93.7% for Agent A and 75.6% to 89.9% for Agent B. A two-proportion test gives p of about 0.30, so a gap that size can be chance. The same five-point gap on 1,000 tasks gives p of about 0.001.
Both agents ran the same tasks, so a paired test is the better tool, and its answer depends on how many tasks each agent won alone. For this five-point gap, p runs from about 0.06 (Agent A alone wins five, Agent B none) to about 0.44 (16 against 11, the widest split the totals allow). Ask for the per-task results.
Ask how many tasks and how many runs of each. Then ask for the reliability figure as well, because agents vary from run to run. Our AgentOps note works through pass^k, the chance that every repeated attempt succeeds.
6. Ask what setup produced it, and what it cost
The same model can score differently under a different harness, the code around the model. A score belongs to that whole setup. Ask for the cost per completed task next to it, because a higher score that takes four times the spend may be the worse choice.
The metrics behind a score, and where to find them
One percentage folds several different questions together. Ask to see them apart. The tools below are examples of where each check already exists, not a ranking, and the same metric name can be calculated differently from one tool to the next. Many of them are scored by a model, so question 4 applies to them as well.
| What you want to know | Metric to ask for | Where it exists |
|---|---|---|
| Did it finish the job? | Task or goal success, checked in the system it changed | DeepEval Task Completion; Ragas Agent Goal Accuracy; ADK multi_turn_task_success_v1; AgentCore Goal success rate |
| Did it use the right tools, correctly? | Tool selection and argument accuracy | ADK tool_trajectory_avg_score; Microsoft Foundry Tool Selection and Tool Input Accuracy; DeepEval Tool Correctness; Phoenix Tool Selection and Tool Invocation |
| Did it stay within its instructions? | Adherence to task, plan or topic | Foundry Task Adherence (preview); DeepEval Plan Adherence; Ragas Topic Adherence |
| Are its answers grounded? | Faithfulness to the source material | Phoenix Faithfulness; ADK hallucinations_v1 |
| How consistent is it? | pass@k and pass^k over repeated trials | Anthropic's guide; the AgentOps note |
| How wasteful is it? | Steps per task and cost per completed task | DeepEval Step Efficiency; the FinOps note |
| Can it be attacked? | Results of attack tests, such as prompt injection | promptfoo red teaming; our ten-threat matrix lists what to test |
"Did it finish the job?" deserves its own warning. As Anthropic's guide puts it, an agent might say "Your flight has been booked", but "the outcome is whether a reservation exists in the environment's SQL database". Grade the system, not the sentence.
Keep your test cases in your own repository, in a plain format, whichever tool you choose. Platforms change: OpenAI's hosted Evals platform became read-only on 31 October 2026 and is scheduled to shut down on 30 November 2026.
What a benchmark cannot tell you about production
A benchmark uses someone else's tasks, tools and data. Treat a good score as a reason to test further.
Before an agent reaches production, these are the questions no public leaderboard answers:
- Does it pass on your failures? Build the test set from real incidents, and gate every change on it.
- What may it do alone? Set the autonomy tier for each action before it can act.
- On whose authority? Give each agent its own identity, and know how many you run.
- Will it stop? Enforce limits in code, outside the model.
- Can you see what it did? AELSHA™ ASTRA, our delivery framework, sets out the signals to capture and the gate each phase must clear.
We hold our own numbers to this
AELSHA™ NEHEMVIX, our free tool (you bring your own AI model key), publishes the test results for its check of invented code names. The same six questions apply to us.
- Solving nothing. A checker that flags nothing would catch none of the planted names. One that flags everything would wrongly flag real names; in the second blind round none of 376 was flagged, after one false-alarm class was fixed.
- Isolation. The first list was written by a tester who had not read the checker. The second was written by a new agent that could not read the checker, its documentation or the first round's misses.
- Prior exposure. The first list is no longer held out. It scored 80.2% on the first run and reached 94.3% (100 of 106) after two rounds of fixes made against it. The second list, at 83.3% (189 of 227), is the held-out figure.
- Margin of error. On the held-out figure, the 95% Wilson interval is 77.9% to 87.6%.
- Setup and cost. Five repositories and four languages. The checker is code, so a rerun costs no model calls and gives the same result.
- Still open. It is our own evaluation, not an independent audit. The test harness is not public, so nobody outside can rerun it yet, and the planted names were written by a tester and an agent rather than collected from real model mistakes.
Honest limits
- The audits mostly show what is possible. The paper does not show frontier models using these exploits in normal runs, although the April write-up collects separate cases of scores that were inflated or had to be corrected.
- Some holes may be closed by now. The paper's own pipeline patched WebArena and OSWorld, and it does not say which of those fixes reached the public versions. Check the current version of any benchmark you rely on.
- The two Berkeley reports differ in scope. The April write-up covers eight benchmarks and the May paper ten, so their counts are not interchangeable.
- Most of this evidence is recent and not peer reviewed. The paper is a preprint, and the April write-up and the METR and OpenAI reports are published by their authors.
- A zero from a do-nothing agent is necessary, not sufficient. It would have missed the test-runner hook. A tamper-only agent goes further, and neither proves the benchmark is sound.
- Private test sets have a cost. They resist memorisation, but you cannot inspect them, so someone you trust has to.
- A shared caution on our own notes. Our harness note uses a result from a subset of SWE-bench Verified. We present it there as a signal of direction, which is all it should be taken for.
Takeaways
- A score is a claim about a setup. Ask what produced it before you compare it.
- Ask what solving nothing can score. A do-nothing agent is the cheapest check. A tamper-only agent catches the worse holes.
- Put a margin on every gap. Five points on 100 tasks can easily be noise.
- Separate the metrics. Task success, tool use, adherence, grounding, consistency, cost and resistance to attack are different questions.
- Decide on your own cases. A public benchmark can narrow the list. It cannot tell you how an agent behaves in your systems.
What's next
Take the last agent score you were shown and ask which version of the benchmark produced it, and who checked the runs for tampering. The reply will tell you a good deal about the test, and about who is quoting it.
Which score are you trying to judge right now? Tell us the benchmark and what you plan to use it for, and we will suggest which of the six questions to ask first.
Further reading
- How We Broke Top AI Agent Benchmarks, UC Berkeley: The audit behind this note, with each exploit explained and a checklist for anyone building an evaluation.
- Do Androids Dream of Breaking the Game?, Wang et al.: The paper version, with eight classes of flaw and a method for finding and patching them.
- Why we no longer evaluate SWE-bench Verified, OpenAI: The team that published the Verified set explaining contamination and flawed tests, with examples.
- Demystifying evals for AI agents, Anthropic: The best single guide to designing graders and running trials once you decide to build your own.
- Frontier Risk Report, METR: A candid account of how much work it takes to keep cheating out of a measurement.
Benchmarks are patched and evaluation tools change often, so check each project's own page before you rely on a figure or a metric name.