What It Takes to Trust Agent Output
In one vendor survey, most respondents said they trust what their coding agents produce, and four in ten hit problems with those agents daily or more. Both numbers came from the same people. What would justify that trust is several independent checks, each assumed to leak, and a named person accountable for what gets through.
Two earlier pieces on this site looked at checking. The verification gap asked whether developers check AI-generated code before they commit it. Checking Isn’t Catching asked whether the checking finds anything. This one asks what is underneath both: when is it reasonable to trust what an agent hands back, and what would have to be true for that trust to be earned rather than assumed?
My answer is that trust belongs to the checks around the tool, not to the people using it, and most teams have fewer of those checks than they think.
What people say they trust tells you very little.
Temporal’s State of Development 2026 surveyed 554 respondents in the US and UK/EMEA between 29 April and 25 May. Asked how much they trusted the accuracy of AI agent output, 85.5% said at least somewhat and 24.7% said completely. The same report found 41.1% hit issues with agents daily or more. Temporal sells an orchestrator for AI agents, so the report has a point of view, and each team decided for itself what counted as an issue. Still, the two figures sit side by side, and the first does not appear to constrain the second.
Stack Overflow’s 2025 survey asked a broader group a similar question. Of 33,244 respondents, 3.1% highly trusted the accuracy of AI tools and 29.6% somewhat trusted them. Of the same respondents, 46% distrusted them. Different populations and question wording gave very different pictures.
Two further studies show why the number moves. Shah and colleagues followed 71 upper-division computer science students using GitHub Copilot. After an hour, 49.3% said their trust had risen and 16.9% said it had fallen. After a ten-day project on a 23,500-line code base, 39.4% said it had risen and 36.6% said it had fallen, and the most common reason was that the tool needed a competent programmer to finish or fix the work. This was one course, one term, and self-reported attitude, not behaviour. The authors say they cannot extrapolate beyond ten days. It still suggests that trust formed on a demo is not trust formed on the job.
Brown and colleagues at Google looked at behaviour instead. They analysed a million multi-line completions shown to 59,000 developers and used acceptance as a stand-in for trust. Acceptance rose with the model’s own score for how likely a suggestion was to be accepted and with the developer’s expertise in the language, and fell in test files. Even at the highest score only about half of suggestions were accepted. The study never measured whether accepted code was correct, its authors call the results correlational, and the tool was a completion engine, not an agent. What it shows is that acceptance follows signals other than correctness, and that nobody checked correctness.
METR’s 2025 trial shows what felt confidence can be worth. Sixteen experienced open-source developers completed 246 tasks, some with AI allowed and some without. Before starting, they forecast that AI would cut their completion time by 24%. Afterwards they estimated it had cut it by 20%. The measured result was that tasks took 19% longer. It was a small study of early-2025 tools, and METR’s 2026 follow-up points the other way, though METR calls that newer data only very weak evidence. What holds up is METR’s own remark that developers’ self-reports of speedup can be quite unreliable.
A green test run is weaker evidence than it looks.
Tests are the obvious first check. Two studies of SWE-bench Verified, a benchmark of real issues from open-source Python projects, put a number on how far that holds.
Wang, Pradel and Liu (ICSE 2026) took 877 patches from three agent systems that had passed the benchmark’s own tests and ran them against each project’s full developer test suite. 7.8% failed. Differential testing, which compares each patch’s behaviour with the human fix, flagged 260 of the 877 as behaving differently, and when the authors read a sample of 77 of those, 22 were certainly incorrect. A separate team, in UTBoost (ACL 2025), generated extra tests and found that the extra tests flagged 92 of 584 passing Verified patches, 15.7%, as erroneous. Eleven of the 45 Verified leaderboard entries changed rank.
Both studies have limits. These are benchmark tasks, not production systems, and most passing patches held up under the stricter tests. The extra tests were written by a model and reviewed by two authors, so they can be wrong too. The lesson is a narrow one. A suite written before the agent arrived covers the failures its authors thought of, and a green run has a false-pass rate whether or not anyone has measured it.
Review is the other check, and the record of it is sparse.
The MSR 2026 paper by Cynthia, Das and Roy studied 9,427 closed pull requests written by coding agents (Claude Code, Cursor, Copilot and Codex) across 1,391 GitHub repositories with at least 100 stars, with data up to August 2025. Of the 6,819 that were merged, the authors report that on average 74.1% went in without any modification. Only 1,501 of the 6,819 had any review comments at all. Developers outside a project’s core group merged without running any checks 19.1% of the time, against 11.2% for core developers. A TU Delft workshop paper that mined 212,297 agent pull requests from 12 agents reports that 73.9% were merged without modification, with a median human triage time of 0.2 hours. Only its abstract is available, and it does not say what that percentage is a share of.
None of that proves anyone was careless. An unmodified merge can follow a proper read, maintainers may be merging work they prompted themselves, and neither paper measured defects after the merge. What the figures do show is the shape of the process: a lot of agent code goes in quickly, with little discussion around it.
Automation research would not be surprised. Parasuraman and Manzey’s 2010 review in Human Factors reports that complacency appears in novices and experts alike and is not removed by simple practice, and that automation bias is not prevented by training or instructions. Goddard, Roudsari and Wyatt’s 2012 review of clinical decision support found that workload, task complexity and time pressure made wrong advice more likely to be followed. Neither looked at code agents, and I would not carry their numbers across. The direction transfers well enough: an instruction to review carefully is a weak control, and heavy workload and time pressure are conditions a team with agents can easily be in.
There is now a paper arguing the logical end of this. Martin Monperrus’s June 2026 preprint says human review of agent-written code tends to become a rubber stamp and should give way to independent agent review for routine changes. It is a position paper with no new data, and the rubber-stamp premise rests on a citation that predates agent-written code. Its own section on counter-arguments raises silent approvals by the reviewing model, correlated blind spots between generator and reviewer, and prompt injection of the reviewer, for which it says there are no fully solved defences. It keeps humans for high-risk and regulated changes. I read it as a plausible description of the failure, asserted more than shown, and an untested proposal for the fix.
What the agent says about its own work is not evidence.
The MSR paper includes one case worth holding on to. On a Microsoft Fluent UI pull request, Copilot wrote that it had resolved the issue and that “all tests are now passing”. CI was still failing. The developer merged the pull request with two integration checks failing. It is a single example, and the paper does not count how often it happens. It does show that a status report from the thing being judged is output of the same process, and carries no more weight than the code does.
OWASP’s announcement of its Top 10 for Agentic Applications in December 2025 names a version of this as a risk in its own right. ASI09, Human-Agent Trust Exploitation, is summarised as “confident, polished explanations” misleading operators into approving harmful actions. In my experience a fluent account of what was done is cheap to produce and easy to mistake for assurance.
One 2026 preprint, Assay, proposes a mechanical version of this discipline: bind each claim an agent makes, such as “tests pass”, to a hash of the code it depends on, so the claim counts as stale the moment that code changes. Its evidence is scripted simulation on five open-source repositories, with no real agents or people, and I would treat it only as a plausible design idea. It is a useful one, because it states the principle plainly. A claim is worth something only for the code state it was made against.
Some claims can be checked with no judgement at all. Spracklen and colleagues generated 576,000 code samples from 16 models and found that the average rate of recommended packages that did not exist was at least 5.2% for commercial models and 21.7% for open-source ones. Of the invented names, 43% came back in all ten repeated runs of the same prompt. The models were 2023-era, and the authors did not register any invented names, so the paper shows nothing about exploitation in the wild. A registry lookup says the package exists, not that it is the right one. But it is mechanical and outside the model, and it costs almost nothing, which is the shape of a check worth having.
Every other check leaks, and they leak in different places.
Most of what an agent produces has no yes or no answer a lookup can give. The usual next steps are scanners, linters, sandboxes and a second model. The evidence for each is thin, but one study shows why it matters that they are independent.
Mahmud and colleagues (2026 preprint) ran 8,918 C++ programs, from three open-weight models and from pre-2021 human contest solutions, through static analysis, sanitizers and a bounded model checker. The model checker flagged 661 programs, and 598 of those were never triggered by the sanitizers. The sanitizers flagged 639, and 576 of those were missed by the model checker. Only 63 programs were flagged by both. Static analysis did not separate the two groups, and once length was controlled it flagged the human code more often. Sanitizers did separate them: 9.3% of AI programs against 2.6% of human ones. Among programs that passed every test, though, humans triggered sanitizers more often (3.6% against 1.8%).
These are short contest programs from open-weight models with one zero-shot prompt, and the authors say the results may not carry to production code. The model checker could not parse more than half of the programs, and sanitizers only see paths that a test exercises. The pattern is still the useful part. Any one check would have given a different answer from the next, and a clean result from one says little about what another would have found.
The single-tool numbers are patchy, and are not comparable with each other because the units differ. Rabbi and colleagues (EASE 2026) found 675 security-related pull requests among 33,596 written by agents and ran Semgrep over each. 104 of them, 15.4%, introduced at least one new alert. 354 of the 675, flawed or not, were merged (52.4%). A separate, non-peer-reviewed test of 522 snippets from six models counted about 153 true positives against 772 false positives from default-configuration static analysis. GitGuardian, which sells secret scanning, reports that 3.2% of Claude Code-assisted commits leaked a secret against a 1.5% baseline, and adds that the gap “should not be read as a simple tool failure”.
Loops make it worse in at least one setting. Shukla and colleagues had GPT-4o repeatedly “improve” ten C and Java programs with no human in between, and found vulnerabilities rising across iterations. It was one model, two languages and a simulated loop that the authors say developers rarely run. Their recommendation is the obvious one: human review between iterations, with a cap on consecutive model-only passes.
Guardrails are the least measured. Anthropic’s documentation for Claude Code’s sandbox is candid that it covers shell commands only and “is not a complete isolation boundary”. That is a design statement, not a result. A second model as reviewer is in the same place. A 2026 preprint (I could read only its abstract) using planted errors reports that a top-tier cross-model reviewer was “not significantly different in F1 from same-model review”, though the two found partly different errors, and earlier ICML work, on benchmark questions and resume screening rather than code, found that a judge inflates the accuracy of models weaker than itself, more so within a provider. A different model should not be treated as independent until someone has checked.
What I would do with this.
Put together, the evidence supports a position. No single check I found makes agent output safe to merge unseen, and that includes the human one. Trust that is earned is the product of several checks that do not share failure modes, so that what one misses another has a chance of catching.
In practice that means a few things. A test suite whose false passes have been looked for, not assumed away. A required status check the agent cannot route around, and a mechanical check wherever the claim allows one. At least one scanner whose alerts someone reads, and a second reader who did not write the prompt. The agent’s own account of its work kept out of the evidence. Review staffed to the volume being generated, because the automation-bias research suggests attention suffers under load, and a cap on how many unreviewed iterations an agent may run.
The last part is not technical. The 2026 AICD and Human Technology Institute guide for company directors says directors “should not rely on AI-generated summaries or analysis as a substitute for their own review and interrogation of board papers”, and that boards remain ultimately accountable for AI outcomes. It quotes the judge in ASIC v Bekier saying that AI assistance is not inherently objectionable, but must not lead to a “quiet normalisation of private reliance” on computer-generated distillations “unregulated by any agreed policy”. I have not read the judgment, and the case itself turned on information not reaching the board, not on AI. A board is not an engineering team, but the point carries. Someone has to decide which changes are low-stakes enough to wave through, write that down, and own the result. That is a leadership decision, and it is easy to leave unmade.