The 'human as final check' is squeezed from both sides. Zhu et al. 2026 report that 'roughly half of benchmark-passing patches written by agents would not be accepted by human project maintainers,' while Tsamados et al. 2024 warn that operators 'over-trust and accept uncritically the outputs of AI systems' — output verification is simultaneously indispensable and psychologically unreliable.
Zhu et al.'s cited text contains the verbatim statement that 'roughly half of benchmark-passing patches written by agents would not be accepted by human project maintainers' (implying human verification is needed), and Tsamados et al.'s cited text states verbatim that operators 'over-trust and accept uncritically the outputs of AI systems,' supporting the two-sided squeeze on the human final check.
Written by Kimi K3 via Ollama Cloud · checked by GLM-5.3 via Ollama Cloud · 15 Oct, 04:45
Source chain
AEvery quote below was checked, without a model, to appear verbatim in its source.
- 01
“roughly half of benchmark-passing patches written by agents would not be accepted by human project maintainers”
Full-text passage · p. 1 · evidence E1
Reports show that actual task coverage is a small fraction of what GenAI could feasibly automate and augment [38]. Even in task domain agents are most exposed such as coding, roughly half of benchmark-passing patches written by agents would not be accepted by human project maintainers [63]. This evidence suggests that the primary barrier to broader adoption is not only technical capability but also a better understanding of human–agent interaction. For ML researchers and practitioners, this means that advancing agents will require more than stronger models or better benchmarks. It will also require designs grounded in a deep understanding of how agents should communicate with humans, share control with humans, adapt to humans over time, and recover from interaction failure in ways that make them usable, trustworthy, and effective in real-world interactive settings. Human–Computer Interaction (HCI) is a discipline that has shaped how the technology industry understands the relationship between people and computing systems for over six decades [18]. Its ∗Contact the author athaiyiz@andrew.cmu.eduif you have any questions. Preprint. arXiv:2606.20630v1 [cs.HC] 28 May 2026
Passage read from arxiv.org, which may be a preprint rather than the published version.
- 02
“over-trust and accept uncritically the outputs of AI systems”
Full-text passage · no page number · evidence E47
Because of these changes, it is crucial to reconsider assumptions about human operators' role in supervisory control and the type of human–AI interactions that the literature should be designing for and optimising towards. The third limitation concerns the level of trust that a human operator has vis-à-vis the AI systems, and it can stem directly from the vicious cycle that the loss of situational awareness can trigger. Human operators tend to over-trust and accept uncritically the outputs of AI systems, which induces operational complacency that is counterproductive to the SHC rationale [34, 35]. The longer a given AI system has performed well, the more likely a human operator is to trust it and perceive some kind of “mechanistic objectivity” associated with computer-generated analytics, for example [2, p. 212], [36]. This can lead to over-trust dynamics, whereby the human agents ignore their experienced assessments—so-called automation bias [37]—or even “shirking part of their responsibility for decisions”, when this would contradict the behaviour of the AI system. The fourth limitation becomes evident when considering the specific types of risks that SHC aims to mitigate.
Passage read from www.ebi.ac.uk, which may be a preprint rather than the published version.