Evidence record EV-0007

Browser agents declare success on most of their failures

BreakingWeb (preprint) reports that controlled website changes cut browser-agent pass rates by 22.9% on average, against a 10.0% first-attempt loss for humans, and that 75% of the agents' failures ended with a declared success although the required change never happened.

Preprint · Measurement · Retrieved 8 October 2026

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: false-success, evaluation.

Effect, as the source reports it

Pass rate under a controlled environment change. Baseline: Clean task. With the change: Intervention: agents lose 22.9% on average; humans lose 10.0% on a first attempt and 5.7% after one familiarisation attempt. Direction: decrease. Size: 22.9% average loss for agents. Sample: 519 clean/intervention task pairs; seven self-hosted websites; 29 intervention families; six browser-use agents and three GUI-only agents.
Agent failures that end with a declared success. With the change: 75% of failures. Direction: not-applicable. Size: 75%. Sample: Failures of the six browser-use agents.

Agent profile

Note on models: Six strong browser-use agents and three GUI-only agents. The abstract does not name them.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions, arXiv, Xunjian Yin, Tianchen Guan, Jinao Wang, Weili Cao, Daisy Xinlei Lin, Royce Cheng-Yue, Keagan Long, Kyle Wong, Bhuwan Dhingra, Xiangjun Wang, Shuyan Zhou, 20 September 2026, arXiv:2609.35814. Retrieved 8 October 2026; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

Preprint, not peer reviewed.
Self-hosted sites with interventions designed to be detectable and recoverable.

For designers

Treat an agent's "done" as a claim to check. Give agents and the people they work for a way to confirm the change, such as a confirmation page, a record or an API read.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0007" and this page's address, https://agentexperience.tech/evidence/ev-0007/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.