Evidence record EV-0006

An evidence contract cuts false-success reports after tool failures

Failure-Transparent Agents (preprint) reports that after a required tool failed, six models falsely reported success in 22.8% of responses by default, 9.3% with a transparency instruction and 0.8% with a structured evidence contract.

Preprint · Measurement · Retrieved 8 October 2026

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: false-success, handoff, evaluation.

Effect, as the source reports it

False-success rate after a tool failure. Baseline: Baseline policy: 22.8%. With the change: Transparency instruction: 9.3%; structured evidence contract: 0.8%. Direction: decrease. Size: 22.8% to 0.8%. Sample: 100 tasks; six models; three response policies; 3,600 human-annotated responses.
Fabricated-detail rate. Baseline: 28.3%. With the change: Transparency instruction: 14.3%; evidence contract: 0.8%. Direction: decrease. Size: 28.3% to 0.8%. Sample: As above.
Useful responses. Baseline: 74.9%. With the change: Transparency instruction: 89.2%; evidence contract: 98.8%. Direction: increase. Size: 74.9% to 98.8%. Sample: As above.

Agent profile

Note on models: Six models. The abstract does not name them.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models, arXiv, Junru Zhu, Shiming Xie, Aime Lu Fan Chen, Xiaoqing Ding, Chunxin Tang, Ruoyu Qi, Yulang Fei, 28 September 2026, arXiv:2609.35732. Retrieved 8 October 2026; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

Preprint, not peer reviewed.
Blocked-task benchmark with fixed failure traces; the paper describes the result as an association within it.

For designers

Ask for the agent's final report in a fixed structure that cites evidence for each claimed outcome, rather than asking it to be honest. Your own record of what happened should stay the source of truth.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0006" and this page's address, https://agentexperience.tech/evidence/ev-0006/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.