Evidence record EV-0025

Run-to-run variance dominates, and a verification tool beats a verification prompt

A preprint reports that about 54% of outcome variance in its coding-agent benchmark came from repeating the same configuration, and that a dedicated verification tool changed verification behaviour substantially where a prompt asking the agent to verify had little effect.

Preprint · Measurement · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: evaluation, measurement.

Effect, as the source reports it

  • Share of outcome variance from repeating an identical configuration. Direction: not-applicable. Size: About 54%. Sample: Four scientific tasks; more than 18,000 released trajectories.
  • Verification behaviour. Baseline: Prompt asking the agent to verify: little effect. With the change: Dedicated verification tool: substantial change. Direction: increase. Size: Qualitative in the abstract. Sample: As above.

Agent profile

  • Note on models: Several backbone models (one of the five configuration factors). The abstract does not name them.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Agents Are Systems, Not Models: Rethinking Agentic Evaluation, arXiv, Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata, 1 October 2026, arXiv:2610.01618. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

  • Preprint, not peer reviewed.
  • Four scientific tasks in which a coding agent must operate a published specialist model.

For designers

Run each test several times before you conclude that a change helped. If you want an agent to check its work, give it a tool for that rather than an instruction.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0025" and this page's address, https://agentexperience.tech/evidence/ev-0025/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.