Evidence record EV-0012

Compact documentation did not help when the source was present

Across two model families and ten repositories, with a positive control, a preprint found that neither compact natural-language documentation nor retrieved context helped coding agents resolve issues better than the issue alone when the source code was present.

Preprint · Negative result · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: docs-for-agents.

Effect, as the source reports it

  • Issue resolution with documentation versus the issue alone (source present). Baseline: Issue alone. With the change: Static compact documentation, or retrieved context. Direction: no-change. Size: Neither beat the issue alone; a positive control confirmed the evaluation could detect a real gain. Sample: Two model families; ten repositories.

Agent profile

  • Note on models: Two model families. The abstract does not name them.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer, arXiv, Md Shohel Arman, Igor Molybog, 25 September 2026, arXiv:2609.31587. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

  • Preprint, not peer reviewed.
  • Applies where the agent can read the source; the paper characterises the boundary where documentation helps, which the abstract does not detail.

For designers

Documentation earns its place where the agent cannot read the source, such as a hosted API. Where the code is already in the workspace, extra summaries may add cost without helping.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0012" and this page's address, https://agentexperience.tech/evidence/ev-0012/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.