Evidence record EV-0018

Skill rules that name a command or path change what agents do

A study of 3,159 skills (preprint) found that adding a checkable rule raised the rate at which four coding agents took the required action by +0.23 on average, with the gain coming mainly from rules naming a command or path the old skill did not mention.

Preprint · Measurement · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: skills, description.

Effect, as the source reports it

  • Rate at which the agent takes the required action after a rule is added. Baseline: Skill before the revision. With the change: Skill with the added rule. Direction: increase. Size: +0.23 on average (+0.16 to +0.36). Sample: Four agents in a sandbox.
  • Single-answer compliance after a rule is added. Baseline: Before. With the change: After. Direction: increase. Size: +0.41 on average. Sample: 16 open-weight models.
  • Final correctness. Baseline: Before. With the change: After. Direction: increase. Size: +0.10 on average (+0.06 to +0.14). Sample: Three agents assessed by blind judges.
  • Share of the action gain kept when the skill body loads only on demand. Direction: not-applicable. Size: About 51% for the four agents; about 38% for the three open models. Sample: As above.
  • Episode tokens when a skill body is loaded. Baseline: Body not loaded. With the change: Body loaded. Direction: increase. Size: +50% on average. Sample: As above.

Agent profile

  • Note on models: 21 models for single answers and four agents in a sandbox. The abstract does not name them.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Agent Skill Evolution: How Revisions Affect Coding Agents, arXiv, Jiajie Wang, Yutong Zhao, Tianlin Li, Huashan Chen, Jinfu Chen, Kebin Peng, Sen He, 4 October 2026, arXiv:2610.04832. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

  • Preprint, not peer reviewed.
  • Limited to revisions that add or remove an automatically checkable rule.

For designers

In a skill, name the exact command or file path the agent should use. Because skills often load only on demand, put the most important rule where the agent sees it without loading the body.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0018" and this page's address, https://agentexperience.tech/evidence/ev-0018/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.