Evidence record EV-0019

Skill selection precision collapses as the skill pool grows

A preprint reports that as the pool of available skills grew from 5 to 100, the precision with which agents actually used the right skill fell from 29.6% to 3.3%.

Preprint · Measurement · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: skills, selection, discovery.

Effect, as the source reports it

  • Actual-use precision of skills. Baseline: Pool of 5 skills: 29.6%. With the change: Pool of 100 skills: 3.3%. Direction: decrease. Size: 29.6% to 3.3%. Sample: 8,135 normalised trial records.
  • How skills help, by mode. Direction: not-applicable. Size: Procedural anchoring 65.7% of skill cases; explicit knowledge injection 4.5%. Sample: 238 labelled records.

Agent profile

  • Note on models: Various benchmarks, harnesses and LLMs. The abstract does not name them.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Demystifying Agent Skills: Why They Work-Until They Don't, arXiv, Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li, 14 August 2026, arXiv:2608.14036. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

  • Preprint, not peer reviewed.
  • The abstract reports that downstream success stayed stable despite confusable distractors, so lower selection precision did not always mean failure.

For designers

Choosing a skill is a retrieval problem that gets harder with every skill installed. Keep the set small, and make each description say when the skill applies and when it does not.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0019" and this page's address, https://agentexperience.tech/evidence/ev-0019/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.