Evidence record EV-0017

Injected skills lowered pass rates and raised token cost on average

WebDev-Skills-Bench (preprint) found that injecting matched public skills reduced mean Pass@2 by 1.3% to 4.2% across four models and raised token cost by 72% to 394%, with gains in only 17% to 36% of skill-project pairs.

Preprint · Negative result · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: skills, cost, context-budget.

Effect, as the source reports it

  • Mean Pass@2 with the target skill injected. Baseline: No skill. With the change: Target skill injected. Direction: decrease. Size: -1.3% to -4.2%. Sample: 31 public WebDev skills; 50 projects; 1,000 ordered tasks; four models.
  • Token cost. Baseline: No skill. With the change: Target skill injected. Direction: increase. Size: +72% to +394%. Sample: As above.
  • Skill-project pairs that gained. Direction: not-applicable. Size: 17% to 36%. Sample: As above.

Agent profile

  • Note on models: Four models. The abstract does not name them.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Signal or Noise? A Benchmark Study of Agent Skills in Web Development, arXiv, Ziyue Yang, Fan Ding, 24 August 2026, arXiv:2608.23067. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

  • Preprint, not peer reviewed.
  • Web development tasks only.
  • Skill rankings transferred weakly across models.

For designers

Treat a skill as a hypothesis about one project and model, not a portable asset. Test it against a no-skill baseline and a length-matched control before you ship it; within helpful skills, anti-pattern rules beat example-heavy content.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0017" and this page's address, https://agentexperience.tech/evidence/ev-0017/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.