Evidence record EV-0029

A warning that names the failing plan redirected agents; a tip did not

Microsoft reports that a documentation tip pointing to the right tool got 1 of 5 agent runs to use it, while a warning naming the agent's failing approach ('Manually updating package.json alone will result in build failures') got 5 of 5.

Vendor measurement · Measurement · Retrieved

Evidence class: Vendor measurement. The vendor chose the tasks, ran the evaluation and wrote it up. Nobody else has reproduced it unless the record says so. Pattern tags: docs-for-agents, propensity.

Effect, as the source reports it

  • Runs in which the agent used the recommended CLI for an SPFx upgrade. Baseline: More direct tip naming the tool: 1 of 5. With the change: Warning naming the failing approach: 5 of 5. Direction: increase. Size: 1/5 to 5/5. Sample: Five runs per condition.

Agent profile

  • Note on models: The post does not state the profile. Microsoft's SPFx case study of the same documentation work used GitHub Copilot Chat in VS Code on Windows with Claude Sonnet 4.6.

Conflicts of interest

Microsoft authors evaluating with Microsoft tools (GitHub Copilot Chat, VS Code) and, where noted, Microsoft products. Microsoft reports the results itself. The documentation measured is Microsoft's SharePoint Framework documentation.

Source

Your agent already has a plan, Microsoft for Developers, Garry Trinder, 26 June 2026. Retrieved ; verification: verified-live.

The full primary source was fetched on 2026-10-08 and every number in this record was found in it. The page displays 25 June 2026; its article:published_time metadata is 2026-06-26T00:00:05Z. Pull request #10921, which shipped the warning across release pages, was confirmed merged on 2026-07-14.

Corroborating sources:

Limitations

  • Vendor-run evaluation, not peer reviewed; no raw data or harness was found published with the post.
  • Five runs per condition; Microsoft scopes each result to the agent profile it measured.
  • One upgrade scenario; the authors advise using this tactic sparingly.

For designers

If agents keep taking a wrong path despite your docs, name that path on the page and say plainly that it fails, then give the right alternative. Keep it for cases where the default really fails.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0029" and this page's address, https://agentexperience.tech/evidence/ev-0029/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.