Evidence record EV-0014

Allow/ask/never policies blocked less overreach than per-action approval

In a study of 113 people without software backgrounds (preprint), user-authored allow/ask/never policies blocked 20.1 percentage points less agent overreach than per-action approval, partly because participants chose 'ask' for 114 of 140 rules and then approved most overreach at runtime.

Preprint · Negative result · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: approval, confirmation.

Effect, as the source reports it

  • Overreach blocked, user-authored policy versus per-action human approval. Baseline: Per-action human-in-the-loop approval. With the change: User-authored allow/ask/never policy. Direction: decrease. Size: -20.1 percentage points, 95% CI [-32.1, -8.1]; -14.5 points versus automated per-action review. Sample: 113 participants; 18-action simulated day with 7 overreach actions.
  • Runtime prompts per participant. Baseline: Per-action approval: 18.0. With the change: Policy: 10.9. Direction: decrease. Size: 18.0 to 10.9; total intervention time not reliably lower once rule setup is included. Sample: As above.
  • How overreach ran under the policy condition. Direction: not-applicable. Size: 133 of 148 executed overreach actions followed human approval; 15 ran under "allow" rules. Sample: Exploratory analysis.

Agent profile

  • Note on models: A language model mapped actions to consequence categories; it is not named in the abstract. Participants supervised a simulated day, not a live agent.

Conflicts of interest

None declared in the abstract. The full text was not checked for a competing-interest statement.

Source

Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?, arXiv, Ting Yan, 27 August 2026, arXiv:2608.27443. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

  • Preprint, not peer reviewed.
  • Pre-defined simulation with non-developers.
  • Some analyses are exploratory, as the abstract states.

For designers

Standing rules do not remove the approval problem if people keep choosing 'ask'. An approval request should show clearly when an action goes beyond what the person originally asked for.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0014" and this page's address, https://agentexperience.tech/evidence/ev-0014/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.