Evidence record EV-0002

Constraints stated only in prose produce silent failures on live APIs

SilentProbe (preprint) reports that only 7.5% of 2,501 public OpenAPI documents declare an enum, and that on live endpoints machine-checkable constraints returned an honest error in 111 of 111 cases while prose-only constraints failed silently in 44 of 61.

Preprint · Measurement · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: schema-enum, error-recovery, false-success.

Effect, as the source reports it

  • Share of OpenAPI documents that encode constraints. Direction: not-applicable. Size: 7.5% declare an enum; 15.2% declare any machine-checkable constraint; 40.1% state at least one constraint in prose that the schema does not encode. Sample: 721,320 parameters across 2,501 independently published OpenAPI documents.
  • Responses to schema-derived invalid requests that fail silently (HTTP 200 with a parsable body). Baseline: Prose-only constraint: 44 of 61 failed silently. With the change: Machine-checkable constraint: 0 of 111 failed silently (111 of 111 honest errors). Direction: decrease. Size: About 72% silent to 0% silent (p = 2e-13). Sample: 219 perturbations against live endpoints of 27 vendors.

Agent profile

  • Note on models: Not an agent run: a static audit of OpenAPI documents plus scripted requests to live endpoints.

Conflicts of interest

The paper states that the authors are affiliated with Monid, Inc., the aggregation layer through which the live endpoints were reached.

Source

SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools, arXiv, Zongrong Li, Shengkun Ye, Feiyou Guo, Zuoyou Dang, 29 August 2026, arXiv:2609.00035. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked.

Limitations

  • Preprint, not peer reviewed.
  • The abstract says constraint form "predicts" honesty; it is an association across endpoints, not a controlled change to one API.
  • Live probes went through one aggregation layer to 27 vendors.

For designers

An API that states its rules only in prose can answer a bad request with an empty success, which an agent cannot tell apart from "no results". Encode the rules in the schema so the server can return an error the agent can act on.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0002" and this page's address, https://agentexperience.tech/evidence/ev-0002/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.