Evidence record EV-0001

An enum in the schema ends silent failures from example-only vocabularies

SilentProbe (preprint) reports that a vocabulary a parameter description only exemplified ("e.g.") was missed on 88 of 88 attempts across twelve models, and that promoting it into the schema cut the failure to 0 of 89.

Preprint · Measurement · Retrieved

Evidence class: Preprint. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Pattern tags: schema-enum, description, false-success.

Effect, as the source reports it

  • Attempts that missed the required vocabulary value. Baseline: Vocabulary only exemplified in the description ("e.g."): 88 of 88 missed. With the change: Vocabulary promoted into the schema: 0 of 89 missed. Direction: decrease. Size: 100% of attempts to 0%. Sample: 88 and 89 attempts; twelve models across eight families.
  • Correct use when the full vocabulary is written out in the description. With the change: 88 to 91% correct. Direction: not-applicable. Size: 88 to 91%. Sample: Twelve models across eight families.
  • Agent behaviour after a silent failure, full agent loop. With the change: Detected the failure in 12% of cases, repaired it in 0%, told the user a false negative in 41%, invented a figure in 12%. Direction: not-applicable. Size: 12% detected, 0% repaired, 41% false negative, 12% invented figure. Sample: Not stated in the abstract.

Agent profile

  • Note on models: Twelve models across eight families. The abstract does not name them.
  • Harness: The authors' agent loop, calling live commercial endpoints through one aggregation layer (Monid).

Conflicts of interest

The paper states that the authors are affiliated with Monid, Inc., the aggregation layer used as the measurement instrument.

Source

SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools, arXiv, Zongrong Li, Shengkun Ye, Feiyou Guo, Zuoyou Dang, 29 August 2026, arXiv:2609.00035. Retrieved ; verification: abstract-only.

Every number in this record was checked against the live arXiv abstract page on 2026-10-08. The full text was not re-checked. The competing-interest statement was read in the arXiv HTML full text on the same date.

Limitations

  • Preprint, not peer reviewed.
  • Endpoints were reached through one aggregation layer run by an organisation the authors are affiliated with.
  • The abstract does not name the twelve models, so the result cannot yet be tied to a model generation.

For designers

When a parameter accepts a closed set of values, put the set in the schema as an enum instead of giving examples in prose. A schema can be enforced by something other than the model; a description cannot.

Cite this record

Cite the original source for any number, and keep the evidence class and model set with the figure. To point at this record, use "AX evidence register, EV-0001" and this page's address, https://agentexperience.tech/evidence/ev-0001/. The record is also in /evidence.json. The register's licence will be confirmed before its source repository is published.