Lab
The AX evidence register.
A public record of measured and observed findings about agent experience: how AI agents discover, choose, call and recover from the tools, APIs, CLIs, docs and approval flows that products offer them. Each record states one finding, gives the numbers as the source reported them, and says what kind of evidence it is and what it does not show.
40 records, last updated . Every source was fetched and checked on the date shown in its record. The register records what others have published, with their caveats; it does not rank anyone, re-run any result or publish experiments of its own.
Machine-readable copy: /evidence.json, with its JSON Schema. Cite the original source for any number you use. The register's licence will be confirmed before its source repository is published.
Jump to: Read this first · Evidence classes · Browse by pattern · All records
Read this first
- Most records are preprints. 27 of the 40 records come from papers that have not been peer reviewed. Many are by one author or a small team, and some authors have a stake in the result.
- Vendor claims are claims. A figure from a launch post or a press release is labelled as the vendor's claim. It can show a direction, not a size.
- Every number belongs to its agent profile. A result holds for the models, harness and dates the source tested. Newer models can behave differently, and one record shows the effect growing with a newer model.
- Many records were checked against the abstract only. 27 records say "abstract only": the numbers are in the live abstract, but the full text was not re-checked.
- Some patterns have no records. As of the last update there is no record that directly measures whether llms.txt, ARD, Server Cards or WebMCP change agent task success. An empty pattern below is information too.
Evidence classes
Read the class before you read the number. Each class lists its records, so this list also works as a filter.
- Peer reviewed (0). Published after independent review at a journal or conference. Review checks the method, not the truth of the result. The result still belongs to the models and dates tested.
- Preprint (27). A paper posted publicly, for example on arXiv, without peer review. Unreviewed. Many are by one author or a small team, and some authors have a stake in the result. Most AX research is in this class today. Records: EV-0001, EV-0002, EV-0003, EV-0004, EV-0005, EV-0006, EV-0007, EV-0008, EV-0009, EV-0010, EV-0011, EV-0012, EV-0013, EV-0014, EV-0015, EV-0016, EV-0017, EV-0018, EV-0019, EV-0020, EV-0021, EV-0022, EV-0023, EV-0024, EV-0025, EV-0026, EV-0027.
- Independent measurement (4). Someone with no stake in the product measured or reproduced a behaviour and published how, for example a bug report with steps to reproduce. Often a single report on one version. Software changes, so check again before you rely on it. Records: EV-0036, EV-0037, EV-0038, EV-0039.
- Vendor measurement (5). A vendor reports its own measurement with enough method to judge it, such as run counts, models and the harness. The vendor chose the tasks, ran the evaluation and wrote it up. Nobody else has reproduced it unless the record says so. Records: EV-0028, EV-0029, EV-0030, EV-0031, EV-0040.
- Vendor claim (4). A vendor states a figure without a published method. A claim, not a measurement. The definitions behind it are usually unstated, and the figure often comes from a launch or funding announcement. It can show a direction, not a size. Records: EV-0032, EV-0033, EV-0034, EV-0035.
- Anecdote (0). A single first-hand account without a method. Shows that something can happen, not how often it happens.
Each record also has a finding type: a measurement, a negative result (the change made things worse or did not work), a null result (no detectable effect), an observation (behaviour or prevalence, not an intervention) or a vendor claim. Negative and null results are recorded on purpose.
Browse by pattern
Records and ax-check rules grouped by the shared AX pattern vocabulary. A record can carry several tags.
All records
One line per record. Open a record for its effect sizes, agent profile, conflicts of interest, source and limitations.
How to cite and propose a record
Cite the original source for any number; the register is a pointer, not the origin of a finding. To cite a record, keep its class and model set with the figure, and link its page, for example: "a preprint testing five OpenAI models reports recovery rising from 45% to 84% (AX evidence register, EV-0003)".
Know of a finding that belongs here, or a mistake in one? Send the source through the contact page. A record needs a public source, the numbers exactly as reported and an honest evidence class. Dated news on the same patterns is in What changed in AX.