# Run an Agent Experience review around one real workflow.

> A step-by-step method for running an Agent Experience review: pick one workflow, map its surfaces, watch the decision points, and turn patterns into product changes.

Published: 2026-08-25  
Updated: 2026-09-02  
Canonical: https://agentexperience.tech/insights/run-an-ax-review/  
Tags: evaluation, review, workflows, decision points

You do not need a large programme to learn something useful. Begin with one task that matters, then follow the decisions that shape it.

**In short.** An Agent Experience review is a bounded, repeatable inspection of one workflow an agent performs for a person. Choose a job with a clear outcome, map every surface the job touches, observe a handful of real runs, record each point where the agent hesitated or guessed, then ship the smallest product change that removes the largest hesitation. It is a product habit, not a launch gate, and it produces edits to descriptions, permissions, documentation, and handoffs — not a score.

The temptation with agent-facing work is to start with the model: swap it, prompt it harder, add a framework. That instinct hides the more tractable problem. Agent-facing failures decompose into distinct stages, and the tool-use literature has treated them separately for years — the API-Bank benchmark (Li et al., EMNLP 2023) was built around exactly this decomposition, with API planning, retrieval, and calling scored as separate tasks over 73 runnable APIs, because they fail independently. If planning, retrieval, and calling can fail independently in a benchmark, they can fail independently in your product, and a single success rate will not tell you which one did.

An AX review is how you find out which one did.

## What is an Agent Experience review?

An AX review is a structured observation of one agent-performed workflow across five surfaces: discovery, context, tool use, recovery, and delegated action. It asks a single question at each step — *what would have to be true for the agent to do this well?* — and records the answer in plain language.

It is not a benchmark. Public agent benchmarks measure a system, not your product, and a published number reflects a model plus a harness plus an environment version plus a verifier rather than a pure model score. It is not an audit either; nothing here produces a certificate. It is diagnostic work whose only output that matters is the next change you make.

## 1. Choose a job with a clear outcome

Pick a meaningful task a person would plausibly delegate. Define what success looks like, what information the agent starts with, and the point at which the workflow should stop or ask for help.

Three tests for a well-chosen job:

- **Someone would actually delegate it.** “Reconcile last month’s invoices against the ledger and flag mismatches” passes. “Use our API” does not.
- **Success is observable without arguing.** You can look at the end state and say yes or no. If the two of you cannot agree on the criterion before the review, you will not agree after it.
- **It has a stopping condition.** Name in advance the situation in which the agent should stop and ask rather than proceed. Reviews without a stated stopping point tend to rate confident overreach as success.

Write these three things down in four sentences. That artefact is the review’s scope, and it is what stops the review expanding into “how good is our AI”.

If choosing the job is itself hard, the underlying problem is probably scope rather than evaluation. [Start with workflows, not autonomy](/insights/start-with-workflows/) covers that failure mode.

## 2. Map the surfaces around the task

List the pages, tools, documentation, inputs, permissions, and people the workflow depends on. This prevents a team from treating a failure as only a model issue when the surrounding product surface is the real constraint.

The public agent stack is layered, and each layer is somebody’s responsibility. The owner column below is organisational convention rather than evidence; substitute the way your own team is arranged.

| Surface | Concrete artefact | Often owned by |
| --- | --- | --- |
| Discovery | Website pages, `robots.txt`, sitemap, `llms.txt`, docs index | Web / content |
| Context | `AGENTS.md`, README, help centre, schema descriptions | Docs / engineering |
| Tools | MCP tool definitions, function schemas, OpenAPI operations | Platform / API |
| Permissions | OAuth scopes, API keys, role boundaries, approval rules | Security / platform |
| Handoff | Approval UI, notification, ticket, escalation route | Product / design |
| Record | Traces, logs, retry history, audit trail | Engineering |

Fill in the actual file paths and URLs, not the categories. A map with real locators tells you who to talk to; a map with abstractions tells you nothing.

Two things are frequently missing from a first map. **Permissions**: teams describe what the agent should do and never write down what it *may* do. **The record**: if nobody can reconstruct what happened, the rest of the review is guesswork.

## 3. Watch the decision points

Pay attention to how the agent finds a capability, chooses between options, forms its inputs, handles missing context, and responds to a blocked action. Record the point of uncertainty in plain language.

Observe several runs of the same job — as a working range, five to ten is enough to distinguish a pattern from an accident and few enough to read every one. That range is a practical judgement, not a sampling rule. Vary the starting conditions deliberately: a complete request, an underspecified one, one where a dependency is unavailable, and one where the right answer is to stop and ask.

For each run, record six things:

1. **The moment the agent chose.** Which tool, which page, which branch.
2. **What it had to go on.** The description text, the page content, the schema.
3. **What it assumed.** Every unstated assumption is a piece of missing context.
4. **Where it hesitated, retried, or backtracked.** Retries are evidence, not noise.
5. **Whether it stopped at the right place.** Both silent overreach and unnecessary escalation are failures.
6. **What a person watching would have wanted to know.**

Two failure modes deserve particular attention because they are easy to misread.

**Near-miss tool selection.** The agent picked a neighbouring tool that partly works. This reads as a model error and is usually a description problem — see [Tool descriptions are product surfaces](/insights/tool-descriptions/) and [An agent’s tool catalog needs a pruning strategy](/insights/agent-tool-catalogs/).

**Successful runs with ugly paths.** A run that ends correctly after four wrong turns will pass any outcome-based check and will fail in production the first time one of those turns is slower. This is the argument in [Evaluate decisions, not just answers](/insights/evaluating-agent-workflows/).

## 4. Test the moments that carry risk

Not every step deserves equal scrutiny. Concentrate on the choices that create side effects, expose information, change a customer outcome, or become expensive to reverse.

Three checks earn their place in almost every review.

**Does the boundary hold when the model is wrong?** The current MCP specification (2026-07-28) is explicit that tool annotations are untrusted unless the server itself is trusted, and that the harness — not the model — must validate and authorise. If your safe path depends on the agent reading a hint correctly, it is not a boundary. It is a preference.

**What happens after untrusted content enters the context?** Indirect prompt injection is a demonstrated attack: Greshake et al. (2023) showed retrieved content steering model behaviour into unintended API calls, and a 2026 empirical study of injection in the wild documented attacks that were hidden and strategically placed across many models and page representations. Design-pattern work by Beurer-Kellner et al. (2025) argues the durable mitigation is structural — deliberately narrow the set of consequential actions available once untrusted input has been ingested. Ask, for your workflow: what can the agent still do after it reads a stranger’s text?

**Is there a route back?** Recovery should be part of the interaction model, not an exception path. The MCP tools specification includes multi-round-trip flows for when a tool needs more human input, and OAuth’s step-up authentication (RFC 9470) exists for tasks that become sensitive partway through. [Design for recovery, not perfect runs](/insights/design-for-recovery/) covers the product shape of this.

## 5. Turn patterns into product changes

Look for the smallest change that would make the path clearer: a better tool description, a missing label, an explicit approval step, a narrower capability, or a recovery route. Then review the workflow again.

Sort every finding into one of four buckets, because each has a different owner and a different cost:

- **Wording** — a tool description, a heading, a button label, a schema field description. Cheapest, and often the real fix.
- **Structure** — a missing page, an unlabelled control, a capability that should be two capabilities.
- **Boundary** — a permission that is too broad, an approval that should exist, a stopping condition that was never encoded.
- **Record** — something happened that nobody can reconstruct.

Then pick one. The discipline is choosing the single change with the best ratio of clarity gained to work required, shipping it, and running the review again. A review that produces a twenty-item backlog produces nothing.

Promote the instructive failures into a standing check as you go. That is how a review compounds into an evaluation loop rather than repeating itself — see [Build an evaluation loop that improves the product](/insights/build-a-reliable-agent-evaluation-loop/).

## 6. Keep the question alive

Agent-facing systems change with the product, the model, and the task. Treat the review as a repeatable product habit, not a once-and-done launch gate.

The dependencies genuinely move underneath you. Public benchmark infrastructure has had to solve the same problem: StableToolBench (ACL 2024 Findings) introduced simulated APIs and caching specifically because real APIs drift constantly, and Wu et al. (ACL 2026 Findings) argue that static toolsets are unrealistic and that agents need continual documentation adaptation as APIs version, deprecate, and reappear. Your review is a snapshot of a system with moving parts.

A workable cadence: re-run a review when the workflow changes materially, when a model or harness version changes, when a dependency’s contract changes, and otherwise on a fixed interval you actually keep. Record the date, the versions in play, and the one change you made. Six months of those entries is a more useful artefact than any single score.

## Frequently asked questions

**How long does an Agent Experience review take?**
There is no standard duration, and any number quoted here would be a working estimate rather than a measured one. What makes a first review finishable is scope: one job with a clear success condition, a small number of observed runs, and one written list of decision points. Reviews that begin as a broad programme tend to stall before they produce a product change.

**Who should run an AX review?**
Whoever owns the workflow, with someone who can read traces. It is a product review, not a model evaluation, so the useful room contains a product owner, an engineer who can see the tool calls and logs, and a writer or designer who owns the documentation and interface copy the agent depends on.

**What is the difference between an AX review and an eval suite?**
A review is qualitative and diagnostic; an eval suite is quantitative and regressive. The review finds where the agent’s path breaks and why. The eval suite locks in the cases you already understand so they do not silently return. Reviews should feed the eval suite, not compete with it.

**How do we know the review worked?**
By naming a specific product change and a specific decision point that changed with it. A review that ends in a document has not finished. A review that ends in an edited tool description, an added approval step, a narrowed permission, or a new recovery route has — and the next review tells you whether that change held.

A useful review leaves the team with one sharper workflow and a concrete next improvement.