Run an Agent Experience review around one real workflow.

With agent-facing work it is tempting to start with the model: swap it, prompt it harder, add a framework. That instinct hides the problem you can more easily fix. Agent-facing failures break down into distinct stages, and the tool-use literature has treated them separately for years. The API-Bank benchmark (Li et al., EMNLP 2023) was built around exactly this decomposition, with API planning, retrieval, and calling scored as separate tasks over 73 runnable APIs, because they fail independently. If planning, retrieval, and calling can fail independently in a benchmark, they can fail independently in your product, and a single success rate will not tell you which one did. An AX review is how you find out.

What is an Agent Experience review?

An AX review is a structured observation of one agent-performed workflow across five surfaces: discovery, context, tool use, recovery, and delegated action. It asks a single question at each step (what would have to be true for the agent to do this well?) and records the answer in plain language.

It is not a benchmark. Public agent benchmarks measure a system, not your product, and a published number reflects a model plus a harness plus an environment version plus a verifier rather than a pure model score. It is not an audit either, and nothing here produces a certificate. It is diagnostic work, and the output that matters is the next change you make.

1. Choose a job with a clear outcome

Pick a meaningful task a person would plausibly delegate. Define what success looks like, what information the agent starts with, and the point at which the workflow should stop or ask for help.

Three tests for a well-chosen job:

• Someone would actually delegate it. “Reconcile last month’s invoices against the ledger and flag mismatches” passes. “Use our API” does not.
• Success is observable without arguing. You can look at the end state and say yes or no. If the two of you cannot agree on the criterion before the review, you will not agree after it.
• It has a stopping condition. Name in advance the situation in which the agent should stop and ask rather than proceed. Reviews without a stated stopping point tend to rate confident overreach as success.

Write these three things down in four sentences. That is the review’s scope, and it stops the review expanding into “how good is our AI”.

If choosing the job is itself hard, the underlying problem is probably scope rather than evaluation. Agent workflow design: scoping the first one covers that failure mode.

2. Map the surfaces around the task

List the pages, tools, documentation, inputs, permissions, and people the workflow depends on. This stops a team treating a failure as only a model issue when the surrounding product surface is what actually limits the agent.

The public agent stack is layered, and each layer is somebody’s responsibility. The owner column below is organisational convention rather than evidence; substitute the way your own team is arranged.

Fill in the actual file paths and URLs instead of the categories. A map with real locators tells you who to talk to, and a map of abstractions tells you nothing.

Two things are frequently missing from a first map. Permissions: teams describe what the agent should do and never write down what it may do. The record: if nobody can reconstruct what happened, the rest of the review is guesswork.

3. Watch the decision points

Pay attention to how the agent finds a capability, chooses between options, forms its inputs, handles missing context, and responds to a blocked action. Record the point of uncertainty in plain language.

Observe several runs of the same job. As a working range, five to ten is enough to tell a pattern from an accident and few enough to read every one. That range is a practical judgement, not a sampling rule. Vary the starting conditions deliberately: a complete request, an underspecified one, one where a dependency is unavailable, and one where the right answer is to stop and ask.

For each run, record six things:

1. The moment the agent chose. Which tool, which page, which branch.
2. What it had to go on. The description text, the page content, the schema.
3. What it assumed. Every unstated assumption is a piece of missing context.
4. Where it hesitated, retried, or backtracked. Retries are evidence, not noise.
5. Whether it stopped at the right place. Both silent overreach and unnecessary escalation are failures.
6. What a person watching would have wanted to know.

Two failure modes need particular attention because they are easy to misread.

Near-miss tool selection. The agent picked a neighbouring tool that partly works. This reads as a model error and is usually a description problem: see Write tool descriptions an agent can act on and Agent tool catalogues need a pruning strategy.

Successful runs with ugly paths. A run that ends correctly after four wrong turns will pass any outcome-based check and will fail in production the first time one of those turns is slower. This is the argument in Evaluate the decisions in an agent workflow.

4. Test the moments that carry risk

Concentrate on the choices that create side effects, expose information, change a customer outcome, or become expensive to reverse.

Three checks belong in almost every review.

Does the boundary hold when the model is wrong? The current MCP specification (2026-07-28) is explicit that tool annotations are untrusted unless the server itself is trusted, and that the harness, not the model, must validate and authorise. If your safe path depends on the agent reading a hint correctly, it is a preference rather than a boundary.

What happens after untrusted content enters the context? Indirect prompt injection is a demonstrated attack: Greshake et al. (2023) showed retrieved content steering model behaviour into unintended API calls, and a 2026 empirical study of injection in the wild documented attacks that were hidden and strategically placed across many models and page representations. Design-pattern work by Beurer-Kellner et al. (2025) argues the durable mitigation is structural: deliberately narrow the set of consequential actions available once untrusted input has been ingested. Ask, for your workflow: what can the agent still do after it reads a stranger’s text?

Is there a route back? Recovery should be part of the interaction model instead of an exception path. The MCP tools specification includes multi-round-trip flows for when a tool needs more human input, and OAuth’s step-up authentication (RFC 9470) exists for tasks that become sensitive partway through. Agent error recovery belongs on the main path covers the product shape of this.

5. Turn patterns into product changes

Look for the smallest change that would make the path clearer: a better tool description, a missing label, an explicit approval step, a narrower capability, or a recovery route. Then review the workflow again. For public examples of each kind of change, see the annotated agent experience examples.

Sort every finding into one of four buckets, because each has a different owner and a different cost:

• Wording: a tool description, a heading, a button label, a schema field description. Cheapest, and often the real fix.
• Structure: a missing page, an unlabelled control, a capability that should be two capabilities.
• Boundary: a permission that is too broad, an approval that should exist, a stopping condition that was never encoded.
• Record: something happened that nobody can reconstruct.

Then pick one: the single change with the best ratio of clarity gained to work required. Ship it and run the review again. A review that produces a twenty-item backlog produces nothing.

Promote the instructive failures into a standing check as you go. That way the reviews build into an evaluation loop instead of repeating themselves. See An agent evaluation loop turns one failure into a standing check.

6. Keep the question alive

Agent-facing systems change with the product, the model, and the task, so treat the review as a habit you repeat instead of a launch gate you pass once.

The dependencies really do move underneath you. Public benchmark infrastructure has had to solve the same problem: StableToolBench (ACL 2024 Findings) introduced simulated APIs and caching specifically because real APIs drift constantly, and Wu et al. (ACL 2026 Findings) argue that static toolsets are unrealistic and that agents need continual documentation adaptation as APIs version, deprecate, and reappear. A review is a snapshot of a system with moving parts.

A workable cadence: re-run a review when the workflow changes materially, when a model or harness version changes, when a dependency’s contract changes, and otherwise on a fixed interval you actually keep. Record the date, the versions in play, and the one change you made. Six months of those entries is more useful than any single score.

Frequently asked questions

How long does an Agent Experience review take? There is no standard duration, and any number quoted here would be a working estimate rather than a measured one. What makes a first review finishable is scope: one job with a clear success condition, a small number of observed runs, and one written list of decision points. Reviews that begin as a broad programme tend to stall before they produce a product change.

Who should run an AX review? Whoever owns the workflow, with someone who can read traces. It is a product review, not a model evaluation, so the useful room contains a product owner, an engineer who can see the tool calls and logs, and a writer or designer who owns the documentation and interface copy the agent depends on.

What is the difference between an AX review and an eval suite? A review is qualitative and diagnostic; an eval suite is quantitative and regressive. The review finds where the agent’s path breaks and why. The eval suite locks in the cases you already understand so they do not silently return. Reviews should feed the eval suite, not compete with it.

How do we know the review worked? By naming a specific product change and a specific decision point that changed with it. A review that ends in a document has not finished. A review that ends in an edited tool description, an added approval step, a narrowed permission, or a new recovery route has, and the next review tells you whether that change held.

A useful review leaves the team with one sharper workflow and a concrete next improvement.