# Decide what you are measuring before you measure it.

> A practical guide to measuring Agent Experience: what construct you are actually measuring, the four families of AX measurement, the instrumentation a workflow needs before any number means anything, and the claims a measurement cannot support.

Published: 2026-09-02  
Updated: 2026-09-02  
Canonical: https://agentexperience.tech/insights/measure-agent-experience/  
Tags: evaluation, measurement, instrumentation, reliability, workflows

Most arguments about agent metrics are really arguments about constructs. Two teams say “success rate” and mean different events, different denominators, and different runs.

**Measuring Agent Experience is a measurement-design problem before it is a data problem.** Name the construct, choose an instrument that matches it, write the run record that instrument depends on, and keep the claim inside what the construct can support. Skip the first step and every later argument becomes unresolvable, because the disagreement is about what the number means rather than about what it says.

This page covers the constructs and the instrumentation. It does not report scores, and it does not say which measurement is best — those would be claims this site does not make.

## Why does "how well is the agent doing?" fail as a question?

Because it names no object. There are at least four different things a team can mean by it, and they come apart immediately in practice.

Consider a workflow that reconciles an account. It succeeds. Did it succeed because the product was legible, or because the model was strong enough to work around an ambiguous tool catalogue? Did it succeed once, or does it succeed every time? Did it succeed in four steps or in forty, with two human rescues? Did it succeed by a route the team would endorse if they read it?

Those are four questions, four instruments, and four different denominators. "How well is the agent doing?" collapses them, and a single number that answers it is a number that answers none of them. The [decisions-not-answers](/insights/evaluating-agent-workflows/) position is the same argument made about a single run; this page is that argument made about a measurement programme.

## What are the four families of AX measurement?

Each family answers a different question, needs a different instrument, and supports a different kind of claim.

| Family | Question | Instrument | What it cannot tell you |
|---|---|---|---|
| **Declared surfaces** | What has the product made explicit? | Inspection, no agent run required | Whether any agent uses it well |
| **Path quality** | Which decisions did the agent make, and were they right? | Trace review and trajectory metrics | Whether the path repeats |
| **Reliability** | Does the same task succeed across repeats and perturbations? | Repeated runs, fault injection | Why a failure happened |
| **Cost of the path** | What did the run consume? | Step, token, latency and intervention counts | Whether the outcome was worth it |

### Declared surfaces

The cheapest family, and the only one that needs no agent. You inspect what the product has put on the record: discovery files, structure and semantics, machine-readable content, action safety, recovery routes, policy signals. [The Open Agent-Readiness Rubric](/insights/agent-readiness-rubric/) is this site's version — 36 checks across six dimensions, each with a concrete request or inspection that settles it and a published weight.

Public tooling measures related objects. Cloudflare's Agent Readiness score, announced 17 April 2026 and published through `isitagentready.com`, evaluates edge-observable signals — `robots.txt`, `sitemap.xml`, `Link` headers, Markdown content negotiation, AI bot rules, Content Signals, an MCP server card, an API catalogue, and agentic-commerce declarations — and was added to the URL Scanner in the Cloudflare dashboard on 12 May 2026. That is a scan of a live domain from the outside; a rubric run by hand is a self-assessment that can be run against staging. They measure overlapping surfaces from different vantage points and their numbers are not interchangeable.

The important property of this whole family is what it excludes. A declared-surface score is a precondition check. It does not predict agent success, and the rubric says so about itself.

### Path quality

The family that requires actually watching a run. The measurable events are decisions: was the right capability selected, were the inputs well formed, was a boundary respected, did the agent stop when it should have, was recovery possible.

Public evaluation work has moved the same way, which makes it a useful source of measurement design. AgentBoard reports progress rate and grounding accuracy alongside success rate, precisely because a pass/fail verdict hides how the result was reached. Mind2Web 2 scores citation-backed answer quality through an agent-as-a-judge rubric rather than string matching. Benchmarks in the MCP family measure the server rather than the agent — task completion, latency and token consumption held under matched agent settings — which is a reminder that "the agent" is rarely the only thing under test.

Borrow the decomposition. Do not borrow the numbers: a published score is a model plus a harness plus an environment version plus a verifier, and several of these suites state plainly that their own releases are not comparable across versions.

### Reliability

Path quality on one run tells you almost nothing about the run a user will get. Reliability is the family that repeats.

The construct worth knowing here is `pass^k`, introduced in the τ-bench work (Yao et al., ICLR 2025): a task counts as successful only if the agent succeeds on *all* `k` attempts, rather than on any one of them. It matters because production users experience the distribution of runs, not the best run — and because a workflow with an aggressive retry loop can post a strong "any attempt" number while being unpleasant to depend on.

The second construct is perturbation. ReliabilityBench varies task perturbations and injected tool or API failures and reports a reliability surface rather than a single scalar. The practical version for a product team is fault injection: simulate timeouts, rate limits, partial responses, and schema drift, and measure whether the workflow recovers or merely repeats. That measurement is only possible if the product [designs for recovery](/insights/design-for-recovery/) in the first place — you cannot measure a route that does not exist.

### Cost of the path

Steps, tokens, wall-clock time, price per task, and — the one most often left out — human interventions per completed task.

Intervention count is the AX-specific member of this family, and it is often the most honest single number a team has. A workflow that completes reliably but needs a person twice per run has not been delegated; it has been supervised. Counting interventions, and classifying what each one was *for* (missing context, ambiguous instruction, unsafe action, plain failure), turns a vague sense that "it still needs babysitting" into something a product decision can attach to.

## What has to be instrumented first?

Almost every metric above is a query over a single artefact: the run record. Write it badly and no amount of later analysis recovers the information.

A run record that supports AX measurement carries, at minimum:

- **A stable task identity** — the same job, recognisably the same across runs, so a denominator exists.
- **The tool calls in order**, with inputs and outcomes, so selection and input formation are inspectable rather than inferred.
- **Every attempt, with its trigger and what changed.** [Retries are product history](/insights/retries-are-product-history/): without `trigger` you cannot tell an automatic backoff from a person clicking the button again, and without a record of what changed since the previous attempt you cannot tell a recovery from a repetition.
- **Each human intervention**, with what it was for.
- **A terminal state that distinguishes four things**, not two: success, partial progress, refused, and unsafe. Collapsing refused into failure is the single most common way a run record destroys its own usefulness — a correct refusal and a broken workflow are not the same event, and a metric that cannot separate them will punish the safe behaviour.

Two disciplines apply to the record itself. Redact at write time rather than at analysis time, and keep only what someone debugging at an awkward hour would actually need; a fuller log is not automatically better governance. The [AX review](/insights/run-an-ax-review/) treats this record as its primary evidence, which is a good test of whether yours is adequate — if a review cannot be run from it, it is not instrumented yet.

## Can you measure the experience by asking the agent?

It is a real technique with a real limitation, and it is worth understanding both.

The approach evaluates a completed session's transcript and derives dimensions of the experience the agent encountered. DX's Agent Experience Score, documented publicly for coding-agent sessions, scores three constructs — Requirements ("whether the goal was clear and the initial context helped the agent begin"), Steering ("whether follow-up input helped guide progress"), and Scope ("whether the output stayed aligned to a useful task boundary") — each on a 1–5 scale, averaged into an overall figure. The evaluation is performed after the session by a separate model reading a scrubbed transcript, not by the coding agent itself, and sessions with insufficient evidence or unreadable transcripts can go unevaluated.

What this family measures is genuinely hard to get any other way: friction that only the thing doing the work can see. A human developer needs a survey to report that a repository was hard to navigate. The transcript already contains it.

The limitation is the same one that applies to every judge-based measurement, and it is structural rather than a flaw in any particular implementation: the judge becomes part of the instrument. A change in the judge model, the prompt, or the scrubbing rules moves the score without anything about the product changing. Treat a transcript-derived score as a signal that travels with its own version number, compare it only against itself over time, and expect it to answer "where did friction appear?" rather than "how good is this?"

Note also the scoping. This construct is defined over coding-agent sessions inside a repository — the inward, developer-tooling sense of AX. The outward sense, where an agent works through a product on behalf of a customer, is a different population and would need its own definition of what a session even is. [AX compared with UX, DX, and GEO](/insights/ax-vs-ux-dx-geo/) works through why those two senses of the term keep getting conflated.

## What claims can these measurements not support?

Worth writing down before anyone builds a slide:

- **Not a comparison to a public benchmark.** A benchmark number is model plus harness plus environment version plus verifier. Several suites state that their own releases are not comparable across versions.
- **Not a claim about a model.** If the harness changed, the measurement moved for reasons that have nothing to do with the model or the product.
- **Not a prediction from a readiness score.** Declared surfaces are a precondition, not a forecast.
- **Not a stable number across judge changes**, wherever a judge is in the loop.
- **Not a substitute for the qualitative review.** Metrics tell you that something regressed. They rarely tell you why, and the why is where the product change lives.

## Where to start

One workflow, one construct, one week. Choose the workflow people actually want delegated. Write the run record described above, even by hand, for ten runs. Count two things: how many terminal states were success versus partial versus refused versus unsafe, and how many human interventions each run needed.

That is a small measurement, and it will already do the thing measurement is for — it will make a disagreement resolvable. Then turn it into something that persists: the [evaluation loop](/insights/build-a-reliable-agent-evaluation-loop/) is the version of this that survives contact with a roadmap, because it converts a recurring failure into a durable check rather than a recurring conversation.

## Frequently asked questions

**How do you measure agent experience?**
By naming the construct first, then choosing an instrument that matches it. Four families cover most of it: declared surfaces (what the product has made explicit, checked by inspection), path quality (which decisions the agent made and whether they were right), reliability (whether the task succeeds across repeats and perturbations), and cost of the path (steps, tokens, latency, and human interventions). Each needs a different instrument, and none substitutes for another.

**What is the difference between an agent-readiness score and an agent-experience measurement?**
A readiness score describes what a product has declared — files, headers, markup, schemas, error payloads — and can be settled by inspection without running an agent at all. An agent-experience measurement describes what happened when an agent worked through the product. Readiness is a precondition you can check on a static site; experience requires runs, traces, and a definition of what counted as success.

**What should we instrument before we start measuring?**
The run record: stable task identity, tool calls in order with inputs and outcomes, every retry with its trigger and what changed since the previous attempt, each human intervention and what it was for, and a terminal state that separates success, partial progress, refused, and unsafe. Almost every AX metric worth having is a query over that record.

**Can we compare our agent's numbers to a public benchmark?**
Not directly. A published number reflects a model plus a harness plus an environment version plus a verifier, and benchmark releases are frequently not comparable across their own versions. Use public benchmarks to borrow measurement design — trajectory metrics, repeats, perturbation, cost reporting — not to position a product against a leaderboard.

A measurement is only as good as the decision it changes. Instrument the run record first; the numbers are downstream of it.