Decide what you are measuring before you measure it.

Measuring Agent Experience is a measurement-design problem before it is a data problem. Name the construct, choose an instrument that matches it, write the run record that instrument depends on, and keep the claim inside what the construct can support. If you skip the first step, later arguments cannot be settled, because people are disagreeing about what the number means, not about what it says.

This page covers the constructs, the key metrics, the comparison that tells you whether a change helped, and the instrumentation. It does not report scores, and it does not say which measurement is best, because this site does not make those claims.

Scope: this page is about AI agents doing work for people. Searches for “agent experience metrics” also return contact-centre measures of the working life of human support staff. That is a different subject with its own literature; the meanings of AX sorts the two apart.

Why does “how well is the agent doing?” fail as a question?

Because it names no object. A team can mean at least four different things by it, and they come apart quickly in practice.

Consider a workflow that reconciles an account. It succeeds. Did it succeed because the product was legible, or because the model was strong enough to work around an ambiguous tool catalogue? Did it succeed once, or does it succeed every time? Did it succeed in four steps or in forty, with two human rescues? Did it succeed by a route the team would endorse if they read it?

Those are four questions, with four instruments and four different denominators. “How well is the agent doing?” collapses them, and a single number that answers it answers none of them. The decisions-not-answers position is the same argument made about a single run; this page is that argument made about a measurement programme.

What are the four families of AX measurement?

Each family answers a different question, needs a different instrument, and supports a different kind of claim.

Declared surfaces

This is the cheapest family. You inspect what the product has put on the record: discovery files, structure and semantics, machine-readable content, action safety, recovery routes, policy signals. The Open Agent-Readiness Rubric is this site’s version. It checks one job across six tracks, from finding the page to handing the result back, and gives a profile rather than a score. Most of its checks need no agent run; its top level, Shown, adds repeated observed runs from the reliability family below.

Public tooling measures related objects. Cloudflare’s Agent Readiness score, announced 17 April 2026 and published through isitagentready.com, evaluates edge-observable signals (robots.txt, sitemap.xml, Link headers, Markdown content negotiation, AI bot rules, Content Signals, an MCP server card, an API catalogue, and agentic-commerce declarations) and was added to the URL Scanner in the Cloudflare dashboard on 12 May 2026. That is a scan of a live domain from the outside; a rubric run by hand is a self-assessment that can be run against staging. They measure overlapping surfaces from different vantage points and their numbers are not interchangeable.

What matters most about this family is what it leaves out. A declared-surface score is a precondition check. It does not predict agent success, and the rubric says so about itself.

Path quality

This family requires actually watching a run. The measurable events are decisions: was the right capability selected, were the inputs well formed, was a boundary respected, did the agent stop when it should have, was recovery possible.

Two questions in this family now have names. Microsoft’s “What is Agent Experience (AX)?” (6 October 2026) splits it into propensity, whether the agent finds and chooses your technology when the task does not name it, and efficacy, whether it uses your technology correctly when the task does (developer.microsoft.com). The test design follows from the definitions: write some tasks that never mention your product and some that do, and score them separately. A product can be chosen often and used badly, or used well by the few agents that find it.

Public evaluation work has moved the same way, so it is a useful source of measurement design. AgentBoard reports progress rate and grounding accuracy alongside success rate, because a pass/fail verdict hides how the result was reached. Mind2Web 2 scores citation-backed answer quality through an agent-as-a-judge rubric rather than string matching. Benchmarks in the MCP family measure the server rather than the agent (task completion, latency and token consumption held under matched agent settings), and “the agent” is rarely the only thing under test.

Borrow the decomposition. Do not borrow the numbers: a published score is a model plus a harness plus an environment version plus a verifier, and several of these suites state plainly that their own releases are not comparable across versions.

Reliability

Path quality on one run tells you almost nothing about the run a user will get, so this family repeats the run.

Start with plain repetition. The same prompt, model and setup can produce different results from run to run, so a single run is an anecdote. Microsoft’s guidance is to run each scenario “at least 5 times”, and its measurement post suggests five to ten, reading the distribution rather than one result (developer.microsoft.com).

The construct worth knowing next is pass^k, introduced in the τ-bench work (Yao et al., ICLR 2025): a task counts as successful only if the agent succeeds on all k attempts, rather than on any one of them. It matters because production users get the whole distribution of runs, not the best run, and because a workflow with an aggressive retry loop can post a strong “any attempt” number while being unpleasant to depend on.

The third construct is perturbation. ReliabilityBench varies task perturbations and injected tool or API failures and reports a reliability surface rather than a single scalar. The practical version for a product team is fault injection: simulate timeouts, rate limits, partial responses, and schema drift, and measure whether the workflow recovers or merely repeats. You can only measure that if the product designs for recovery in the first place, since there is no route to measure otherwise.

Cost of the path

Steps, tokens, wall-clock time, task cost, and human interventions per completed task (the one most often left out).

Task cost is the money it takes to complete one task with a given setup, counting every token at its price rather than only the token count, and reported next to quality every time. Microsoft’s primer gives the reason with one of its own measurements: on upgrades of its SharePoint Framework projects in GitHub Copilot Chat, a newer model with 33% lower per-token pricing cost 3.7 times more per run, across 3 scenarios and 15 runs per model. That is Microsoft’s measurement of its own scenarios on one profile, not a general result, but the point carries over: a cheaper token does not make a cheaper task.

Intervention count is the AX-specific member of this family, and it is often the most honest single number a team has. A workflow that completes reliably but needs a person twice per run has been supervised, not delegated. Counting interventions, and classifying what each one was for (missing context, ambiguous instruction, unsafe action, plain failure), turns a vague sense that “it still needs babysitting” into something a product decision can be based on.

What are the key metrics for agent experience?

The key metrics are the ones that answer a decision, read against a baseline on a stated setup. This table puts them in their families.

None of these is a score for AX as a whole. Each is a reading of one construct, and each moves with the setup it was measured on, which is what the next section deals with.

How do you know a change helped?

Compare it against a bare baseline. Microsoft’s Agent Experience series supplies the vocabulary, and it fits the families above without replacing them.

• Bare baseline: the same tasks run with no agent extensions loaded, only the harness and the model. It is what every change is compared against.
• Agent extensions: what you put in front of the agent to shape its behaviour, such as skills, MCP servers, instruction files and custom agents.
• Lift and drag: against the bare baseline, a change produces lift if outcomes improve and drag if they stay the same or get worse. Microsoft adds a third verdict: lift that is too expensive to keep.
• Agent profile: the operating system, the harness, the model and its settings, and the extensions loaded. In Microsoft’s words, “A result belongs to the profile it was measured with.”

All four definitions are from Microsoft’s 6 October 2026 primer. Its measurement post adds the method: hold the model, the harness, the prompt and the workspace fixed, so the only variable is the change (developer.microsoft.com). A comparison then has six steps:

1. Write real tasks in your users’ words. Include some that do not name your product, to test propensity, and some that do, to test efficacy.
2. Fix one agent profile and record it. Choose the setup most of your users have. Microsoft reports that in its evaluations the operating system and the harness produced the widest variance of the hidden variables it examined (developer.microsoft.com).
3. Run the bare baseline. Each task at least five times, with the run record below.
4. Make one change. A rewritten page, a clearer error, a tool description, an extension. One at a time, so the result has a cause.
5. Run the same tasks again on the same profile, the same number of times.
6. Compare outcome and cost together, and call the result lift, drag, or lift that costs too much to keep. Scope the conclusion to that profile, and run again when the profile changes.

Microsoft’s own results show why you need the comparison and cannot rely on intuition. In one, a documentation tip pointing agents to the right tool got 1 of 5 runs to use it, while a warning that named the failing approach got 5 of 5; in another, a JSON input mode added to a command-line tool for agents cost 4 to 11 times more per task than plain arguments, over 5 runs per profile. Both are Microsoft’s measurements, on its own scenarios and profiles, and the primer says each result “holds for the scenarios and agent profile we measured.” Treat them as reasons to test your own change, not as results you should expect.

A worked example

The numbers below are invented to show the arithmetic. They are not a measurement.

A travel site wants agents to answer “cancel my booking and tell me what it will cost”. Its cancellation fees sit in a table that only renders after a click. The team writes four tasks that name the site, to test efficacy, and two that do not (“I need to cancel a hotel I booked last week”), to test propensity. It fixes one profile: a stated operating system, one agent app at a recorded version, one model with recorded settings, and no extensions. That is the bare baseline.

Baseline, five runs per task. On the four efficacy tasks, twenty runs in all: 9 succeed, 6 end in partial progress (the agent found the booking but stopped without a fee to report), 4 are refused correctly because the agent could not see the fee, and 1 is unsafe (it cancelled without stating the fee). There are 11 human interventions across the 9 successes. On the two propensity tasks, the agent chose the site in 3 of 10 runs.

The change: state the fees in plain text on the page and in its Markdown version. Same tasks, same profile, five runs each. On the efficacy tasks: 16 succeed, 2 partial, 2 refused, 0 unsafe, and 4 interventions across the 16 successes. Task cost per success falls slightly, because fewer runs wander. That is lift on efficacy, on the terminal-state mix and on interventions per completed task, at no extra cost, and it holds only for that profile until the team runs it on another one. The propensity tasks still chose the site in 3 of 10 runs: the change did nothing for propensity, which is a separate problem with a separate fix.

What has to be instrumented first?

Almost every metric above is a query over a single artefact: the run record. If it is written badly, no later analysis can recover the information.

A run record that supports AX measurement carries, at minimum:

• A stable task identity: the same job, recognisably the same across runs, so a denominator exists.
• The agent profile: operating system, harness and its version, model and its settings, and the extensions loaded with their versions. Without it, two runs cannot be compared and a result cannot be scoped.
• The tool calls in order, with inputs and outcomes, so selection and input formation are inspectable rather than inferred.
• Every attempt, with its trigger and what changed. Retries are product history: without trigger you cannot tell an automatic backoff from a person clicking the button again, and without a record of what changed since the previous attempt you cannot tell a recovery from a repetition.
• Each human intervention, with what it was for.
• Tokens and their price at the time of the run, so task cost can be computed later.
• A terminal state that distinguishes four things, not two: success, partial progress, refused, and unsafe. Collapsing refused into failure is the most common way a run record ruins its own usefulness. A correct refusal and a broken workflow are different events, and a metric that cannot separate them will punish the safe behaviour.

Two disciplines apply to the record itself. Redact at write time rather than at analysis time, and keep only what someone debugging at an awkward hour would actually need; a fuller log is not automatically better governance. The AX review treats this record as its primary evidence, which gives you a test of whether yours is adequate: if a review cannot be run from it, it is not instrumented yet.

Can you measure the experience by asking the agent?

It is a real technique, with a real limitation.

The approach evaluates a completed session’s transcript and derives dimensions of the experience the agent encountered. DX’s Agent Experience Score, documented publicly for coding-agent sessions, scores three constructs, Requirements (“whether the goal was clear and the initial context helped the agent begin”), Steering (“whether follow-up input helped guide progress”), and Scope (“whether the output stayed aligned to a useful task boundary”), each on a 1–5 scale, averaged into an overall figure. The evaluation is performed after the session by a separate model reading a scrubbed transcript, not by the coding agent itself, and sessions with insufficient evidence or unreadable transcripts can go unevaluated.

What this family measures is hard to get any other way: friction that only the thing doing the work can see. A human developer needs a survey to report that a repository was hard to navigate, but the transcript already contains it.

The limitation applies to every judge-based measurement and is structural, not a flaw in any particular implementation: the judge becomes part of the instrument. A change in the judge model, the prompt, or the scrubbing rules moves the score without anything about the product changing. Treat a transcript-derived score as a signal that travels with its own version number, compare it only against itself over time, and expect it to answer “where did friction appear?” rather than “how good is this?”

The scope matters too. This construct is defined over coding-agent sessions inside a repository: the inward, developer-tooling sense of AX. The outward sense, where an agent works through a product on behalf of a customer, is a different population and would need its own definition of what a session even is. AX compared with UX, DX, and GEO works through why those two senses of the term keep getting conflated.

What claims can these measurements not support?

Write these down before anyone builds a slide:

• Not a comparison to a public benchmark. A benchmark number is model plus harness plus environment version plus verifier. Several suites state that their own releases are not comparable across versions.
• Not a claim about a model. If the harness changed, the measurement moved for reasons that have nothing to do with the model or the product.
• Not a result outside its agent profile. A lift measured on one operating system, harness and model is a hypothesis everywhere else.
• Not a prediction from a readiness score. Declared surfaces are a precondition, not a forecast.
• Not a stable number across judge changes, wherever a judge is in the loop.
• Not a substitute for the qualitative review. Metrics tell you that something regressed. They rarely tell you why, and the why is where the product change lives.

Where to start

One workflow, one construct, one week. Choose the workflow people actually want delegated, and the agent profile most of them use. Write the run record described above, even by hand, for five runs of each task before a change and five after it. Count three things: how many terminal states were success versus partial versus refused versus unsafe, how many human interventions each run needed, and what each completed task cost.

That is a small measurement, but it is already enough to settle a disagreement. Then make it last: the evaluation loop is the version of this that keeps running alongside a roadmap, because it turns a recurring failure into a durable check instead of a recurring conversation.

Frequently asked questions

How do you measure agent experience? By naming the construct first, then choosing an instrument that matches it, and comparing every change against a baseline. Four families cover most of it: declared surfaces (what the product has made explicit, checked by inspection), path quality (which decisions the agent made and whether they were right), reliability (whether the task succeeds across repeats and perturbations), and cost of the path (steps, tokens at their price, latency, and human interventions). To know whether a change helped, run the same tasks with and without it, several times each, on one recorded agent profile.

What are the key metrics for agent experience? Propensity (does the agent find and choose the right capability when the task does not name it), efficacy (does it use it correctly when it does), pass rate across repeated runs and pass^k, the mix of terminal states (success, partial, refused, unsafe), task cost counted at token prices, and human interventions per completed task. A declared-surface profile, such as a readiness rubric, sits beside them as a precondition check. Read each one against a bare baseline on a stated agent profile.

What is a bare baseline in agent evaluation? The same set of tasks run with no agent extensions loaded, only the harness and the model, so every change has something to be compared against. The term comes from Microsoft’s Agent Experience series. Against it, a change produces lift if outcomes improve and drag if they stay the same or get worse, and lift that multiplies the cost may not be worth keeping.

What should we instrument before we start measuring? The run record: stable task identity, the agent profile, tool calls in order with inputs and outcomes, every retry with its trigger and what changed since the previous attempt, each human intervention and what it was for, tokens and their price, and a terminal state that separates success, partial progress, refused, and unsafe. Almost every AX metric worth having is a query over that record.

What is the difference between an agent-readiness score and an agent-experience measurement? A readiness score describes what a product has declared (files, headers, markup, schemas, error payloads) and can be settled by inspection without running an agent at all. An agent-experience measurement describes what happened when an agent worked through the product. Readiness is a precondition you can check on a static site; experience requires runs, traces, and a definition of what counted as success.

What is agent experience analytics? Agent experience analytics is the reporting layer over an agent’s run record: how often a task completed, which tools were chosen, where retries and human interventions clustered, and what each run cost in steps, tokens and time. It is descriptive rather than evaluative, so it shows where to look without settling whether the path taken was the right one. Without a run record written at the time of the run, there is nothing for it to report on.

Can we compare our agent’s numbers to a public benchmark? Not directly. A published number reflects a model plus a harness plus an environment version plus a verifier, and benchmark releases are frequently not comparable across their own versions. Use public benchmarks to borrow measurement design (trajectory metrics, repeats, perturbation, cost reporting), not to position a product against a leaderboard.

A measurement is only useful if it changes a decision. Instrument the run record first, because every number comes from it.