An agent evaluation loop turns one failure into a standing check.

A team runs an evaluation in March, finds that the agent keeps choosing the wrong export tool, rewrites the description, and watches the problem disappear. In September the same complaint arrives through support. Nobody changed the description. Something upstream changed the tool catalogue, the March case was never kept, and the evaluation that would have caught it was a document rather than a job.

What makes a loop different from a benchmark run?

A loop has six stages, and each one hands something concrete to the next.

Task set. A fixed collection of real jobs with defined success conditions. It can start small, with representative successful jobs, boundary cases and observed failures. Review coverage as the product and its users change.

Run. Execute the task set against pinned versions of everything, several times per task rather than once.

Judge. Score the run. Deterministic checks decide facts, a rubric decides judgement, and the two stay separate.

Triage. Assign each failure to a class and to the first decision that broke, not the last error observed.

Fix. Change the product: a description, a schema, a boundary, a recovery path, a page.

Re-run. Check the change against the affected cases and the wider suite. Retain the failing case so a later recurrence can be detected under those conditions.

A benchmark run stops after the judge, so it produces a number but no improvement. Public benchmarks are open about their own numbers going stale: the tau-bench repository notes that results before a given release are not comparable with later graded results in affected domains, and OSWorld 2.0 states plainly that its results are not directly comparable to OSWorld 1.0 unless the release matrix matches. If a maintained public benchmark cannot compare its own runs across versions without care, an internal score recorded in a slide deck has no chance.

What has to stay fixed?

Everything you are not currently testing, which is a longer list than it first appears:

• model version, and any decoding settings that vary output
• harness, step budget and retry policy
• system prompt and instructions
• tool catalogue, tool descriptions and schemas
• environment or fixture version
• the judge, including its prompt and its model

There are good reasons for this. Harness-Bench (2026) exists to show why tasks should be held fixed while model backends and harnesses vary explicitly, and a companion line of work argues for disclosing harness details precisely because cross-model comparisons are unsound without them (harness disclosure, 2026). SWE-bench-Live makes the same point operationally by asking users to pin the split and release, since a continuously updated task set improves freshness at the cost of date-to-date comparability.

Record the pinned set with every run. You cannot act on a result you cannot attribute, and six months later nobody remembers which of the four things changed that week.

How should failures be classified?

By the decision that broke, and by the first decision that broke rather than the last. A run that fails at the final step because it selected the wrong capability at the start is a selection failure, and filing it under the closing error is how a problem stays open for a year.

Five classes cover most of what a product-facing loop encounters:

This decomposition comes from existing evaluation work. API-Bank (Li et al., EMNLP 2023) scores planning, retrieval and calling as separate tasks over 73 runnable APIs on the basis that the parts of tool use fail independently, and AgentBoard reports progress rate and grounding accuracy alongside success rate for the same reason. Once a failure has a class, the fix has an owner, and that is what a taxonomy gives you over a list of incidents.

Keep a sixth bucket labelled unclassified and watch it. A class you keep failing to assign is usually a real class you have not named yet.

Where do humans belong?

In the sampling, not in the whole pipeline.

A loop that requires a person to read every run will not keep up with a release schedule. A loop with no human review drifts, because the judge becomes part of the measurement and nothing checks the judge. Mind2Web 2 (2025) uses an agent-as-a-judge rubric for long-horizon search where the final answer is dynamic, and judge quality is treated there as a stated validity threat rather than an implementation detail.

Three sampling rules keep the cost bounded:

1. Review every failure in a class that has not been seen before.
2. Review a fixed share of judged passes, chosen at random, to catch the judge scoring a bad path as a good one.
3. Review every case where the deterministic checks and the rubric disagree, since one of the two is wrong and it matters which.

Step-level human feedback has a precedent in public evaluation as well: BrowserArena (2025) collects it specifically to surface failure modes that aggregate success rates hide.

What does a result change?

A finding that does not become a change stays an observation. Route each one into one of four destinations, then check that it landed:

Wording. A tool description, a schema field description, a heading, an error message. Often a small change to try when trace evidence points to ambiguous wording.

Structure. A split capability, a new page, a preflight check, a status endpoint.

Boundary. A narrower scope, an added approval, an encoded stopping condition.

Standing check. The failing case joins the task set permanently.

Record cost, step count and latency next to correctness while you do it. MCPBench (2025) measures task-completion accuracy, latency and token consumption under matched agent settings, and the same kind of measurement works for a product loop: a workflow that becomes correct and four times more expensive has not straightforwardly improved. Choose what to measure, and how covers the instrumentation side, and Evaluate the decisions in an agent workflow covers how to score the path rather than the outcome.

For the periodic wider sweep, the open agent-readiness rubric gives a job-level profile that uses the same repeated runs for its Shown level, which makes it a reasonable quarterly companion to a weekly loop.

A minimal loop template

The table below is a starting shape, not a standard. It is small enough on purpose to run by hand for the first month.

Three practices add more to that table than any extra column would. Repeat each task and look at the worst run rather than the best; the pass^k metric introduced in the tau-bench work (Yao et al., ICLR 2025) formalises the same instinct by counting a task as successful only when all k attempts succeed. Perturb the inputs, since ReliabilityBench (2026) reports a reliability surface over perturbations and injected failures rather than a single scalar. Inject faults, because you do not know whether a recovery path works until it has been exercised.

Start with a small, maintained task set. Regular reruns can reveal regressions that a one-off suite misses, but case count and cadence alone do not establish coverage.

How do you know the evaluation itself works?

Before a failed run becomes a product change, check that the judge can distinguish a correct artifact from a broken one. Microsoft’s AX Practitioner Playbook announcement, published October 9, 2026, adds a useful discipline to this loop: calibrate the measurement, check that the generated project actually works, and inspect the evidence before deciding what to fix. The following fixture turns that discipline into something small enough to review alongside a pull request.

Give the judge examples with known answers

Choose one decision your product team can verify, such as the SDK release required by a particular scenario. Create a known-good artifact and a deliberately wrong one. Add a third artifact for which the check genuinely does not apply. Write separate pass, fail and skip conditions, then score the same artifacts repeatedly. A domain owner should verify both the expected answers and the judge’s explanations. Changing a criterion until a surprising production result disappears would invalidate the comparison.

Chapter 5, on criteria, separates this calibration from the measured evaluation. Keep those records apart. Lock the criteria-set version after calibration; when the definition of success changes, create and calibrate a new version. Preserve earlier results with their original version. Pin the judge model and prompt as well, since a changed judge can change the measurement even when the criterion text stays fixed.

This original JSON is an illustrative contract, not a Microsoft schema or an executed test. The package name and releases are fictional. Expected verdicts describe the intended calibration answers; they are not measured results.

{
  "example_only": true,
  "execution_status": "not_run",
  "criteria_set": {
    "id": "ledger-sdk-choice",
    "version": "1.0.0",
    "lock_after_calibration": true,
    "criterion": {
      "id": "requested-sdk-release",
      "pass": "The dependency @example/ledger-sdk is exactly 2.4.1.",
      "fail": "The dependency exists but is any other release or range.",
      "skip": "The artifact does not use @example/ledger-sdk."
    }
  },
  "calibration_fixtures": [
    {"id": "known-good", "dependencies": {"@example/ledger-sdk": "2.4.1"}, "expected": "pass"},
    {"id": "known-bad", "dependencies": {"@example/ledger-sdk": "1.9.0"}, "expected": "fail"},
    {"id": "not-applicable", "dependencies": {}, "expected": "skip"}
  ],
  "scenario_gates": [
    {"id": "prerequisite", "requires": "Uses the requested ledger SDK"},
    {"id": "build", "after": "prerequisite", "command": "npm run build", "required_exit_code": 0}
  ],
  "run_plan": {
    "profiles": ["bare-baseline", "candidate-extension"],
    "repeats_per_profile": 5,
    "pin": ["scenario", "workspace", "model", "harness", "judge", "criteria_set"],
    "retain": ["gate-results", "verdicts", "artifacts", "tool-events", "cost"]
  },
  "trace_diagnosis": {
    "unavailable": "Loading or registration evidence confirms the extension was absent.",
    "ignored": "The extension was available, but the complete trace has no invocation.",
    "misapplied": "The extension was invoked, but its guidance was used incorrectly."
  }
}

Make the build a separate gate

The empty artifact tests the judge’s skip rule; it is not a successful response to the ledger scenario. Its prerequisite gate would fail. Report skipped criteria separately, and never turn “not applicable” into a pass. Likewise, a dependency criterion cannot establish that the project compiles. The fixture’s build command is a proposed check; the example records no build result.

Use sequential gates for prerequisite, build, tests, runtime and deployment where each applies. Stop downstream gates when an earlier one fails, and preserve that reason. A gate that did not run has not passed. Chapter 2’s evaluation model makes this separation explicit: judgments about product choices do not replace mechanical checks of working code.

Repeat to find cases worth investigating

For an extension comparison, hold the scenario, workspace, model, harness, judge and criteria fixed, and change the extension condition. Keep the bare baseline alongside the candidate and retain cost as well as verdicts. Five repeats per scenario and profile follow the playbook’s diagnostic starting point. Five runs are not statistical proof of reliability or improvement; even five matching outcomes leave uncertainty. Use them to locate repeated failures and inconsistent decisions that deserve investigation. Stronger claims need an evaluation design and sample size suited to those claims.

Use the trace to choose the next edit

A failed criterion identifies the symptom. The trajectory chapter distinguishes three useful questions: was the extension available, was it invoked, and was the information applied correctly?

• Unavailable: inspect startup, registration and loading evidence. Repairing documentation cannot make an absent tool callable.
• Ignored: confirm availability, then inspect descriptions, task vocabulary and neighbouring capabilities. No invocation alone does not explain the decision.
• Misapplied: follow the returned guidance into the generated artifacts. Check the source content, conflicting instructions and the first observable departure from the required behaviour.

Treat those labels as investigations, not diagnoses established by a score. If the trace is incomplete, record “unknown” rather than guessing. Fix the earliest supported cause, preserve the failing case, and rerun the locked comparison. Start by writing one criterion and its three calibration fixtures; do not expand the suite until that small measurement behaves as intended.

Frequently asked questions

What is an agent evaluation loop? An agent evaluation loop is a repeating cycle of six stages: a fixed task set, a run against pinned versions, a judge that scores the run, triage that assigns each failure to a class, a product change, and a re-run that checks whether the change held for the tested cases. The defining property is that every stage feeds the next and the last one feeds the first. A single scored run with no re-run is a measurement, not a loop.

Why is a one-off benchmark run not an evaluation loop? Because it cannot tell you whether anything you fixed stayed fixed. A one-off run produces a number that is valid for one model, one harness, one environment version and one afternoon. Public benchmark maintainers say as much themselves: the tau-bench repository notes that results before a given release are not comparable with later graded results in affected domains, and OSWorld 2.0 states that its scores are not directly comparable to OSWorld 1.0 unless the release matrix matches.

What should be held constant between evaluation runs? The model version and decoding settings, the harness and step budget, the system prompt and tool descriptions, the tool catalogue and schemas, the environment or fixture version, and the judge. Harness-focused evaluation work exists because the scaffold can affect the result alongside the model, so changing more than one of those at a time makes attribution harder without a design that separates those changes.

How much human review does an evaluation loop need? Enough to keep the automated judge honest, sampled rather than exhaustive. A practical pattern is to review every failure in a new class, a fixed share of judged passes, and any case where the deterministic checks and the judge disagree. Several web-agent studies collect step-level human feedback specifically because success rates hide failure modes, and the same reasoning applies at product scale.

Check whether a retained failure case still detects the defect it was designed to catch, and review what the suite leaves uncovered.