Record agent retries as product history.

Retries are normal in agent-facing work. Context changes, an external service is unavailable, a tool input is incomplete, or a step takes a wrong turn. The trouble starts when the last outcome is treated as the whole story.

Why does hiding retries create false confidence?

Because a last-write-wins record turns a distribution into a single point.

Public evaluation work already accounts for this. The τ-bench work (Yao et al., ICLR 2025) evaluates reliability with pass^k (a task counts as a success only if the agent succeeds on all k attempts) because a single lucky run is not evidence of a dependable workflow. Reliability-focused work in 2026 goes further, varying task perturbations and injecting tool and API failures to report a reliability surface rather than one scalar. Benchmark methodology papers make the same warning: a score reflects a model plus a harness plus a retry policy plus a step budget, so retries are part of what is being measured whether or not anyone says so.

Your product has a retry policy too. If it is invisible, you are reporting a best-run number to yourself.

There is a second, less obvious cost. If a person cannot see that a task took four attempts, they cannot tell the difference between a system that is working and a system that is being carried by its retry loop. That is a trust problem as well as a metrics problem, because trust in delegated work depends on being able to reconstruct what happened.

Keep each attempt visible

Show the latest result first, then make earlier attempts easy to find. A person should be able to see what changed, whether the retry was automatic or deliberate, and whether the same problem is recurring.

A pattern that works is summary in front, history one click behind. A result line reads: “Completed: 4 attempts, 1 human intervention.” The detail behind it holds each attempt with what was tried and what came back.

An attempt record needs a handful of fields to be useful. The shape below is illustrative:

{
  "attempt": 3,
  "at": "2026-08-30T14:22:10Z",
  "trigger": "automatic",
  "changed_since_previous": "narrowed date range to a single month",
  "step": "call_tool:export_ledger",
  "failure_class": "external_dependency_timeout",
  "outcome": "failed"
}

trigger and changed_since_previous are the two fields most often omitted and most often needed. Without trigger, you cannot tell an automatic backoff from a person clicking the button again. Without changed_since_previous, you cannot tell a recovery from a repetition.

Treat the record like any other trace. Redact at write time and keep only what a support engineer would actually need at an awkward hour, because a fuller log is not automatically better governance. NIST’s AI Risk Management Framework frames trustworthy AI as balancing accountability and transparency against security and privacy in context, rather than maximising any one of them.

Distinguish recovery from clean success

When work succeeds only after intervention or repetition, label it accordingly. That does not make the workflow a failure. It points at something to improve: the context, a preflight check, the tool contract, or the route for recovery.

Four outcome labels are usually enough. The names below are a working vocabulary offered here, not an industry standard:

The distinction between self-recovered and repeated is the one worth enforcing in code. A retry that reissues the identical call with identical arguments is unlikely to succeed for a different reason unless the world changed underneath it. Blocking it and asking for a change instead (a narrower scope, a refreshed credential, a different tool) turns a loop into a decision. That is a design recommendation argued from the mechanism, not an established practice.

Standards already cover some of these changes. When a tool needs more human input mid-task, the MCP tools specification (2026-07-28) supports multi-round-trip flows rather than a bare failure. When a task becomes sensitive partway through, OAuth step-up authentication (RFC 9470) is the designed answer, and short-lived tokens with rotation reduce the blast radius when one leaks. A retry that refreshes an expired token is a recovery; a retry that hammers an expired token is a bug.

Look for repeatable failure patterns

One failed attempt can be noise. The same failure across similar tasks is a product signal. Group attempts by the decision point that went wrong (selection, input formation, permissions, missing context, or an external dependency) rather than treating every error as unique.

The five classes follow how the tool-use literature decomposes failure: API-Bank (Li et al., EMNLP 2023) scores planning, retrieval, and calling as separate tasks over 73 runnable APIs on the basis that they fail independently, and the recurring failure modes named across the tool-learning literature are wrong-tool selection, malformed parameters, tool bypass, and documentation drift.

Each class points somewhere different:

• Selection: the agent chose a neighbouring capability. Fix the wording. See Write tool descriptions an agent can act on and Agent tool catalogues need a pruning strategy.
• Input formation: arguments malformed or incomplete. Fix the schema: enums, formats, identifier sources, required fields.
• Permissions: blocked, or escalated silently. Fix the boundary and the approval. See Human approval workflows for AI agents.
• Missing context: the agent guessed at something the product knew. Fix the page, the documentation, or the handoff. See An agent context handoff should carry the reason for the work.
• External dependency: a timeout, a rate limit, a partial response, a changed contract. Fix the backoff, the preflight check, or the fallback path.

Tag at the first failing step, not the last. A run that fails at the end because it selected the wrong tool at the start is a selection failure, and filing it under the final error is how the same problem stays open for months.

Let history guide the next design change

Retry history is most useful when it feeds a small improvement loop. Choose the recurring pattern, change the surface around it, and watch whether the next set of attempts becomes clearer and more stable.

Four numbers are enough to run that loop without building a dashboard nobody reads:

1. Attempts per completed task, as a distribution rather than a mean. The tail is the product problem.
2. Share of completions that were assisted. This is your true human-in-the-loop cost.
3. Top failure class at the first failing step. This is your next piece of work.
4. Repeat rate for a fixed pattern after you changed it. This is the only number that tells you whether the change worked.

Then turn the recurring case into a standing check so a fixed problem cannot quietly return. That is the mechanism described in An agent evaluation loop turns one failure into a standing check, and the diagnostic pass that finds these patterns is in Run an Agent Experience review. When the recurring cause is a missing idempotency key, an error that names no next step or an exit code that reports success for a command that changed nothing, the recovery patterns name the usual fix.

What not to do with retry data

Do not turn retry count into a target. Optimising the number directly produces silent swallowing: failures that stop being recorded rather than stop happening.

Do not treat every retry as a defect. External services fail; a workflow that recovers from a transient timeout is working as designed. What tells you something is the pattern across events.

Do not compare your attempt counts to a public benchmark’s. Retry policy, step budget, and harness are part of what those numbers measure, and benchmark releases are frequently not comparable even with their own earlier versions.

Do not keep the history where nobody looks. A trace store that requires an engineer and a query language is not product history; it is archaeology. The people who need this are the ones deciding what to build next.

Frequently asked questions

Should an AI agent’s retries be shown to users? Show the outcome first and make the attempts reachable. Most users do not need every attempt in the foreground, but they do need to know whether a result came on the first try or the fourth, whether a person intervened, and whether the same problem is recurring. Hiding attempts entirely makes a product look more reliable than it is.

How many retries are too many? There is no universal number, and a fixed cap is the wrong control on its own. A more useful design rule is that each retry should carry new information: a different input, a refreshed token, a narrower scope. A retry that repeats the same call with the same arguments is unlikely to succeed for a different reason, so it is a reasonable candidate to block.

What is the difference between a retry and a recovery? A retry repeats an attempt. A recovery changes something first: the input, the scope, the credential, the tool, or the path. Recording them identically is what produces false confidence, because a workflow that only succeeds after intervention is a different product from one that succeeds directly.

How should retries be measured? Measure the distribution, not the final state. Public evaluation practice uses pass^k (success on all k attempts rather than any one) for exactly this reason (Yao et al., ICLR 2025), and reliability work varies perturbations and injected tool failures rather than reporting a single score. Track attempts per completed task, the share of tasks needing intervention, and the failure class at the first failing step.

Reliable systems retry too. What matters is whether you can understand and reduce the work the retries create.