# Record agent retries as product history.

> Why hiding an AI agent's retry history creates false confidence, and how to record attempts so a team can tell recovery from clean success.

Published: 2026-09-18  
Updated: 2026-09-18  
Changelog: 2026-09-18: Retitled away from the X-not-Y headline template while keeping retries and product history in the title, H1 and card; body unchanged.  
Canonical: https://agentexperience.tech/insights/retries-are-product-history/  
Tags: reliability, recovery, evaluation, observability

**Summary**

A workflow that succeeds on the fourth attempt has told you something the fourth attempt alone will not. Keep every attempt in the record, show the latest result first with earlier ones one click away, and label recovery differently from clean success. Group failures by the decision point that broke (selection, input formation, permissions, missing context, or an external dependency) so the pattern becomes a product change rather than a support anecdote.

1. Retries are product history, and a record that keeps only the last write turns a distribution into a single point and reports a best-run number.
2. For each attempt, record what triggered it and what changed since the previous one.
3. Label outcomes as clean, self-recovered, assisted or failed, so that success after repetition or intervention is never reported as clean.
4. A retry that reissues the identical call with identical arguments is unlikely to succeed for a different reason, so block it and ask for a change instead.
5. Group failures by the decision point that broke, tagged at the first failing step: selection, input formation, permissions, missing context or an external dependency.
6. Track attempts per completed task as a distribution, the share of assisted completions, the top failure class and the repeat rate after a fix, and never make retry count a target.

A workflow that succeeds after several attempts has told you something important. Hiding that history makes the product look more reliable than it is.

Retries are normal in agent-facing work. Context changes, an external service is unavailable, a tool input is incomplete, or a step takes a wrong turn. The trouble starts when the last outcome is treated as the whole story.

## Why does hiding retries create false confidence?

Because a last-write-wins record turns a distribution into a single point.

Public evaluation work already accounts for this. The τ-bench work (Yao et al., ICLR 2025) evaluates reliability with `pass^k` (a task counts as a success only if the agent succeeds on *all* k attempts) because a single lucky run is not evidence of a dependable workflow. Reliability-focused work in 2026 goes further, varying task perturbations and injecting tool and API failures to report a reliability surface rather than one scalar. Benchmark methodology papers make the same warning: a score reflects a model plus a harness plus a retry policy plus a step budget, so retries are part of what is being measured whether or not anyone says so.

Your product has a retry policy too. If it is invisible, you are reporting a best-run number to yourself.

There is a second, less obvious cost. If a person cannot see that a task took four attempts, they cannot tell the difference between a system that is working and a system that is being carried by its retry loop. That is a trust problem as well as a metrics problem, because trust in delegated work depends on being able to reconstruct what happened.

## Keep each attempt visible

Show the latest result first, then make earlier attempts easy to find. A person should be able to see what changed, whether the retry was automatic or deliberate, and whether the same problem is recurring.

A pattern that works is *summary in front, history one click behind*. A result line reads: “Completed: 4 attempts, 1 human intervention.” The detail behind it holds each attempt with what was tried and what came back.

An attempt record needs a handful of fields to be useful. The shape below is illustrative:

```json
{
  "attempt": 3,
  "at": "2026-08-30T14:22:10Z",
  "trigger": "automatic",
  "changed_since_previous": "narrowed date range to a single month",
  "step": "call_tool:export_ledger",
  "failure_class": "external_dependency_timeout",
  "outcome": "failed"
}
```

`trigger` and `changed_since_previous` are the two fields most often omitted and most often needed. Without `trigger`, you cannot tell an automatic backoff from a person clicking the button again. Without `changed_since_previous`, you cannot tell a recovery from a repetition.

Treat the record like any other trace. Redact at write time and keep only what a support engineer would actually need at an awkward hour, because a fuller log is not automatically better governance. NIST’s AI Risk Management Framework frames trustworthy AI as balancing accountability and transparency against security and privacy in context, rather than maximising any one of them.

## Distinguish recovery from clean success

When work succeeds only after intervention or repetition, label it accordingly. That does not make the workflow a failure. It points at something to improve: the context, a preflight check, the tool contract, or the route for recovery.

Four outcome labels are usually enough. The names below are a working vocabulary offered here, not an industry standard:

| Label | Meaning | What it suggests |
| --- | --- | --- |
| **Clean** | Succeeded on the first attempt | Nothing to fix here |
| **Self-recovered** | Succeeded after the agent changed something | The first attempt lacked information the product could supply |
| **Assisted** | Succeeded after a person intervened | A missing handoff, boundary, or context surface |
| **Failed** | Did not succeed | Recovery path or stopping condition to design |

The distinction between *self-recovered* and *repeated* is the one worth enforcing in code. A retry that reissues the identical call with identical arguments is unlikely to succeed for a different reason unless the world changed underneath it. Blocking it and asking for a change instead (a narrower scope, a refreshed credential, a different tool) turns a loop into a decision. That is a design recommendation argued from the mechanism, not an established practice.

![How an attempt is recorded, from top to bottom. An attempt that succeeds first time, with no help, is clean. A failed attempt is recorded with its trigger, step and failure class. The next question is what changed since the last attempt: if nothing did, the repeat is blocked and the agent asks for a change. Otherwise it tries again with a new input, scope or tool, or a person steps in. The outcome, highlighted, is labelled self-recovered, assisted or failed, and never clean.](https://agentexperience.tech/diagrams/retry-or-recovery.svg)

*Figure: Attempts recorded so that recovery is never reported as clean success.*

Standards already cover some of these changes. When a tool needs more human input mid-task, the MCP tools specification (2026-07-28) supports multi-round-trip flows rather than a bare failure. When a task becomes sensitive partway through, OAuth step-up authentication (RFC 9470) is the designed answer, and short-lived tokens with rotation reduce the blast radius when one leaks. A retry that refreshes an expired token is a recovery; a retry that hammers an expired token is a bug.

## Look for repeatable failure patterns

One failed attempt can be noise. The same failure across similar tasks is a product signal. Group attempts by the decision point that went wrong (selection, input formation, permissions, missing context, or an external dependency) rather than treating every error as unique.

The five classes follow how the tool-use literature decomposes failure: API-Bank (Li et al., EMNLP 2023) scores planning, retrieval, and calling as separate tasks over 73 runnable APIs on the basis that they fail independently, and the recurring failure modes named across the tool-learning literature are wrong-tool selection, malformed parameters, tool bypass, and documentation drift.

Each class points somewhere different:

- **Selection:** the agent chose a neighbouring capability. Fix the wording. See [Write tool descriptions an agent can act on](/insights/tool-descriptions/) and [Agent tool catalogues need a pruning strategy](/insights/agent-tool-catalogs/).
- **Input formation:** arguments malformed or incomplete. Fix the schema: enums, formats, identifier sources, required fields.
- **Permissions:** blocked, or escalated silently. Fix the boundary and the approval. See [Human approval workflows for AI agents](/insights/approval-is-a-workflow/).
- **Missing context:** the agent guessed at something the product knew. Fix the page, the documentation, or the handoff. See [An agent context handoff should carry the reason for the work](/insights/preserve-context-through-handoffs/).
- **External dependency:** a timeout, a rate limit, a partial response, a changed contract. Fix the backoff, the preflight check, or the fallback path.

Tag at the *first* failing step, not the last. A run that fails at the end because it selected the wrong tool at the start is a selection failure, and filing it under the final error is how the same problem stays open for months.

## Let history guide the next design change

Retry history is most useful when it feeds a small improvement loop. Choose the recurring pattern, change the surface around it, and watch whether the next set of attempts becomes clearer and more stable.

![Retry history as a design loop in five steps, from top to bottom. 1. Record each attempt: its trigger, what changed, the step and the failure class. 2. Group the failures by the first failing step, not the last. 3. Change the surface, highlighted: the wording, the schema, the boundary, the context or the fallback. 4. Watch the repeat rate: did the same failure come back? 5. Keep a standing check so the fixed problem cannot quietly return. A dashed line goes back to the first step for the next pattern.](https://agentexperience.tech/diagrams/retry-improvement-loop.svg)

*Figure: Retry history feeding one design change at a time.*

Four numbers are enough to run that loop without building a dashboard nobody reads:

1. **Attempts per completed task**, as a distribution rather than a mean. The tail is the product problem.
2. **Share of completions that were assisted.** This is your true human-in-the-loop cost.
3. **Top failure class at the first failing step.** This is your next piece of work.
4. **Repeat rate for a fixed pattern after you changed it.** This is the only number that tells you whether the change worked.

Then turn the recurring case into a standing check so a fixed problem cannot quietly return. That is the mechanism described in [An agent evaluation loop turns one failure into a standing check](/insights/build-a-reliable-agent-evaluation-loop/), and the diagnostic pass that finds these patterns is in [Run an Agent Experience review](/insights/run-an-ax-review/). When the recurring cause is a missing idempotency key, an error that names no next step or an exit code that reports success for a command that changed nothing, the [recovery patterns](/insights/approval-handoff-recovery-patterns/#recovery-patterns) name the usual fix.

## What not to do with retry data

**Do not turn retry count into a target.** Optimising the number directly produces silent swallowing: failures that stop being recorded rather than stop happening.

**Do not treat every retry as a defect.** External services fail; a workflow that recovers from a transient timeout is working as designed. What tells you something is the pattern across events.

**Do not compare your attempt counts to a public benchmark’s.** Retry policy, step budget, and harness are part of what those numbers measure, and benchmark releases are frequently not comparable even with their own earlier versions.

**Do not keep the history where nobody looks.** A trace store that requires an engineer and a query language is not product history; it is archaeology. The people who need this are the ones deciding what to build next.

## Frequently asked questions

**Should an AI agent’s retries be shown to users?**
Show the outcome first and make the attempts reachable. Most users do not need every attempt in the foreground, but they do need to know whether a result came on the first try or the fourth, whether a person intervened, and whether the same problem is recurring. Hiding attempts entirely makes a product look more reliable than it is.

**How many retries are too many?**
There is no universal number, and a fixed cap is the wrong control on its own. A more useful design rule is that each retry should carry new information: a different input, a refreshed token, a narrower scope. A retry that repeats the same call with the same arguments is unlikely to succeed for a different reason, so it is a reasonable candidate to block.

**What is the difference between a retry and a recovery?**
A retry repeats an attempt. A recovery changes something first: the input, the scope, the credential, the tool, or the path. Recording them identically is what produces false confidence, because a workflow that only succeeds after intervention is a different product from one that succeeds directly.

**How should retries be measured?**
Measure the distribution, not the final state. Public evaluation practice uses `pass^k` (success on all k attempts rather than any one) for exactly this reason (Yao et al., ICLR 2025), and reliability work varies perturbations and injected tool failures rather than reporting a single score. Track attempts per completed task, the share of tasks needing intervention, and the failure class at the first failing step.

Reliable systems retry too. What matters is whether you can understand and reduce the work the retries create.