Insights

Reliable tool execution and recovery

How to engineer retries, idempotency, reconciliation, and compensation around AI-driven actions.

In brief

What should happen when a tool call may have succeeded but its response was lost?

  • Track the operation lifecycle and use idempotency across system boundaries.
  • Distinguish a retryable transport failure from a business rejection or ambiguous outcome.
  • Reconcile with the system of record before repeating an irreversible effect.

A tool call can fail in several different ways. The request may never reach the service. The service may reject it. The operation may succeed while the response is lost. An agent that treats all three conditions as the same error can create duplicate records, repeated messages, or inconsistent workflows.

Reliable execution begins by representing uncertainty about external effects. The model can decide what it wants to accomplish, while a durable execution layer tracks what was requested, what was confirmed, and what still requires reconciliation. This is a distributed-systems problem with an AI component inside it.

Model the operation lifecycle

A useful lifecycle distinguishes prepared, authorized, submitted, confirmed, rejected, and unknown outcomes. Unknown means the system lacks enough evidence to determine whether the external effect occurred. It should remain a first-class state until a reconciliation process resolves it.

Each logical operation needs an identifier independent of a single network attempt. The execution record should include the target service, normalized parameters, authorization context, attempt history, and result. Repeated transport attempts can then be associated with one intended effect.

This record belongs in durable storage. Keeping it only in the model’s conversational context makes recovery depend on a lossy narrative. After a process restart, the system should be able to resume or investigate the operation without asking the model to reconstruct events from memory.

A synthetic duplicate-order failure

Imagine an agent creates an internal equipment request. The service accepts the request, but the network connection closes before the response arrives. The agent sees a timeout and submits the same request again. If the service creates a new record each time, the organization now has two requests.

Event What actually happened What the caller knows
First submission Request created Awaiting response
Network interruption Response lost Outcome unknown
Blind retry Second request created One success received

The remedy depends on the service contract. If it supports idempotency keys, the caller can reuse the same key for retries of the same logical request. If it provides a lookup by external reference, the caller can query for the first operation. If neither exists, automated retries may need to stop for reconciliation.

AWS’s engineering guidance describes caller-provided request identifiers as a way to express intent and make retries safe. The key must identify the intended operation consistently, and the receiving service must enforce the corresponding semantics. Making retries safe with idempotent APIs.

Idempotency is a contract across boundaries

Generating a unique string does not by itself prevent duplicate effects. The receiving service must associate that key with the operation and its result. The caller must also avoid reusing the key for a different request. A changed quantity, recipient, or resource should trigger an explicit decision about whether this is a revision or a new operation.

Retention matters. A provider may remember keys for a limited period. Retrying after that interval can create a new effect. The execution layer should record the provider’s documented behavior and apply a retry deadline compatible with it.

Concurrency also matters. Two workers may attempt the same task simultaneously. A durable claim or lock can reduce duplicate execution, while provider-side idempotency protects the boundary if both requests still escape. These mechanisms complement each other because they cover different failure modes.

Classify failures before retrying

Transient capacity errors and temporary connection failures may justify retries. Invalid parameters, missing permissions, and unsupported operations usually require a changed request or human intervention. A timeout after submission requires special care because the external outcome may be unknown.

Retry policy should specify which errors qualify, the maximum number of attempts, the total time budget, and backoff behavior. Jitter helps avoid synchronized retry bursts. A global budget prevents one struggling dependency from consuming all available workers through repeated attempts.

The model should receive structured error categories and safe next actions. It can repair a malformed business parameter when the evidence supports a correction. It should not invent a new identifier or recipient merely to make a rejected operation succeed. Recovery must preserve the original intent and authority.

Reconcile with the system of record

Reconciliation asks the external system what actually happened. It may query an operation status, retrieve a resource by correlation identifier, or inspect a transaction ledger. The answer should update the durable operation record and determine whether another attempt is safe.

Some services are eventually consistent. An immediate lookup can return no result even though the write succeeded. The reconciliation policy must account for the service’s visibility delay. Absence at one instant is weaker evidence than a definitive rejection.

A human review path is appropriate when the consequences are material and the available interfaces cannot resolve uncertainty. The reviewer should see the exact operation, attempts, and evidence gathered. This keeps manual recovery bounded and avoids asking someone to infer the entire history from scattered logs.

A reconciliation queue should have its own service objective. Unknown outcomes that remain unresolved for days can undermine otherwise successful automation. Track their age, assign ownership, and escalate according to consequence. The user interface should show that an operation is being checked and should prevent a second submission that would create the same uncertainty. Once the result is confirmed, update both the operational record and the user-visible task status.

Coordinate several operations deliberately

A workflow may create a record, attach a document, update a status, and notify a stakeholder. These steps may span services without a shared transaction. Partial completion is therefore expected, and the workflow needs a policy for each interruption point.

Compensating actions can reverse some effects, such as canceling a draft reservation. Others cannot be fully undone, such as an external message already delivered. The plan should identify which steps are reversible and place irreversible actions after prerequisites are confirmed when the workflow permits it.

Compensation is itself an operation that can fail. It requires authorization, tracking, and reconciliation. Describing a workflow as transactional does not make cross-service rollback automatic. A clear state machine is often more useful than a broad promise of automatic recovery.

Measure completion and recovery quality

A tool success rate measures only part of the system. Track confirmed business outcomes, unknown outcomes, duplicate prevention events, retry counts, reconciliation time, and manual recovery effort. These measures reveal whether a nominally reliable tool creates operational work elsewhere.

Service objectives should reflect user-visible behavior. Google’s SRE guidance distinguishes indicators from the objectives set for them. For an agent workflow, an indicator might be the proportion of valid requests completed once, with a confirmed result, within the agreed deadline. Service Level Objectives.

Separate dependency failure from model failure while preserving the end-to-end outcome. Users experience the complete workflow. Engineering teams need the decomposition to improve it. Both views are necessary for accurate prioritization.

Test interruptions at the awkward moments

Fault injection should target the boundaries where uncertainty appears. Terminate a worker after submission but before persistence. Lose a response after the external write. Run two workers against the same task. Expire an authorization during a retry. Delay a status query beyond the usual consistency window.

The expected behavior should be specified before the test. Some cases should complete automatically, some should remain pending, and some should require review. A test passes when the system reaches the correct state without unauthorized or duplicate effects.

Reliable tool execution makes an agent’s intentions operationally meaningful. The surrounding architecture provides durable state, controlled retries, and evidence of what happened. With that foundation, recovery becomes a designed part of the workflow, and users can distinguish progress, uncertainty, and completion with confidence.

Sources & further reading

  1. Making retries safe with idempotent APIs ↗
  2. Service Level Objectives ↗