Testing Idempotency and Retry Safety in Agent Tool Calls

Agent retries become dangerous when the tool changes the world. A timeout after a payment, email, reservation, or deployment may look like failure to the agent even though the side effect committed. Repeating the call can create a second real action.

Test the operation boundary, not just the HTTP response. Give each business intent a stable key, persist its lifecycle durably, inject faults before and after the side effect, and reconcile the authoritative downstream state. The core assertion is one intended effect, even when invocation is repeated.

Inventory Tools by Side-Effect Semantics

Classify each tool as pure read, observational read with audit effects, reversible mutation, irreversible mutation, or workflow trigger. Document what counts as the business effect: one charge, one notification per recipient, one reservation, one deployment request, or one state transition.

The MCP tools specification defines discovery and invocation through tool names, input schemas, and tools/call messages. That contract tells a client how to call a tool, but retry semantics remain an application responsibility. Put idempotency requirements in the trusted implementation contract and test them at the provider boundary.

Define Stable Operation Identity

Generate the key at the boundary that knows the user's business intent, not inside each network attempt. Scope it by tenant or account, operation type, and trusted caller. Bind it to a canonical hash of effect-bearing arguments such as amount, currency, recipient, and target resource.

Retries use the same key. A deliberate second operation uses a new key even when arguments match. If a key arrives with a different request hash, return a conflict and create no side effect. Otherwise, an agent could accidentally reuse a key and receive a success result for the wrong action.

Do not derive keys solely from model-written text. Prefer a server-issued operation ID attached to an approved plan step or user confirmation. Keep secrets and raw personal data out of the key; store sensitive bindings in the protected operation record.

Persist a Durable Idempotency State Machine

Use an atomic create-if-absent operation for the ledger. Useful states include pending, succeeded, failed_retryable, failed_final, and unknown. Store request hash, timestamps, tool contract version, provider correlation ID, result reference, and reconciliation history.

from dataclasses import dataclass

@dataclass(frozen=True)
class Operation:
    key: str
    request_hash: str
    status: str
    provider_id: str | None
    result_ref: str | None

def retry_action(op: Operation, incoming_hash: str) -> str:
    if op.request_hash != incoming_hash:
        return "reject_conflict"
    if op.status == "succeeded":
        return "return_recorded_result"
    if op.status in {"pending", "unknown"}:
        return "reconcile_before_retry"
    return "apply_retry_policy"

Never treat an old pending row as proof that no effect happened. The process may have crashed after provider commit and before ledger completion. Resolve ambiguity using the provider's operation lookup, a local outbox, or a manual queue for high-consequence actions.

Inject Faults at Every Commit Boundary

Map the sequence: validate, reserve key, dispatch provider request, provider commit, receive response, persist result, return to agent. Inject a crash or timeout between each pair. The most important case is after provider commit but before the caller receives success.

Also test connection reset, response corruption, duplicate delivery, process restart, lock expiry, database failover, delayed provider callback, and two workers handling the same key concurrently. Repeat with the same arguments and with a mutated payload.

For each fault, assert ledger state, provider effect count, returned operation identity, retry decision, and eventual reconciliation. A test that only checks status code can miss the duplicate that matters.

Distinguish Transport Retries From Agent Replanning

Transport middleware may retry without the agent knowing. The agent may also call the tool again after seeing an error, or regenerate a plan that expresses the same business intent with different wording. Trace all three layers and propagate one operation key through them.

The OpenAI Agents SDK tracing guide describes tool calls as traceable spans within an agent run. Attach operation key, attempt number, and sanitized provider correlation data to your own trace fields so duplicate planning can be separated from network retry behavior.

Test conversation resume and checkpoint replay. The restored agent must recover the prior operation identity rather than minting a new key for an already-approved step. If the intent genuinely changes, require a new approval or explicit replacement operation.

Verify Authoritative Side Effects

Count effects in the system that owns them. Query the payment ledger, message provider, reservation service, job queue, or deployment controller. Verify business identifiers and recipient sets, not merely request rows in the wrapper database.

Check secondary effects too. One order creation might emit duplicate emails or analytics events even when the primary database row is deduplicated. Follow the operation key through outbox events and downstream consumers, and make those consumers idempotent where delivery can repeat.

Reconciliation should end in a state the user can understand. Returning "failed" after a confirmed charge is incorrect even if the charge was unique. Return or surface the original result and keep the same operation identifier.

Test Concurrency and Scope Isolation

Launch simultaneous calls with the same key and identical payload. Exactly one worker should own execution while others wait, poll, or receive the recorded result according to the API contract. Then send the same key under a different tenant and confirm that scoping prevents cross-tenant result disclosure or unintended deduplication.

Exercise lock expiration while the provider is slow. A second worker must not assume the first action is absent simply because a lease ended. Durable operation state and provider reconciliation are stronger evidence than an in-memory mutex.

Test bulk tools carefully. Decide whether one key covers the whole batch or each item has a child key. Partial success must be explicit so a retry does not repeat successful items while skipping failed ones unpredictably.

Handle Non-Idempotent Dependencies Deliberately

If a provider accepts an idempotency token, pass your stable key or a derived provider-safe token and test its documented behavior. If it does not, serialize through a durable local ledger, store provider correlations, and query status before retrying ambiguous calls.

Some effects cannot be automatically reconciled. An email provider may accept a message and lose the response without offering lookup by client ID. For sensitive communications, an unknown state may require human review rather than a blind resend. State this limitation in the tool result so the agent cannot convert uncertainty into another call.

Compensation is not idempotency. Refunding a duplicate charge may restore money but still creates extra ledger entries, user concern, and fees. Test compensation, but keep duplicate prevention as the primary requirement.

Define Retry Budgets and Final Outcomes

Specify which errors are retryable, backoff behavior, maximum attempts, total deadline, and who owns retries. Mark numeric budgets as illustrative until measured against the dependency and business risk. Retrying validation errors or permission denials wastes capacity and can repeat unsafe intent.

Return structured states such as succeeded, failed_final, unknown_requires_reconciliation, and rejected_key_conflict. Avoid a generic exception that invites the planner to try again without context. The result should include the stable operation ID but not leak another tenant's record.

Release gates should include zero duplicate critical effects in the fault matrix, correct conflict handling, bounded pending operations, and successful reconciliation drills. Report test counts and injected boundaries rather than claiming broad exactly-once delivery.

Retry-Safety Checklist

• Classify every tool's primary and secondary side effects.
• Define one business operation independently of network attempts.
• Bind a stable key to tenant, tool version, and canonical request hash.
• Reserve the key atomically in durable storage.
• Reject key reuse when effect-bearing arguments differ.
• Inject failures before dispatch, after commit, and before result persistence.
• Test concurrency, process restart, lock expiry, and checkpoint resume.
• Audit authoritative downstream state and emitted events.
• Reconcile pending and unknown before allowing another mutation.
• Give the agent structured final and ambiguous outcomes.

Action Plan: Prove One Mutation Under Ambiguous Delivery

Select the highest-consequence mutation and diagram every persistence boundary. Add a server-issued operation key, canonical request hash, durable state machine, and provider correlation. Build a fake or sandbox provider that can commit and then drop the response, and run concurrent duplicate attempts against it.

Verify the external ledger and all downstream events after each injected fault. Then test production-shaped retries, agent resume, and manual reconciliation while preserving the same operation identity. Enable the tool only after the team can demonstrate one intended effect and explain every ambiguous state without asking the agent to guess.