Debugging Duplicate Agent Side Effects After LangGraph Resume

A duplicate email, payment, ticket, or database mutation after LangGraph resume is not merely a graph-control bug. It is a distributed commit failure until evidence proves otherwise. The checkpoint store, worker, tool adapter, and external service each have independent durability and retry behavior; the duplicate appears where their records disagree about whether one logical action completed.

Preserve the thread and checkpoint history before rerunning anything. Then correlate every attempted graph node with a stable business operation ID and the external provider's request record. The central question is not "Did this node run twice?" but "How many execution attempts represented the same intended effect, and which component authorized each attempt?"

Contain the incident and preserve state

Disable automatic resume for the affected action type or route it to approval while the incident is active. If the side effect is financial, destructive, or externally visible, stop retries at the queue and tool-adapter layers as well as in the graph. A graph-level switch does not prevent a delivery worker from retrying an already published command.

Snapshot the checkpoint rows, graph definition identifier, thread configuration, pending tasks, worker leases, queue delivery metadata, tool-call arguments, idempotency headers, external response, and resulting domain records. Record clocks from each system and prefer causal sequence identifiers over timestamp ordering. Clock skew can make the external action appear to happen after the resumed node even when it preceded the crash.

Do not delete the duplicate during evidence collection without preserving its identifiers and state transitions. Reconciliation is necessary, but the original records are the only reliable way to locate the commit gap.

Build a root-cause tree around commit boundaries

Under graph state, investigate the wrong thread_id, resume from an older checkpoint, changed graph topology, state reducer behavior, or a side-effect result that was never persisted. Under worker execution, investigate two workers holding the same logical task, expired leases, process termination, and manual reruns. Under tool transport, inspect client retries, proxy retries, timeout ambiguity, and reconnect behavior.

Under external service, inspect ignored or regenerated idempotency keys, duplicate acceptance, delayed responses, and a successful action followed by an error response. Under application design, inspect check-then-act flags, unstable operation IDs, side effects embedded in model reasoning loops, and compensation mistaken for deduplication. Under operations, inspect an operator resuming from a copied thread or replaying a dead-letter message without the original business key.

Each hypothesis needs a join. The graph run ID alone is insufficient because a resume can create another execution attempt for the same business action. Use a logical operation ID that survives process, node, and transport retries.

Reconstruct checkpoints and attempts

Order checkpoint snapshots for the affected thread and identify the last durable state before interruption. The LangGraph persistence documentation explains that checkpointers persist thread-scoped graph state and that the thread_id selects that scope. Confirm that the original and resumed invocations used the intended same thread, then inspect what the durable state said about the action.

Create one timeline row per attempt: graph run, checkpoint or task reference, node name, worker, attempt number, logical operation ID, dispatch time, external request ID, response category, state update, and next checkpoint. Mark unknowns rather than inferring success from a missing error.

The revealing pattern is often: checkpoint says "ready," external service records success, worker receives no definitive response, and no later checkpoint records completion. On resume, the graph legitimately sees "ready" and dispatches again. That does not absolve the graph design; it identifies the non-atomic boundary that must be protected.

Distinguish replay from independent retries

Count attempts at every layer. A node may execute once while an HTTP library retries twice. A tool adapter may dispatch once while an external queue redelivers. Two graph workers may each execute the same task. Conversely, a resumed node can appear twice in traces while an external idempotency key collapses both requests into one effect.

Tag retries with both logical_operation_id and attempt_id. A new attempt ID is expected for another try; a new logical ID for unchanged intent is usually the defect. Inspect whether model-generated arguments changed materially. Two emails to the same recipient with different generated text may still represent one business action and need the same deduplication scope.

Avoid using trace span IDs as business idempotency keys. Spans identify executions and normally change when work is retried. The stable key should come from domain intent, such as tenant + invoice + send-receipt, with explicit behavior when the intent itself changes.

Inject failures at the side-effect boundary

Create deterministic failpoints around the tool adapter: before durable reservation, after reservation, before external dispatch, after dispatch but before response handling, after response but before result persistence, and after graph-state update. Resume the same thread after each injected failure and observe business effects, not just graph completion.

Also force ambiguous network outcomes. Have the fake provider commit the action and close the connection before the client receives a response. Then test a timeout before the provider commits. The adapter must query by idempotency key or safely retry with the same key; it cannot distinguish those cases from transport status alone.

The official LangGraph testing guide shows compiling graphs with fresh checkpointers and using persistence to establish partial execution state. Extend that pattern with a fake external service and fault injection so tests can resume from the exact pre-action boundary.

Repair with a durable operation ledger

Put an atomic idempotency decision in a durable store controlled by the application, even when the provider also supports idempotency. Insert or claim the logical operation before dispatch. Repeated attempts with the same key should return the recorded outcome, wait for an active claim, or enter reconciliation if the outcome is unknown. They should not create a fresh effect by default.

This Python sketch is illustrative and omits database-specific locking details:

async def execute_once(store, provider, operation_id: str, payload: dict):
    record = await store.claim(operation_id, payload_hash(payload))

    if record.status == "succeeded":
        return record.result
    if record.status == "in_flight" and not record.claimed_by_me:
        return await store.wait_for_result(operation_id)

    try:
        result = await provider.send(payload, idempotency_key=operation_id)
        await store.mark_succeeded(operation_id, result.external_id, result)
        return result
    except AmbiguousDelivery:
        await store.mark_reconciliation_required(operation_id)
        raise

The store must enforce uniqueness on the operation ID and verify that repeated use carries the same semantic payload. An outbox is another strong pattern: commit intended work with domain state, publish it reliably, and let a consumer deduplicate. This adds storage, workers, and reconciliation latency, but it turns an invisible gap into explicit states.

Keep model reasoning outside the commit authority

The model may propose an action and arguments, but deterministic application code should assign the operation ID, validate authorization, enforce approval, and commit the effect. Never ask the model whether an action "looks already done" based only on conversation memory. Checkpoint state is useful workflow context; it is not a substitute for the authoritative business ledger.

For high-impact actions, separate proposal, approval, execution, and observation. Bind approval to a payload hash and operation ID so a resumed graph cannot reuse approval for changed arguments. Expire approvals deliberately and surface uncertain outcomes to an operator rather than allowing the agent to improvise a retry.

Sanitize trace payloads and restrict access because tool arguments and provider responses may contain personal or financial data. Preserve hashes and IDs needed for correlation when raw values must be redacted.

Assign ownership across the transaction

The graph team owns node boundaries, thread and checkpoint configuration, resume semantics, and state transitions. The tool-platform team owns stable operation propagation, retry policy, timeout classification, and adapter observability. The domain service owns uniqueness, authoritative effect status, reconciliation, and compensation rules. Infrastructure owns worker leases, queue delivery, and storage durability.

Security owns authorization and approval binding; incident operations owns containment and customer remediation. No owner can guarantee exactly-once behavior alone. The shared contract is effectively-once business behavior under duplicate execution attempts, with explicit unknown and reconciliation states.

Gate the repair under adversarial timing

Run a matrix across every failpoint, duplicate worker start, queue redelivery, provider timeout, process kill, and repeated resume. The following gate values are illustrative only:

{
  "illustrativeThresholds": true,
  "releaseGate": {
    "maximumBusinessEffectsPerLogicalOperation": 1,
    "requiredInjectedFailureScenariosPassing": 12,
    "maximumUncorrelatedExternalRequests": 0,
    "maximumApprovalPayloadMismatches": 0,
    "ambiguousOutcomesMustEnter": "reconciliation-required"
  }
}

Verify that duplicate attempts return the same recorded result and that changed payloads under an existing key fail explicitly. Measure added ledger and reconciliation latency against the product objective. For irreversible actions, correctness should dominate a small latency gain; for reversible low-risk actions, the product may choose a different operational balance, but it must be documented.

Execute the resume-safety action plan

Freeze retries, reconstruct checkpoint and external timelines, assign one logical operation ID to every attempt, locate the commit gap, and reproduce it with a failpoint. Add a durable idempotency or outbox boundary, bind approval to stable intent, and test every crash position with the same thread.

Close the incident only when repeated resumes, worker duplication, and ambiguous provider responses produce one authoritative effect or an explicit reconciliation state. Graph completion is not the release criterion. The business system's durable outcome is.