The approval worked once. Why did the agent use it twice?

A refund agent receives one human approval for an $80 payment, then two workers both report that they used it successfully. The customer sees two credits, while the audit screen shows one green approval. That is not a flaky test or a harmless retry, but an authorization artifact being treated as reusable state.

A useful agent approval token replay testing strategy proves which action was approved, which execution claimed it, whether the claim was atomic, and whether a retry could repeat the side effect while preserving evidence that separates an attack from ordinary queue redelivery.

Why one approval can authorize two actions

An approval token is a capability, even when the product calls it a confirmation id, resume key, checkpoint id, or signed decision. Possession lets some later component cross a boundary that it could not cross before. If the component checks only that the token exists or that its signature is valid, the same capability can cross that boundary again.

The failure often hides in an otherwise reasonable flow:

1. The agent proposes a tool call.
2. The backend stores the proposed action and pauses the run.
3. A human approves it.
4. A worker resumes the run with an approval token.
5. The worker validates the token and calls the tool.

Step five is where teams tend to stop thinking. Signature validation answers whether the token came from a trusted issuer and whether its protected contents changed. It does not, by itself, answer whether another execution already used it. A correctly signed, unexpired token can still be a replay.

The approval must bind to the action the reviewer actually saw. For a refund, that scope could include tenant, customer, original payment, amount, currency, destination, tool name, and a version of the proposal. A token that says only approved: true is transferable across every pending action that accepts the same shape. Even a token bound to refund is too broad if the amount or payment id can change after review.

Treat the approval as a small state machine. It starts as issued. One execution may move it to consumed. Time can move it to expired, and an operator can move it to revoked. No path returns a consumed approval to issued. A new attempt with the same token should receive a decision such as replayed, not another approved.

Concurrency makes the state transition important. This sequence is unsafe:

1. Worker A reads consumed_at = null.
2. Worker B reads consumed_at = null.
3. Worker A calls the external tool.
4. Worker B calls the external tool.
5. Both update the row as consumed.

There was a replay check, but it was a read followed by a write. The check did not serialize claimants. Production code needs a database transaction, a conditional update, or another compare-and-set primitive that permits one claimant. A process-local lock can demonstrate the rule in a unit test, but it is not a substitute for cross-process storage.

A second boundary sits beside token consumption: side-effect idempotency. Suppose a worker claims an approval, sends a refund request, and crashes before saving the response. On redelivery, the token is already consumed. Blindly rejecting the job may leave the system unsure whether the refund happened. Blindly allowing it may pay twice. The usual design is to create an idempotent command record in the same transaction that claims the approval, then use the command's stable key with the downstream adapter. A retry looks up that command and resumes or returns its recorded outcome. It does not consume the approval again.

That distinction gives the test its most important oracle:

• An unused token plus an exact action may create one command.
• The same execution id may observe the existing command after redelivery.
• A different execution id may not create another command from the token.
• A changed action may not use the token, even before consumption.
• An expired or revoked token may not create a command.

Do not use final database state as the only assertion. If two tool calls produce the same end state, such as setting a flag to true, a replay can disappear from the snapshot. Count authorization claims and adapter invocations as separate facts.

Build a replay test that can fail for the right reason

Start at the narrow authorization boundary. The following reference implementation is intentionally small. It uses an opaque random value, stores only its hash, binds it to a normalized action digest, and serializes consumers with a lock. The lock makes this example correct inside one Python process. Replace it with a transactional storage operation when workers run in different processes or hosts.

from __future__ import annotations

from dataclasses import dataclass, replace
from datetime import datetime, timedelta, timezone
from hashlib import sha256
from json import dumps
from secrets import token_urlsafe
from threading import Lock
from typing import Literal

Decision = Literal[
    "approved", "duplicate_delivery", "replayed",
    "scope_mismatch", "expired", "unknown_token"
]


def action_digest(action: dict[str, object]) -> str:
    encoded = dumps(
        action,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    ).encode("utf-8")
    return sha256(encoded).hexdigest()


@dataclass(frozen=True)
class ApprovalRecord:
    token_hash: str
    subject: str
    tenant: str
    action_hash: str
    expires_at: datetime
    consumed_by: str | None = None


class ApprovalStore:
    def __init__(self) -> None:
        self._records: dict[str, ApprovalRecord] = {}
        self._lock = Lock()

    def issue(
        self,
        *,
        subject: str,
        tenant: str,
        action: dict[str, object],
        now: datetime,
    ) -> str:
        raw_token = token_urlsafe(32)
        token_hash = sha256(raw_token.encode("utf-8")).hexdigest()
        self._records[token_hash] = ApprovalRecord(
            token_hash=token_hash,
            subject=subject,
            tenant=tenant,
            action_hash=action_digest(action),
            expires_at=now + timedelta(minutes=10),
        )
        return raw_token

    def consume(
        self,
        *,
        raw_token: str,
        tenant: str,
        action: dict[str, object],
        execution_id: str,
        now: datetime,
    ) -> Decision:
        token_hash = sha256(raw_token.encode("utf-8")).hexdigest()
        with self._lock:
            record = self._records.get(token_hash)
            if record is None or record.tenant != tenant:
                return "unknown_token"
            if record.expires_at <= now:
                return "expired"
            if record.action_hash != action_digest(action):
                return "scope_mismatch"
            if record.consumed_by == execution_id:
                return "duplicate_delivery"
            if record.consumed_by is not None:
                return "replayed"

            self._records[token_hash] = replace(
                record,
                consumed_by=execution_id,
            )
            return "approved"


UTC = timezone.utc

The example's JSON encoding is suitable only for a domain object whose allowed keys and value types have already been validated. It is not a universal canonicalization standard. If one service represents money as 80, another as 80.0, and a third as the string "80.00", define one money representation before calculating the digest. The scope-mutation article linked with this topic goes deeper into that boundary.

The first worked case races two independent executions. A sequential "use it, then use it again" test is necessary, but it will not find a check-then-update race. A barrier holds both tasks until they are ready, and a thread pool releases them together. The pass condition is one approval and one replay decision.

from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from threading import Barrier


def test_two_executions_race_for_one_approval() -> None:
    store = ApprovalStore()
    now = datetime(2026, 8, 4, 10, 0, tzinfo=timezone.utc)
    action = {
        "tool": "issue_refund",
        "payment_id": "pay_4102",
        "amount_minor": 8000,
        "currency": "USD",
    }
    token = store.issue(
        subject="reviewer-17",
        tenant="shop-a",
        action=action,
        now=now,
    )
    barrier = Barrier(2)

    def attempt(execution_id: str) -> str:
        barrier.wait()
        return store.consume(
            raw_token=token,
            tenant="shop-a",
            action=action,
            execution_id=execution_id,
            now=now,
        )

    with ThreadPoolExecutor(max_workers=2) as pool:
        results = list(pool.map(attempt, ["exec-a", "exec-b"]))

    assert Counter(results) == Counter({"approved": 1, "replayed": 1})


def test_redelivery_does_not_become_a_second_authorization() -> None:
    store = ApprovalStore()
    now = datetime(2026, 8, 4, 10, 0, tzinfo=timezone.utc)
    action = {"tool": "close_case", "case_id": "case-9"}
    token = store.issue(
        subject="reviewer-17",
        tenant="support",
        action=action,
        now=now,
    )

    first = store.consume(
        raw_token=token,
        tenant="support",
        action=action,
        execution_id="exec-91",
        now=now,
    )
    retry = store.consume(
        raw_token=token,
        tenant="support",
        action=action,
        execution_id="exec-91",
        now=now,
    )

    assert first == "approved"
    assert retry == "duplicate_delivery"

The second case does not say a duplicate delivery is authorized. It says the system recognizes the same execution. The caller must fetch the existing command instead of invoking the tool again. Add a fake adapter with an invocation counter at the service layer and assert that the counter remains one across both deliveries.

Next, mutate one field at a time. Replaying an unchanged action tests single use. Changing the action tests binding. These are related controls with different failure messages, and combining them into one broad assertion makes triage harder.

from copy import deepcopy

import pytest


@pytest.mark.parametrize(
    ("field", "new_value"),
    [
        ("payment_id", "pay_9999"),
        ("amount_minor", 80_000),
        ("currency", "EUR"),
        ("tool", "send_payment"),
    ],
    ids=["destination", "amount", "currency", "tool"],
)
def test_approval_is_bound_to_reviewed_action(
    field: str,
    new_value: object,
) -> None:
    store = ApprovalStore()
    now = datetime(2026, 8, 4, 10, 0, tzinfo=timezone.utc)
    approved = {
        "tool": "issue_refund",
        "payment_id": "pay_4102",
        "amount_minor": 8000,
        "currency": "USD",
    }
    token = store.issue(
        subject="reviewer-17",
        tenant="shop-a",
        action=approved,
        now=now,
    )
    mutated = deepcopy(approved)
    mutated[field] = new_value

    decision = store.consume(
        raw_token=token,
        tenant="shop-a",
        action=mutated,
        execution_id="exec-mutated",
        now=now,
    )

    assert decision == "scope_mismatch"

A strong matrix also covers a token from another tenant, an unknown token, expiry at the exact boundary, revocation, a missing scope field, an added scope field, and a different principal if the principal matters to execution. Keep the expected reason explicit. A generic assert decision != "approved" lets a broken parser pass because it returned unknown_token for every request.

Read the evidence before naming the bug

The symptom "the action happened twice" does not prove token replay. Duplicate side effects can come from an adapter retry, a webhook consumer, a user submitting the same request twice, or an upstream service ignoring an idempotency key. The authorization record must connect issuance, claim, command creation, and adapter invocation.

For each approval, retain these fields in structured events:

• approval_id or a one-way token fingerprint
• tenant_id and the approving subject
• action_hash plus a safe summary of the action
• issued_at, expires_at, and consumed_at
• execution_id, queue message id, and command id
• the decision and a stable reason code
• the trace or correlation id used across services

Never put the raw bearer token in logs, traces, screenshots, or assertion errors. A log collector has a wider audience and longer retention than an authorization endpoint. If engineers can copy a token from a failed CI artifact and use it, the diagnostic system has created another replay path.

The clearest replay evidence is two distinct execution ids associated with the same approval fingerprint, where one event says approved and a later event attempts authorization after consumed_at. If both events say approved, the authorization control failed. If only one says approved but there are two adapter calls for one command id, authorization worked and idempotent execution failed. If there are two separate approvals for identical actions, look at the proposal or user interface layer instead.

A small audit scanner catches the first pattern without exposing token values. It expects newline-delimited JSON events and reports any fingerprint attached to more than one approved execution.

from __future__ import annotations

import json
import sys
from collections import defaultdict
from pathlib import Path


def approved_executions(path: Path) -> dict[str, set[str]]:
    by_token: dict[str, set[str]] = defaultdict(set)
    with path.open(encoding="utf-8") as events:
        for line_number, line in enumerate(events, start=1):
            event = json.loads(line)
            if event.get("event") != "approval_decision":
                continue
            if event.get("decision") != "approved":
                continue
            fingerprint = event["token_fingerprint"]
            execution_id = event["execution_id"]
            by_token[fingerprint].add(execution_id)
    return by_token


if __name__ == "__main__":
    audit_path = Path(sys.argv[1])
    offenders = {
        token: sorted(executions)
        for token, executions in approved_executions(audit_path).items()
        if len(executions) > 1
    }
    if offenders:
        print(json.dumps(offenders, indent=2, sort_keys=True))
        raise SystemExit(1)
    print("No approval fingerprint authorized multiple executions")

Run the scanner against a sanitized test artifact, then inspect the named executions in the trace system your team already uses.

python tools/find_approval_replays.py artifacts/approval-events.jsonl
pytest tests/agent/test_approval_replay.py -vv --log-cli-level=INFO

When the race test fails, pytest will identify the parameter or test name and show the actual counter, for example two approved values instead of one approved and one replayed. That output is more useful than "duplicate refund detected" because it locates the defect before the external tool. Add decision reason, approval fingerprint, and execution id through structured logging so captured test logs show the same chain.

Three near-misses deserve separate checks.

First, queue redelivery usually has the same message id and execution id. It should resolve to the existing command. If the consumer creates a fresh execution id on every delivery, the transport layer destroys the evidence needed to distinguish a retry. Fix identity propagation before changing token policy.

Second, clock skew produces expired on one worker and approved on another near the deadline. That is not replay, although it can create inconsistent results. Inject a clock in tests, compare timestamps in UTC, and define the exact expiry rule. The reference code treats now == expires_at as expired.

Third, duplicated proposals can lead to two valid approvals for what a human thinks is one action. The tokens differ, so replay detection will stay green. Look for a repeated proposal id, two approval prompts, or an interface that resubmits after a timeout. Deduplicate proposals or show their identities clearly rather than weakening single-use semantics.

Separate replay from duplicated audit delivery

Two approved rows in a log search can look like two successful claims even when the authorization service made one decision. Audit events are often delivered through infrastructure that can redeliver after an acknowledgement is lost. If the log sink appends the same event twice and the dashboard counts rows, the incident looks like token replay. The root cause is duplicated observability data, and changing approval consumption will not remove it.

Give each authorization decision an immutable event identity at its producer and preserve that identity through delivery. Then read the approval fingerprint, decision identity, execution ID, command ID, and adapter invocation together. A duplicated audit delivery has the same event identity, fingerprint, execution, decision, command, and source timestamp in both rows. The approval store still shows one claim, the command store shows one command, and the adapter evidence shows one invocation. A true authorization failure has distinct claim attempts, usually distinct executions, and either two successful decision identities or one successful decision followed by a replay denial.

The raw count is the misleading value. approved rows=2 does not say whether there were two decisions. A healthy retry can show several queue deliveries while retaining one execution and command. A blocked replay shows one approved execution, a second execution with a replay reason, and one command. A broken claim boundary shows the same token fingerprint attached to two approved executions and commonly two command IDs. Put those values on one investigation screen or export; asking separate teams for screenshots of separate systems makes ordering uncertain.

Source timestamps alone cannot deduplicate events. Two distinct decisions can occur at the same clock resolution, and one event can receive a new ingestion timestamp when it is redelivered. Payload equality is also insufficient because two workers may produce identical fields legitimately. Use producer-owned identity for the event, then keep ingestion identity only to diagnose the delivery path. The token bearer value remains absent from both.

For an existing suite, land the decision identity and command correlation before enforcing new replay outcomes. Update audit consumers and dashboards to tolerate and deduplicate the new identity, then deploy the producer that emits it. Next, make queue consumers preserve message and execution identities, and make adapters report the stable command identity. Only after those joins work should replay denial become a release gate. Recovery jobs that currently present the approval token again instead of resuming the command will break first. Row-count alerts may also change because duplicate audit delivery stops inflating totals.

Evidence is working when an injected audit redelivery creates two ingested rows but one producer decision, while the concurrent-claim fixture creates two decision attempts and exactly one command. Keep both fixtures. If only the authorization race is tested, the team may declare an incident from duplicated telemetry. If only audit deduplication is tested, a real second claim can be hidden by an overly broad grouping rule.

This separation costs storage and coordination. The audit path must retain producer event identities long enough to recognize delivery repeats, and consumers may need a uniqueness check or deduplication state. The authorization path still needs its conditional claim and command transaction. Those are two independent controls with separate failure modes, so one database constraint cannot stand in for both.

The authorization team owns atomic claim state and decision identity. The workflow or queue team owns message and execution identity across redelivery. The adapter team owns command idempotency and invocation evidence. The audit platform owns event deduplication without collapsing distinct decisions, while the product UI owns proposal identity and the action shown to the reviewer. A handoff should contain the token fingerprint, producer decision identities, executions, queue message identities, command IDs, adapter invocation count, source and ingestion ordering, and the stored approval state. Raw tokens and sensitive action payloads do not belong in that packet.

Replay protection does not catch a bad action that a human approves once. If the interface hides a destination, displays stale scope, or persuades the reviewer to authorize a malicious request, a correctly bound single-use token can work exactly once and still cause harm. That needs trustworthy action presentation, reviewer authorization, scope validation, and tests that compare what was displayed with what was executed.

Fix the consumption boundary without breaking retries

The production fix belongs where the approval changes state, not in the prompt. Telling the model not to reuse a token may reduce accidental attempts, but a model instruction cannot serialize workers or revoke a capability. Authorization code must reject the second claimant regardless of why it arrived.

In a relational store, the central operation is a conditional state change. Conceptually, update the approval to consumed only when its current state is issued, its expiry is in the future, its tenant and action digest match, and it has not been revoked. Check the affected-row count. Zero means the caller must read the current record and return a precise decision. Do not implement this as an unrestricted update preceded by a select.

Create the command record in the same transaction. Its key should remain stable across transport retries. After the transaction commits, a worker can deliver that command to the adapter. If the adapter supports idempotency keys, pass the stable command id. If it does not, the system needs its own reconciliation strategy and must accept that a crash at the network boundary can be ambiguous.

This design costs storage and control-flow complexity. Every approved action now has an approval record, a command record, and several audit events. Transactions may create contention when many workers target the same approval, although that contention is exactly what enforces one winner. Expired records also need a retention policy that preserves investigations without retaining sensitive payloads indefinitely.

Do not silently convert every replay into success. Returning the previous command result is correct only when the same authenticated caller presents the same tenant, action, and execution identity. A new execution, changed scope, or unknown caller should receive a denial and create a security-relevant event. Otherwise an attacker can learn or reuse results by copying a key.

The human interface needs one change too. After approval, disable or replace the approval control with consumed state based on server data. This is not the security boundary, because two clients can click before either receives the update, but it reduces accidental repeats and gives operators an honest picture. Show the action summary and the eventual command id so support can trace what the approval authorized.

Roll the control into an existing agent suite

A replay fix can break legitimate retries if introduced as a single release gate. Start by instrumenting issuance and consumption without changing decisions. Generate token fingerprints at the authorization service, propagate execution ids, and count distinct claimants. Redact payload fields that contain customer data. Run long enough to find consumers that drop or regenerate identities.

Next, add deterministic unit tests for state transitions and action binding. Run them on every change to the authorization package. Add the concurrent race case at the service or database integration level because a mocked repository cannot prove that the real conditional write is atomic.

Then place the new rejection in observe-only mode. When a request would be rejected as a replay, record the decision and follow the current behavior in a non-production environment. For production observation, do not knowingly permit dangerous duplicate side effects merely to collect data. Restrict observation to read-only actions or compare against historical events.

Move low-risk tools first. A knowledge-base lookup can tolerate an unexpected denial while the team fixes identity propagation. Payments, deletions, account changes, and outbound messages need stronger pre-release proof, a sandbox adapter, and an operator recovery path before enforcement changes.

Wire the deterministic suite separately from slower sandbox scenarios. This workflow is illustrative and assumes the example files live in a Python project with pytest declared by the project.

name: approval-replay-contract

on:
  pull_request:
    paths:
      - "agent_auth/**"
      - "tests/agent/test_approval_replay.py"

jobs:
  replay-contract:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements-test.txt
      - run: python -m pip install -r requirements-test.txt
      - run: >-
          python -m pytest
          tests/agent/test_approval_replay.py
          -vv
          --junitxml=artifacts/approval-replay.xml
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: approval-replay-evidence
          path: artifacts/
          retention-days: 7

The explicit cache-dependency-path is not decoration. With cache: "pip" enabled, actions/setup-python looks only for **/requirements.txt and **/pyproject.toml, so a repository that pins its test dependencies in requirements-test.txt fails at the setup step before a single replay assertion runs.

Keep a small set of end-to-end cases for each dangerous adapter: approve once, deliver the command twice, and prove the sandbox records one effect. Also test the recovery path after a simulated crash between command creation and adapter response. Those scenarios are slower, but they cover a boundary the in-memory unit test deliberately does not.

A sensible release gate blocks when one approval authorizes two execution ids, a mutated action is accepted, or the adapter receives multiple commands from one approval. It should not block merely because the system observed duplicate delivery and returned the stored result. That is the retry mechanism working.

During migration, keep old and new decision fields side by side rather than reinterpreting old logs. A historical success may mean signature accepted, command created, or tool completed. New events should use narrower names such as approval_claimed, command_dispatched, and adapter_result_recorded. Investigators cannot reconstruct a replay if one broad status overwrites each stage.

Know when replay protection is the wrong test

Do not require single-use approval for an action that is intentionally a durable policy grant. A human might allow an agent to read a named calendar for seven days or query a specific repository during one incident. That grant still needs an audience, subject, resource boundary, expiry, revocation, and audit trail, but consuming it once would contradict the product design. Test scope enforcement and expiration instead.

Avoid using approval tokens as idempotency keys. An approval answers "may this action proceed?" An idempotency key answers "have I already processed this logical command?" One approval may produce exactly one command whose delivery is retried many times. Combining the identities makes recovery harder and tempts code to authorize a retry simply because a downstream request failed.

Skip live destructive tools in broad mutation suites. The authorization decision is deterministic and can be tested against fakes. Use a tightly controlled sandbox for the small number of integration cases that must prove wiring. A replay test that sends real email, deletes real files, or moves real money creates more risk than evidence.

Do not treat every duplicated read as a security incident. A search or status lookup may be safely repeatable, and infrastructure retries can be expected. You still want accurate command counts for cost and rate-limit reasons, but the release severity should reflect the effect. Reserve the strongest gate for actions where repetition changes state, exposes protected data, spends money, or contacts people.

Finally, do not accept a green sequential test as proof against races. It proves the basic state transition and nothing more. Keep the concurrent integration case, preserve distinct execution identities, and verify the adapter invocation count. That combination tells you the approval was single use at the point where it mattered.