Prove a retried API call cannot create a second side effect

A payment request times out, so the client sends it again. Both responses look harmless, but the ledger now contains two charges with different IDs. The retry exposed the bug; the missing atomic deduplication created it.

Decide what idempotent means for this operation

A useful test follows one logical operation across every attempt. It records the idempotency key, request fingerprint, reservation state, response, and durable side effect. The pass condition is not "the second call returned 200." It is "the system applied the intended effect no more than once under the contract we published."

RFC 9110 defines an idempotent HTTP method by its intended effect on the server: several identical requests have the same intended effect as one. Safe methods, PUT, and DELETE are idempotent under the standard's method semantics. The server may still log each request or retain revision history, and repeated responses do not have to be byte-for-byte identical.

That definition immediately fixes two common test mistakes. First, response equality is not the primary oracle. A repeated DELETE can return one status when it removes a resource and another after the resource is already gone while preserving the same requested end state. Second, an idempotent method name does not excuse an implementation that repeats the intended business effect. A PUT endpoint that appends a new row on every identical call violates its advertised semantics.

POST is not idempotent by default. A product can make a particular POST operation safely repeatable by adding an application-level deduplication contract. An idempotency key is a common design, but the header name alone has no magic. The service must define and enforce how the key works.

Before writing automation, settle these decisions:

• Scope: Is the key unique globally, per tenant, per account, or per operation route?
• Request identity: Which method, path, body fields, and relevant headers contribute to the fingerprint?
• Concurrent duplicates: Does the second request wait, receive an in-progress response, or get the completed outcome?
• Payload mismatch: What happens when the same key arrives with different semantics?
• Stored outcome: Are successful responses replayed? Are deterministic client errors stored? What about temporary server failures?
• Retention: When can the record expire, and what does a request after expiry mean?
• Response metadata: Which operation or resource identifier lets the client reconcile an uncertain outcome?

These are product choices. Do not borrow one provider's behavior and call it an HTTP rule. Put the decisions in the API description or developer documentation, then make the tests quote that local contract.

The key must represent one logical operation. A client retry reuses it. A user intentionally submitting a second order needs a new key, even if the body is identical. Tests that generate a fresh key on every attempt can never exercise deduplication. Tests that reuse one hard-coded key across unrelated cases create pollution and confusing replay results.

Fingerprinting protects the other direction. Without it, a client can accidentally reuse a key for a different amount or recipient and receive the old result. Hash only canonical business inputs that define the operation. Raw JSON bytes are often too strict because harmless property order or whitespace can differ. Ignoring a meaningful field is too loose because a changed operation can collide.

A concrete payment contract might say:

• Scope is merchant ID plus POST /charges plus idempotency key.
• Currency, amount, customer ID, and merchant reference form the fingerprint.
• The first request atomically reserves the key.
• A completed duplicate receives the stored operation ID and outcome.
• A different fingerprint receives a conflict response and creates no charge.
• A request while the first is pending receives the documented in-progress behavior.
• Records remain usable for the published retry window.

Once this is written, every assertion has an owner. A scope failure belongs to key lookup. A duplicate charge belongs to reservation or transaction logic. An incorrect replay body belongs to outcome storage. An unexpected post-expiry replay belongs to retention.

Make the side effect your primary oracle

A black-box test needs two observation points: the public API response and an authoritative read of the resulting business state. For a payment, that might be a test ledger query. For an order, it might be a read endpoint plus an event sink. For a background job, it might be a job record and the downstream action it performs.

The following Node test targets an illustrative test environment. This example contract returns 201 for the first creation, 200 for a replay, and a stable chargeId. It also exposes a protected test-support query for records created under an isolated merchant reference. Replace those routes with supported observation points in your system. Do not add a production backdoor for the sake of a test.

import assert from "node:assert/strict";
import test from "node:test";
import { randomUUID } from "node:crypto";

type ChargeResponse = {
  chargeId: string;
  merchantReference: string;
  amountMinor: number;
  currency: string;
};

const baseUrl = process.env.API_BASE_URL;
const testToken = process.env.API_TEST_TOKEN;

assert.ok(baseUrl, "API_BASE_URL is required");
assert.ok(testToken, "API_TEST_TOKEN is required");

async function createCharge(
  key: string,
  merchantReference: string,
): Promise<{ status: number; body: ChargeResponse }> {
  const response = await fetch(baseUrl + "/charges", {
    method: "POST",
    headers: {
      authorization: "Bearer " + testToken,
      "content-type": "application/json",
      "idempotency-key": key,
    },
    body: JSON.stringify({
      amountMinor: 1250,
      currency: "INR",
      customerId: "customer-fixture-7",
      merchantReference,
    }),
  });

  return {
    status: response.status,
    body: (await response.json()) as ChargeResponse,
  };
}

async function findTestCharges(
  merchantReference: string,
): Promise<ChargeResponse[]> {
  const url = new URL("/test-support/charges", baseUrl);
  url.searchParams.set("merchantReference", merchantReference);

  const response = await fetch(url, {
    headers: { authorization: "Bearer " + testToken },
  });
  assert.equal(response.status, 200);

  return (await response.json()) as ChargeResponse[];
}

test("replaying one logical charge creates one ledger record", async () => {
  const key = randomUUID();
  const reference = "idem-" + randomUUID();

  const first = await createCharge(key, reference);
  const replay = await createCharge(key, reference);

  assert.equal(first.status, 201);
  assert.equal(replay.status, 200);
  assert.equal(first.body.chargeId, replay.body.chargeId);
  assert.equal(first.body.merchantReference, reference);
  assert.equal(replay.body.merchantReference, reference);

  const records = await findTestCharges(reference);
  assert.equal(records.length, 1);
  assert.equal(records[0].chargeId, first.body.chargeId);
  assert.equal(records[0].amountMinor, 1250);
});

The two status assertions belong to this example contract, not to HTTP in general. Another API can document a different replay status. The important point is to assert the published behavior without pretending that matching response bodies prove idempotency.

A reliable oracle counts durable effects after both requests finish. If the operation publishes messages, count the downstream business event, not every internal delivery attempt. At-least-once messaging may legitimately deliver the same envelope more than once while the consumer deduplicates the effect. Define the boundary where "once" matters to the user.

Use isolated accounts and unique references. Cleanup can hide a duplicate if it runs before evidence is captured, so save the two HTTP results and the state query first. Then remove fixtures through supported test cleanup. Never run duplicate-charge tests against real payment rails or shared customer data.

The operation record itself is useful evidence when the service exposes it safely. A row or document should show the key scope, request fingerprint, state, and final resource ID. Tests do not need direct production database access, but a component test can inspect the repository transaction and a staging test can use a support API with strict access controls.

Reproduce the three failures that matter most

Failure one: a sequential replay creates a second effect.

This is the smallest case. Send one request to completion, then repeat the exact method, route, key, and payload. If two effects appear, the key may not be stored, the lookup scope may differ, or the replay path may generate a new resource instead of returning the stored outcome.

Inspect the normalized fingerprint on both attempts. If it differs, compare canonicalization and fields such as default currency, headers, or server-added values. If the fingerprint matches but two reservations exist, inspect the unique constraint and lookup scope. If one reservation points to two charge IDs, the state transition is broken.

A near-miss looks similar: only one charge exists, but the second response names a different charge ID. That is not a harmless presentation issue. The client may reconcile or refund the wrong resource. Assert both the business count and stable identity.

Failure two: two concurrent duplicates both pass a check-then-insert path.

Code that reads "key not found" and later inserts a reservation has a race unless the boundary is atomic. Two workers can both read the gap before either writes. A sequential test will stay green forever.

Use the same payload and release both network calls together. Promise.all does not guarantee they arrive on the same CPU cycle, but it is a useful deployed regression. For a deterministic component test, add a controllable barrier around the reservation boundary in test-only wiring and prove both workers contend on the real database constraint.

For this example contract, the duplicate waits for the owner to complete, then receives the stored result with status 200. The owner returns 201. If your API returns an in-progress response instead, assert that response and follow it with a later replay rather than copying these status expectations.

import assert from "node:assert/strict";
import test from "node:test";
import { randomUUID } from "node:crypto";

test("concurrent duplicates resolve to one charge", async () => {
  const key = randomUUID();
  const reference = "race-" + randomUUID();

  let releaseStart!: () => void;
  const start = new Promise<void>((resolve) => {
    releaseStart = resolve;
  });

  const attempt = async () => {
    await start;
    return createCharge(key, reference);
  };

  const firstAttempt = attempt();
  const secondAttempt = attempt();
  releaseStart();

  const [first, second] = await Promise.all([
    firstAttempt,
    secondAttempt,
  ]);

  assert.deepEqual(
    [first.status, second.status].sort((left, right) => left - right),
    [200, 201],
  );
  assert.equal(first.body.chargeId, second.body.chargeId);

  const records = await findTestCharges(reference);
  assert.equal(records.length, 1);
  assert.equal(records[0].chargeId, first.body.chargeId);
});

The shared promise releases both callers from the same point. Network and scheduling still add variation, so repeat this focused case in a bounded loop during a race suite rather than hiding failures with runner retries.

Illustrative failure evidence might look like this:

logical_reference=race-4c5f
attempt_a.status=201
attempt_a.charge_id=ch_81
attempt_b.status=201
attempt_b.charge_id=ch_82
idempotency_rows=1
ledger_charge_ids=[ch_81,ch_82]

These are invented identifiers that show the shape of a defect, not results from an experiment. One idempotency row beside two ledger rows suggests the side effect escaped the transaction or was invoked before the reservation owner was settled.

Failure three: the first request commits, but the client loses the response.

This is the production-shaped case teams often skip. A client timeout does not tell you whether the server applied the operation. If the client retries with a new key, duplicate prevention cannot connect the attempts. If it retries with the original key, the service should follow the documented replay behavior.

Inject the fault after the durable commit but before the response reaches the client. Doing this precisely usually requires component-level fault injection or a controlled proxy. A generic client timeout against a slow endpoint is not enough because the request might be cancelled before commit.

In the test, capture the key before sending. Force the first response to become unavailable at the controlled boundary. Query the authoritative store until the committed operation is visible, within a bounded wait. Retry the identical request with the same key. Assert the returned resource ID matches the already committed effect and that no second effect appears.

Do not claim the transport can always cancel server work. An aborted client connection and an application rollback are separate behaviors. The trace must show where the commit occurred relative to the lost response. Without that evidence, "timeout after commit" is only a guess.

A different-payload mutation belongs beside these three cases:

import assert from "node:assert/strict";
import test from "node:test";
import { randomUUID } from "node:crypto";

test("one key cannot authorize a different amount", async () => {
  const key = randomUUID();
  const reference = "mismatch-" + randomUUID();

  const first = await createCharge(key, reference);
  assert.equal(first.body.amountMinor, 1250);

  const response = await fetch(baseUrl + "/charges", {
    method: "POST",
    headers: {
      authorization: "Bearer " + testToken,
      "content-type": "application/json",
      "idempotency-key": key,
    },
    body: JSON.stringify({
      amountMinor: 5000,
      currency: "INR",
      customerId: "customer-fixture-7",
      merchantReference: reference,
    }),
  });

  assert.equal(response.status, 409);

  const records = await findTestCharges(reference);
  assert.equal(records.length, 1);
  assert.equal(records[0].amountMinor, 1250);
});

The 409 status is part of this example contract, not an HTTP-wide rule for idempotency mismatches. Some APIs choose another client-error status. The non-negotiable property is that the changed request must not silently reuse the old authorization or create another side effect.

Use evidence to separate races, scope bugs, and expiry

A duplicate effect with two keys is usually a client retry bug or a missing higher-level operation ID. A duplicate effect with one key is usually a server enforcement bug. Capture the actual header on every attempt before deciding ownership.

If the keys match, compare their scope. The same text key under two tenants may be valid. The same key under one tenant but two routes may be valid if routes are part of scope. A lookup performed globally when the contract promises tenant scope can replay another customer's result, which is more serious than duplication.

Payload mismatch failures need the canonical fingerprint inputs. Save sanitized field names and the resulting digest, not card data or credentials. If semantically identical JSON produces different fingerprints, canonicalization is too sensitive. If different amounts produce the same fingerprint, a required field is missing from the identity.

Expiry creates a recognizable pattern. The first replay works, a later replay creates a new operation, and the key record's retention timestamp falls between them. That can be correct after the published window. Use a controllable application clock in component tests instead of sleeping for hours or editing database timestamps behind the service.

A retention test should cover three points: safely inside the window, exactly at the documented boundary, and just after it. The boundary behavior must be explicit. Inclusive and exclusive expiry comparisons can disagree at one instant, and clocks on different processes can create confusing results if the service relies on local wall time.

Pending operations create another near-miss. Two responses may differ because the duplicate arrives while the owner is still working. If the contract returns "in progress," assert that no second owner starts and that a later replay reaches the final stored result. If the duplicate waits, assert a timeout bound so one stuck owner does not hold every retry forever.

Failure caching is not one universal choice. A deterministic validation failure can often be recomputed safely, while a 500 caused by a temporary dependency may need another attempt. Caching every failure can make an outage sticky. Retrying every failure can repeat a side effect after an uncertain commit. Define state transitions for reserved, completed, safely failed, and indeterminate outcomes.

The following PostgreSQL sketch shows the database property a component test should prove. It creates a unique scope over tenant, operation, and key. Adapt types, retention, and privileges to your schema.

psql "$DATABASE_URL" -v ON_ERROR_STOP=1 <<'SQL'
CREATE TABLE idempotency_record (
  tenant_id uuid NOT NULL,
  operation text NOT NULL,
  idempotency_key text NOT NULL,
  request_fingerprint text NOT NULL,
  state text NOT NULL CHECK (
    state IN ('pending', 'completed', 'failed')
  ),
  resource_id text,
  response_status integer,
  response_body jsonb,
  expires_at timestamptz NOT NULL,
  PRIMARY KEY (tenant_id, operation, idempotency_key)
);
SQL

A primary key prevents two rows with the same scope. It does not by itself make the business effect atomic. The owner must reserve the key and apply the effect under a transaction or recovery design that cannot leave an untracked effect. Component tests should stop execution at each boundary, restart the worker, and verify the record can recover without a duplicate.

Logs help only when they carry the same logical identifiers. Record the scoped key or a safe digest, fingerprint digest, reservation owner, state transition, and resource ID. Do not log raw sensitive request bodies. When two services participate, propagate an operation identifier distinct from the transport request ID.

Roll out atomic handling without breaking existing clients

Begin with observation. Add server-side logging or metrics for repeated keys, mismatched fingerprints, pending duration, and duplicate-effect alarms. Do not change response behavior before you know which clients already reuse keys incorrectly.

Next, create the reservation store and run it in shadow mode for a bounded period. Calculate what the service would have done, but keep the old behavior. Compare would-be replays and mismatches with actual outcomes. Shadow mode must not reserve keys in a way that blocks real requests.

Enforce the contract for a small operation first. Payment-like writes need careful reconciliation and support procedures. A lower-risk job submission can prove the storage and client integration before the design protects irreversible effects.

Database migration has concrete costs. Every write gains at least one reservation lookup or insert. Completed outcomes consume storage until expiry. Hot tenants can contend on indexes. Storing full response bodies raises privacy and schema-retention questions. Measure these in your system instead of repeating a generic overhead percentage.

Atomic reservation commonly relies on a unique database constraint because application locks do not coordinate reliably across processes without shared state. The winner inserts the scoped key. A loser reads the existing record and follows its state. If the business write and reservation share a database, one transaction can simplify ownership. If they span systems, an outbox, workflow, or reconciliation process may be necessary.

Client rollout matters just as much. Generate a key once when the user begins one logical operation. Persist it across transport retries. Generate a new key only when the user intentionally creates a new operation. Mobile clients may need to retain the key through process suspension; browser clients may need to avoid duplicate event handlers issuing separate keys.

Run old and new behavior side by side in tests:

• Existing clients without a key receive the documented legacy behavior.
• Updated clients with a key get replay protection.
• Missing, malformed, or oversized keys follow an explicit validation rule.
• The same key cannot cross tenants or operations.
• The key survives a client retry after an ambiguous network outcome.

Decide when the key becomes required. Making it mandatory immediately can break older clients. An API version, capability flag defined by your product, or staged warning period may be safer. Avoid an undocumented silent fallback where some requests deduplicate and others do not.

CI should keep deterministic storage races separate from broader load tests:

name: idempotency-contract

on:
  pull_request:
    paths:
      - "src/charges/**"
      - "src/idempotency/**"
      - "tests/idempotency/**"
  workflow_dispatch:

jobs:
  component:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
      - run: npm ci
      - run: npm run test:idempotency
        env:
          DATABASE_URL: "postgres://postgres:test@localhost:5432/postgres"

The script name is a repository-specific placeholder. The test job should use isolated keys, bounded concurrency, and a real unique constraint. A scheduled load job can explore pool pressure, but it should not replace the precise two-worker race that tells you why a duplicate occurred.

Separate duplicate creation from duplicate delivery

Two ledger effects after one public request do not always mean two API workers won the idempotency race. The API can reserve one key and emit one business command, while a downstream consumer applies that command twice after redelivery. From the caller, both defects look the same: one key, plausible responses, and two visible effects. Even the diagnostic line idempotency_rows=1 appears in both cases.

The evidence that separates them is identity at the producer and consumer boundaries. Trace the stable identity of the business command or event created for the logical operation. Two distinct command identities tied to one reservation point back to producer recovery or dispatch. One command identity observed in more than one consumer delivery points downstream. Then inspect the consumer's durable effects and any deduplication record it owns. If the architecture carries no stable business-command identity, add one before assigning the incident from timestamps alone. Transport request IDs are insufficient because redelivery creates another transport attempt for the same business work.

Read the existing failure output in that order. Matching public keys and one reservation row are healthy only through the reservation boundary. One resource ID in the completed reservation and one ledger effect is the complete healthy result. Two ledger IDs with two producer command identities implicate the producer side. Two ledger IDs with one repeated command identity implicate consumer application or recovery. The misleading value is a successful replay status: it says the API returned its stored outcome, but it says nothing about whether downstream work was applied once.

For an existing suite, retain the current API replay and race cases. Land command-identity capture and a read-only downstream oracle before changing enforcement. Add a controlled consumer replay of the same recorded command, then prove that the user-visible effect remains singular. Keep that case separate from the two-request API race so a failure names the boundary. The rollout is working when the suite can fail producer uniqueness while consumer replay remains green, and can produce the opposite result without changing the public fixture.

Consumer deduplication has a specific cost. Remembering processed command identities adds a persistent lookup or write to message handling, retains identifiers for a chosen period, and creates cleanup and recovery rules. A short retention period can reopen an old delivery; indefinite retention grows storage. The retention decision belongs to the maximum redelivery and reconciliation risk of that workflow, not to a generic idempotency default.

The API team owns reservation and command creation. The messaging platform team owns delivery evidence but not the business decision to apply an effect. The consumer team owns atomic handling of a command and its side effect. The ledger or domain team owns the authoritative effect count. A handoff needs the scoped public key digest, reservation and resource identities, producer command identity, every observed consumer delivery of it, and the final business rows. Without that chain, sending a duplicate screenshot to the API team only moves the guesswork.

This technique does not catch a second side effect that the chosen oracle never observes. A test can prove one ledger row while two emails, shipments, or external provider calls occur outside that store. List every irreversible effect in the operation contract and give high-risk effects their own observation or reconciliation check. One authoritative database count is not automatically authoritative for the whole workflow.

Skip idempotency keys when another contract already solves it

Do not add a key to a read-only GET merely to look consistent. GET is already defined as safe and idempotent at the HTTP level. Test that the implementation honors those semantics and fix unexpected side effects directly.

A natural resource identifier can make PUT the clearer design. If the client owns an order ID, repeated PUT /orders/{id} requests can converge on one resource under a documented replacement or update contract. Adding a second deduplication identity may create conflicts over which identifier wins.

Avoid promising exactly-once transport delivery. Networks and message systems retry. What the application can enforce is one permitted business effect for one logical operation, with durable state and reconciliation. Phrase the contract at that boundary.

A key is also the wrong fix for a UI that sends two intentional operations because two handlers fire. Deduplication may mask the frontend bug and merge actions the user meant to keep separate. Fix event handling, then keep server protection for genuine retries.

Do not retain keys forever without a reason. Indefinite records create storage and privacy costs, and they can prevent a customer from intentionally repeating an old operation. Publish a retention window that matches the business risk and test its edge.

Finally, do not use idempotency as a substitute for authorization or validation. A replay must still belong to the same authorized scope, and a key must not let a caller retrieve another tenant's result. Validate identity and request semantics before exposing stored outcomes.

The engineering cost is real: an extra persistence path, stronger transactions, retained outcomes, client changes, and race tests that are slower than a handler unit test. That cost is justified where a retry can charge twice, submit two jobs, create duplicate orders, or trigger another irreversible effect. Where the effect is naturally replaceable or safely repeatable, use the simpler HTTP contract and test it honestly.