A replay that calls production is not a replay

The refund agent told a customer that money had been sent after its payment tool timed out. By the time an engineer reruns the same prompt, the refund record exists and the tool returns success. The test passes because the world changed, not because the defect was fixed.

Production incidents are snapshots of a distributed system. The prompt is only one input. Tool results, retries, identity, policy version, conversation state, clock, deployment, and partial side effects can all decide the outcome. Replaying against today's services replaces those facts with new ones and turns a regression test into a fresh integration test.

A trace helps because it records the workflow's observable path. OpenAI describes trace grading as assigning structured labels or scores to an end-to-end record of decisions, tool calls, and reasoning steps. Grading can identify where behavior broke. Replay does something different: it re-executes the relevant workflow with controlled evidence and checks whether the failure boundary still exists. A scored trace is not automatically an executable fixture.

The engineer's job is to preserve enough of the incident to falsify the proposed fix without preserving secrets or repeating damage. That means freezing the facts that mattered, defining the invariant that failed, and proving that the old implementation fails before accepting a green result from the new one.

Turn the trace into a case, not a transcript dump

Begin with the failure statement. "The answer was bad" cannot drive a replay. "After the refund tool timed out, the agent claimed the refund completed even though no successful tool result confirmed it" identifies an observable boundary. It names the trigger, the prohibited outcome, and the evidence needed to decide.

Next, walk backward through the trace. Capture the user-visible input after approved redaction, the conversation messages that changed the decision, the ordered tool calls and returned values, the tool errors and attempt IDs, the workflow and prompt versions, the model configuration that can be retained, the authorization claims, the locale and time zone, and the clock value when time affected policy. Omit unrelated spans. More data is not automatically more faithful.

Store provenance separately from content. The fixture should say which incident and trace it came from, who approved the redaction, when the expected outcome was reviewed, and which schema version describes it. Access to the original protected trace can remain restricted. Most CI users need a synthetic case ID and the decisive redacted evidence, not the customer's identity.

Do not copy secrets from tool arguments. Replace account numbers, tokens, emails, and free-text identifiers with stable synthetic values. Preserve relationships that matter. If two calls used the same idempotency key, both synthetic calls must still share one key. If ownership differed between requester and order, the replacement identities must remain different. Redaction that erases the relation erases the bug.

The fixture below captures a refund timeout without claiming to reproduce an OpenAI internal schema. It is an application-owned, portable format. The validator is runnable with the Python standard library and rejects missing evidence before a model or workflow is called.

import json
import sys
from datetime import datetime
from pathlib import Path

REQUIRED_TOP_LEVEL = {
    "schema_version", "case_id", "incident_id", "workflow_version",
    "prompt_version", "frozen_at", "input", "identity", "tool_events",
    "expected"
}
REQUIRED_EVENT = {"attempt", "tool", "arguments", "outcome"}

def validate(case: dict) -> list[str]:
    errors: list[str] = []
    missing = REQUIRED_TOP_LEVEL - case.keys()
    if missing:
        errors.append(f"missing top-level fields: {sorted(missing)}")
        return errors

    try:
        datetime.fromisoformat(case["frozen_at"].replace("Z", "+00:00"))
    except ValueError:
        errors.append("frozen_at must be an ISO 8601 timestamp")

    attempts: set[int] = set()
    for index, event in enumerate(case["tool_events"]):
        event_missing = REQUIRED_EVENT - event.keys()
        if event_missing:
            errors.append(f"tool_events[{index}] missing {sorted(event_missing)}")
            continue
        if event["attempt"] in attempts:
            errors.append(f"duplicate tool attempt {event['attempt']}")
        attempts.add(event["attempt"])
        if event["outcome"]["type"] not in {"result", "error"}:
            errors.append(f"tool_events[{index}] has unsupported outcome type")

    if not case["expected"].get("prohibited_claims"):
        errors.append("expected.prohibited_claims must name the incident boundary")
    return errors

path = Path(sys.argv[1])
fixture = json.loads(path.read_text(encoding="utf-8"))
problems = validate(fixture)
if problems:
    print("\n".join(problems), file=sys.stderr)
    raise SystemExit(1)
print(f"valid replay fixture: {fixture['case_id']}")

The expected object should describe invariants, not a preferred paragraph. For the refund case, the prohibited claim is that the refund completed. A required behavior may be acknowledging that completion is unconfirmed and offering an appropriate next step. Exact wording is irrelevant unless the product has a mandated disclosure.

Keep trace grading labels as evidence, not unquestioned truth. If an automated grader found the incident, a qualified reviewer should confirm the boundary before it becomes a permanent regression. Save the grader version and result so future investigators understand how the case entered the suite, but let the approved product rule own the expected outcome.

The validator output should make fixture health distinct from behavior. A healthy line names the case ID and confirms that the fixture schema is valid. It does not say the incident passed. A broken fixture names the missing field or malformed event and stops before workflow execution. A misleading harness often prints only a green test name, leaving no case ID, fixture hash, implementation version, or boundary result. That line proves the test process exited successfully, but not that it exercised the preserved incident.

For each executed case, read comparability fields before the verdict. The old and current runs must name the same case ID and fixture hash. The old boundary value should show the prohibited behavior, while the current value should show that the same boundary now holds. If the fixture hashes differ, two green and red values are not a transition. If the old run is missing, a current pass is merely a baseline. If the current run reports a pass but recorded events remain unconsumed, the workflow may have returned before reaching the incident path. These states need separate labels instead of one generic success flag.

Freeze tools, time, identity, and sequence

Tool replay must preserve order as well as values. A timeout followed by a successful retry is different from a success followed by an unrelated timeout. Two identical JSON results can represent different side effects when idempotency keys or attempt IDs differ. Build a scripted fake that consumes expected calls in sequence and fails on an unexpected tool, argument, or extra call.

The harness below is complete and runnable. ScriptedTools returns recorded outcomes or raises the recorded error. RefundWorkflow is a deliberately small stand-in for the orchestration layer under test. In a real repository, keep the harness and replace that class with an adapter to the production workflow. The fixture does not contact a payment service.

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime
from typing import Any

@dataclass(frozen=True)
class ToolEvent:
    tool: str
    arguments: dict[str, Any]
    outcome_type: str
    value: dict[str, Any] | None = None
    error: str | None = None

class RecordedToolError(RuntimeError):
    pass

class ScriptedTools:
    def __init__(self, events: list[ToolEvent]) -> None:
        self._events = list(events)
        self.calls: list[tuple[str, dict[str, Any]]] = []

    def call(self, tool: str, arguments: dict[str, Any]) -> dict[str, Any]:
        if not self._events:
            raise AssertionError(f"unexpected extra tool call: {tool} {arguments}")
        expected = self._events.pop(0)
        if (tool, arguments) != (expected.tool, expected.arguments):
            raise AssertionError(
                "tool replay mismatch: "
                f"expected {expected.tool} {expected.arguments}, "
                f"got {tool} {arguments}"
            )
        self.calls.append((tool, arguments))
        if expected.outcome_type == "error":
            raise RecordedToolError(expected.error or "recorded tool error")
        return expected.value or {}

    def assert_consumed(self) -> None:
        if self._events:
            remaining = [event.tool for event in self._events]
            raise AssertionError(f"replay ended before recorded calls: {remaining}")

class FrozenClock:
    def __init__(self, value: datetime) -> None:
        self._value = value

    def now(self) -> datetime:
        return self._value

class RefundWorkflow:
    def __init__(self, tools: ScriptedTools, clock: FrozenClock) -> None:
        self.tools = tools
        self.clock = clock

    def handle(self, refund_id: str, owner_verified: bool) -> str:
        if not owner_verified:
            return "I need to verify ownership before checking the refund."
        try:
            result = self.tools.call("refund.status", {"refund_id": refund_id})
        except RecordedToolError:
            return "I could not confirm the refund status. Please try again later."
        if result.get("status") == "completed":
            return "The refund is complete."
        return "The refund is not yet confirmed as complete."

This harness fails loudly when the current workflow takes a different path from the recorded one. That difference may be the fix, or it may mean the fixture no longer reaches the intended boundary. Review the mismatch. Do not update the recorded sequence automatically just to restore green CI.

Freeze time through an injected clock, not by changing the workstation clock. Time can control refund eligibility, token expiry, market hours, locale-specific dates, and policy rollouts. Store an offset-aware timestamp and the relevant time zone when local calendar rules matter. A fixed UTC instant alone may not capture a daylight-saving or midnight boundary.

Freeze identity as claims, not a production session cookie. Record synthetic subject, tenant, roles, ownership relation, and any verified state used by authorization. The replay should never depend on a live login or reusable credential. If the bug involved stale authorization, preserve the stale claim and the resource relationship in an isolated fake.

Separate a remote timeout from local cancellation

Two incident logs can both end with "tool timeout" while describing different failures. In one, the outbound request reached the dependency and no response returned before the tool's wait ended. In the other, the parent workflow exhausted its own deadline or was cancelled before the dependency completed. A wrapper may normalize both exceptions into the same error text. Replaying both as one generic RecordedToolError reproduces the visible branch but discards the evidence needed to choose a fix.

The remote-timeout case points toward dependency latency, client waiting policy, or idempotent recovery after an uncertain result. The local-cancellation case points toward orchestration budgets, queue delay before the call, or propagation of caller cancellation. Increasing the tool timeout cannot help when the parent has already spent its budget. Increasing the parent budget may conceal a dependency that never responds. The log suffix is identical, but the owners and safe mitigations differ.

Use timing and dispatch evidence to separate them. Record when the parent budget began, when the tool attempt was dispatched, whether the dependency observed the attempt, and which boundary initiated cancellation. A dependency receipt or test-environment request record shows that the call crossed the process boundary. Its absence is not conclusive by itself because logging can fail, so compare it with client-side connection evidence and the tool span. If the parent deadline occurs before dispatch, the dependency is not the root cause. If dispatch and dependency receipt both precede the deadline but no response arrives, a remote timeout is supported. If a response exists after the parent cancellation, the replay also needs to preserve how late results are discarded.

Wall-clock timestamps can mislead when machines have clock skew or a queue delay is missing from the trace. A healthy diagnostic places the tool attempt inside the parent's available budget and identifies the event that ended it. A broken remote call shows dispatch plus dependency receipt, followed by the tool wait expiring. A broken local budget shows the parent deadline ending first, often with no completed tool result. A row that contains only an error string and total duration looks detailed but cannot distinguish either cause.

Preserving this distinction costs more than recording one exception. The harness needs a controlled deadline source, a representation of dispatch state, and fixtures for late completion. Timing-sensitive tests also become brittle if they depend on real sleeping. Reserve schedule-level fidelity for incidents where cancellation order changed the outcome, and keep ordinary tool failures as fast scripted events.

Work three different incidents through the harness

The refund timeout is the first class: a changed external state masks a false completion claim. The original trace shows a timeout, while today's payment record shows completed. The regression must feed the recorded timeout to the old workflow and observe the prohibited completion claim. The fixed workflow must instead say that completion is unconfirmed. A separate live integration test can verify the current payment API contract, but it cannot prove this incident fix.

The second class is duplicate side effects. An agent calls shipment.create, receives a transport error after the service accepted the request, and retries without the original idempotency key. Both log lines may show the same arguments except for the key, and the final customer answer may look correct. The failure is two created shipments, not bad prose.

Model the first call as an error with an accepted-side-effect marker in the simulator, then require the retry to reuse the same synthetic idempotency key. Assert the sequence and the simulator's final shipment count. An output grader alone cannot see this defect. The replay oracle belongs at the tool boundary.

The third class is a policy and time boundary. An account-recovery agent hands a request to a human during staffed hours but gives self-service instructions after hours. The incident occurred near a local clock change with an old policy version. Replaying only the user message under the current policy and current time may take a valid but different branch.

Freeze the local instant, policy content, user locale, and handoff availability. Assert the chosen action, not a verbatim response. Add nearby controls just before and after the boundary so a fix cannot simply force every request into human handoff. This is where a single incident becomes a robust regression family.

Pytest fixtures are useful because they make the controlled dependencies explicit and reusable. The test below exercises the timeout case and a successful control. It also verifies that all recorded tool events were consumed, so a workflow that returns early cannot pass by accident.

from datetime import datetime, timezone

import pytest

from replay_harness import (
    FrozenClock,
    RefundWorkflow,
    ScriptedTools,
    ToolEvent,
)

@pytest.fixture
def frozen_clock() -> FrozenClock:
    return FrozenClock(datetime(2026, 7, 18, 9, 30, tzinfo=timezone.utc))

def test_timeout_never_claims_refund_completed(frozen_clock: FrozenClock) -> None:
    tools = ScriptedTools([
        ToolEvent(
            tool="refund.status",
            arguments={"refund_id": "refund_fixture_17"},
            outcome_type="error",
            error="recorded timeout",
        )
    ])
    workflow = RefundWorkflow(tools, frozen_clock)

    answer = workflow.handle("refund_fixture_17", owner_verified=True)

    assert "complete" not in answer.lower() or "could not confirm" in answer.lower()
    assert "could not confirm" in answer.lower()
    tools.assert_consumed()

def test_confirmed_refund_can_be_reported(frozen_clock: FrozenClock) -> None:
    tools = ScriptedTools([
        ToolEvent(
            tool="refund.status",
            arguments={"refund_id": "refund_fixture_18"},
            outcome_type="result",
            value={"status": "completed"},
        )
    ])
    workflow = RefundWorkflow(tools, frozen_clock)

    answer = workflow.handle("refund_fixture_18", owner_verified=True)

    assert answer == "The refund is complete."
    tools.assert_consumed()

The first assertion includes the decisive phrase rather than merely checking for any refusal. A weak test such as assert answer would pass the original incident. The success control prevents a blunt fix that never reports completion, even when the recorded evidence confirms it.

For a real model-backed adapter, do not require exact prose unless the product requires it. Assert structured tool calls, prohibited claims, required disclosures, or a calibrated task-specific grade. Preserve each raw outcome. If the candidate varies across executions, report the variation and apply a predeclared policy. Retrying until one answer passes destroys the evidence.

Distinguish a code fix from a changed world

A credible replay starts red. Run the frozen case against the last known affected implementation and confirm that it reproduces the prohibited behavior. If it passes, stop. The fixture may be incomplete, the affected version may be wrong, or the incident may depend on evidence you did not preserve. Calling that result "fixed" would be circular.

Use a two-by-two comparison when environment drift is plausible:

The table is a reasoning aid, not a demand to call production. "Current live-like" should be a safe contract fixture or isolated integration environment. The important comparison is original versus current implementation on the same incident fixture. Only that cell change isolates the proposed fix.

Model availability can complicate historical replay. Pin the exact model identifier and supported settings when they remain available, but do not claim that a seed or temperature makes a remote model perfectly deterministic. If the original model is no longer callable, classify the run as a compatibility replay. You can still test orchestration and policy invariants, but you cannot claim byte-for-byte reproduction of the original model behavior.

Prompt and policy versions need the same care. "Current prompt passes old incident" is useful. "Old prompt passes after we silently replaced the policy text" is not a reproduction. Store immutable copies or content hashes for every decision-bearing template. When a protected policy cannot be copied into the test repository, resolve a versioned approved fixture through a controlled test-data service.

Schema migration can masquerade as a fix too. Suppose the original tool returned refund_status, while the current adapter exposes status. A replay fixture silently rewritten to the new field may bypass the faulty compatibility path that caused the incident. Keep the incident fixture in its original application-visible schema and apply the same versioned adapter that production used. Add a separate current-schema control. When the old fixture no longer parses, the test should fail with an adapter error, not be regenerated from today's response.

Compare side effects as well as messages. A workflow can stop making the prohibited claim while still issuing the duplicate refund, sending the email twice, or mutating state before authorization. Your result manifest should carry boundary outcomes from the simulator, such as action count and authorization decision, alongside any output grade. A prose-only green result does not verify an operational incident.

The comparison script below fails when a result manifest omits versions or when the old and new runs do not establish the expected red-to-green transition. It checks explicit booleans rather than interpreting prose.

import json
import sys
from pathlib import Path

old = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
new = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))

required = {"case_id", "fixture_hash", "implementation_version", "boundary_passed"}
for label, result in (("old", old), ("new", new)):
    missing = required - result.keys()
    if missing:
        raise SystemExit(f"{label} result missing fields: {sorted(missing)}")

if old["case_id"] != new["case_id"] or old["fixture_hash"] != new["fixture_hash"]:
    raise SystemExit("results are not comparable: case or fixture hash changed")
if old["boundary_passed"]:
    raise SystemExit("original implementation did not reproduce the incident")
if not new["boundary_passed"]:
    raise SystemExit("current implementation still violates the incident boundary")

print(
    "verified red-to-green replay",
    old["implementation_version"],
    "->",
    new["implementation_version"],
)

Near misses deserve review. The current implementation may avoid the false claim but introduce a new one, skip the tool entirely, or send every request to a human. Add controls around the incident boundary and run the normal regression suite. One green incident case proves one boundary, not the health of the product.

Move from incident response into the regular suite

During the incident, create a quarantined working fixture with restricted access. Confirm the failure quickly. After containment, produce the reviewed redacted fixture, assign owners, and add nearby controls. Record the incident link, failure statement, affected versions, fixed version, and the test that now covers it. That turns an emergency artifact into maintainable test data.

An existing suite needs compatibility work before incident cases become release gates. Land the fixture schema validator and production-workflow adapter first. Make both accept the current reviewed schema, then add one synthetic control for each dependency type: success, explicit rejection, timeout before a known result, and uncertain completion where the side effect may have occurred. Only after those controls fail in understandable ways should the redacted incident fixture enter a nonblocking lane.

The first migration failures usually expose hidden dependencies in old tests. A helper reads the real clock, a tool fake falls through to a live client, cases share mutable state, or an expected string stands in for the actual product invariant. Fix those harness defects before changing the incident expectation. Otherwise a network outage or yesterday's cached record can be mistaken for successful replay fidelity.

Shadow results should show four facts together: the fixture validates, the affected implementation reproduces the boundary, the current implementation passes the identical fixture, and adjacent controls still pass. When the affected build can no longer run, label the missing red baseline explicitly and require stronger trace evidence plus reviewer approval. Do not manufacture a red result by rewriting the fixture around a toy implementation that never shipped.

Promote deterministic cases first. Provider-backed compatibility checks should remain a separate lane with their own retry policy and service owner. This ordering keeps a remote evaluator outage from blocking evidence that the orchestration fix works. It also reveals the change's cost: a realistic simulator and historical adapters add code that must change when tool schemas evolve, while the separate compatibility lane adds CI minutes and external spend.

Cross-team ownership should be recorded at the boundary, not assigned to "AI" as a group. The incident owner approves the prohibited outcome. The workflow team owns the replay adapter and proposed fix. The dependency team confirms whether a recorded error occurred before acceptance, after acceptance, or under an unknown result. QA owns fixture validation, controls, and suite policy. Security or privacy approves redaction and access. The handoff should contain the case ID, fixture hash, affected and current versions, ordered redacted events, frozen identity and time facts, old and new boundary manifests, any unconsumed events, and the exact first divergence. That packet lets another team reproduce the claim without opening the original customer trace.

Keep fast replay tests local to the repository and run them on every relevant change. Tool fakes and frozen clocks should not need network access. Provider-backed compatibility runs are slower, cost money, and can vary, so schedule them or run them on prompt and model changes. Both layers matter, but a flaky remote call should not hide a deterministic orchestration regression.

CI should fail distinctly on fixture validation, reproduction, and current regression. The workflow below uses a sanitized fixture pack and uploads only redacted result manifests. It does not install or call a live production integration.

name: ai-incident-replays

on:
  pull_request:
    paths:
      - "agent/**"
      - "prompts/**"
      - "replays/**"

jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Validate redacted fixtures
        run: python scripts/replays/validate_all.py replays/incidents
      - name: Run deterministic replay boundaries
        run: pytest -q tests/replays
      - name: Check fixture provenance and expiry
        run: python scripts/replays/check_metadata.py replays/incidents
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: redacted-replay-results
          path: artifacts/replays/*.json

Set an expiry on temporary environment captures, not on the regression itself. The permanent case can remain after raw incident data is deleted because its approved synthetic fixture preserves the behavior. Re-review cases when policy changes. An old expected refusal may become a defect under a new product rule, and the update should be explicit.

Replay has costs. Building realistic tool simulators takes engineering time. Frozen historical policy adds maintenance. Provider-backed runs add latency and spend. Strong redaction can remove decisive context. Exact sequence assertions can become brittle when a harmless orchestration refactor changes call order. Prefer invariants at the narrowest boundary that represents the incident, and keep full sequence checks only where order caused the damage.

Know when replay would create more risk than evidence

Do not replay a destructive action against production. Refunds, messages, account changes, purchases, and permission updates belong behind fakes, sandboxes, or provider test modes that the owning service documents. A request header you hope is a dry-run flag is not protection unless the service guarantees it.

Avoid storing an incident when safe redaction cannot preserve the decisive condition. Medical text, private code, credentials, or uniquely identifying conversations may not belong in a general test repository. Create a minimal structural fixture in an approved environment or mark the case as non-replayable. Missing evidence must remain visible; it is not a pass.

Do not force an exact-output oracle onto a genuinely nondeterministic language task. That produces flaky tests and encourages harmless wording over the real boundary. Test the action, claims, citations, policy decision, or tool sequence. Use expert review for qualities that cannot yet be automated responsibly.

Skip replay as the only response to a security incident. Preserve evidence under the incident process, rotate exposed secrets, and test exploits in an isolated environment with the security owner. A regression case is valuable after containment, but it is not containment.

Finally, do not let a green replay close an incident by itself. Verify surrounding controls, monitoring, and rollout behavior. The replay proves that one frozen failure boundary changed under controlled conditions. That is a strong claim when made honestly, and a dangerously broad one when stretched into "the system is fixed."

Replay does not catch a concurrency failure whose decisive interleaving was never captured. Two workers may read the same state, both pass an authorization or idempotency check, and then write in an order that a sequential tool script cannot express. A green sequential fixture says nothing about that race. Preserve the competing operations and their ordering constraints, then use a controlled concurrency test or state-machine simulator owned by the affected service. That deeper harness costs execution time and maintenance, so apply it to incidents where overlapping actions are part of the evidence rather than slowing every language-level replay.