A failed agent trace is evidence, not yet an eval case

An agent says a calendar event was booked once, but the attendee receives two invitations. The trace shows a tool timeout, a retry, and two successful side effects. Copying only the user prompt and final answer into an eval dataset would preserve the symptom and delete the cause.

Production traces are valuable because they show decisions and tool activity that black-box outputs cannot. They are also noisy, sensitive, and tied to one runtime. The engineering work is to turn that evidence into a small, controlled case without rewriting history.

Treat the trace as incident evidence first

A trace is an end-to-end record of an agent run, often represented as parent and child spans. Depending on instrumentation, spans may cover model calls, handoffs, tools, guardrails, retrieval, and custom application work. Trace grading can help label behavior across that record, but a grade does not automatically create a reproducible test.

Begin with triage. Confirm the user-visible failure and the supported product behavior. A user complaint, an error span, and a low grader score are leads, not interchangeable truth. The tool may have succeeded despite a client timeout. The final answer may be correct even though an internal span reported a recovered error. The workflow may be unsupported, in which case the trace belongs in product discovery rather than a regression suite.

Identify the earliest decision or event that made the bad outcome likely. The last failing span is often downstream damage. In the duplicate-invitation example, the second successful tool call creates the duplicate, but the decisive defect may be retrying a non-idempotent operation without checking the first result. A dataset row that begins after that decision cannot test the fix.

Record a short incident statement before transforming data:

• user intent and relevant preconditions;
• observed outcome;
• expected supported outcome;
• earliest causal evidence;
• failure family;
• risk or severity rationale;
• trace provenance and access classification;
• reviewer and review date.

This statement prevents a common curation error: selecting whatever span is easiest to export rather than the evidence needed to reproduce the behavior.

Trace completeness must be checked too. Parent-child links should resolve for the relevant path. Tool attempts need distinguishable identities. Arguments, results, errors, and status should be present where the test depends on them. Clock timestamps can help order events, but distributed clocks and asynchronous work can make naive timestamp sorting misleading. Prefer explicit parentage, sequence fields, and application event relationships when available.

If a necessary tool span is missing, do not infer that the tool was never called. Mark the trace incomplete and fix instrumentation. Turning missing telemetry into a negative assertion creates a test for the collector, not the agent.

The first code example walks a normalized span tree and returns the ancestry of a chosen causal span. It validates duplicate and missing parent IDs so a broken trace cannot quietly produce a partial fixture.

type SpanStatus = "ok" | "error" | "cancelled";

type TraceSpan = {
  id: string;
  parentId: string | null;
  kind: "agent" | "model" | "tool" | "guardrail" | "custom";
  name: string;
  status: SpanStatus;
  sequence: number;
};

export function causalPath(spans: TraceSpan[], causalSpanId: string): TraceSpan[] {
  const byId = new Map<string, TraceSpan>();
  for (const span of spans) {
    if (byId.has(span.id)) throw new Error(`Duplicate span id: ${span.id}`);
    byId.set(span.id, span);
  }

  const path: TraceSpan[] = [];
  const visited = new Set<string>();
  let current = byId.get(causalSpanId);
  if (!current) throw new Error(`Unknown causal span: ${causalSpanId}`);

  while (current) {
    if (visited.has(current.id)) throw new Error(`Cycle at span: ${current.id}`);
    visited.add(current.id);
    path.push(current);

    if (current.parentId === null) break;
    const parent = byId.get(current.parentId);
    if (!parent) throw new Error(`Missing parent ${current.parentId} for ${current.id}`);
    current = parent;
  }

  return path.sort((left, right) => left.sequence - right.sequence);
}

The selected path is a starting point, not the whole fixture. A sibling tool result may explain why a retry was unsafe. A retrieval span may contain the stale document that led to a wrong argument. Add neighboring evidence when it changes the expected decision, and document why it is retained.

Sanitize by schema without destroying the failure

Production traces can contain names, email addresses, account identifiers, message content, access tokens, tool credentials, proprietary documents, medical or financial details, and internal system prompts. Copying them into a broadly accessible test repository expands their exposure and retention.

Follow the organization’s data policy before export. Confirm the trace may be used for evaluation, where the resulting case may live, who can review it, and how long provenance remains accessible. Technical redaction does not replace authorization.

Use an allowlist transformation. Define which fields the eval needs and construct a new object from them. A denylist that removes fields named password or email misses secrets inside free text, nested errors, headers, URLs, and tool output. Never serialize the full production object and hope a later regex catches everything.

Typed placeholders preserve semantics better than [REDACTED]. An email can become person_1@example.test, an account can become ACCOUNT_A, and a credential can become SECRET_REMOVED. Stable placeholders let the case retain relationships, such as the same user appearing in input and tool output, without retaining the original value.

Dates, locale, currency, and time zone may be causal. Replacing every date with one constant can erase a daylight-saving boundary. Replacing rupees with dollars can change formatting or policy behavior. Preserve the category and relationship needed by the test, then use synthetic values reviewed for safety.

The next TypeScript creates an allowlisted tool-attempt fixture for the duplicate-booking failure. It deliberately rejects unknown tool names and maps production identifiers to stable synthetic values.

type RawToolAttempt = {
  toolName: string;
  arguments: Record<string, unknown>;
  outcome: "success" | "timeout" | "error";
  sideEffectId?: string;
  errorMessage?: string;
};

type CalendarAttempt = {
  tool: "create_calendar_event";
  attendee: string;
  startsAt: string;
  idempotencyKey: string | null;
  outcome: RawToolAttempt["outcome"];
  syntheticSideEffect: string | null;
};

function requiredString(
  value: unknown,
  field: string,
): string {
  if (typeof value !== "string" || value.length === 0) {
    throw new Error(`Missing string field: ${field}`);
  }
  return value;
}

export function sanitizeCalendarAttempt(
  raw: RawToolAttempt,
  sideEffectMap: Map<string, string>,
  idempotencyMap: Map<string, string>,
): CalendarAttempt {
  if (raw.toolName !== "create_calendar_event") {
    throw new Error(`Unexpected tool: ${raw.toolName}`);
  }

  const originalSideEffect = raw.sideEffectId ?? null;
  let syntheticSideEffect: string | null = null;
  if (originalSideEffect !== null) {
    if (!sideEffectMap.has(originalSideEffect)) {
      sideEffectMap.set(originalSideEffect, `EVENT_${sideEffectMap.size + 1}`);
    }
    syntheticSideEffect = sideEffectMap.get(originalSideEffect) ?? null;
  }

  requiredString(raw.arguments.startsAt, "startsAt");
  const originalKey =
    typeof raw.arguments.idempotencyKey === "string"
      ? raw.arguments.idempotencyKey
      : null;
  let syntheticKey: string | null = null;
  if (originalKey !== null) {
    if (!idempotencyMap.has(originalKey)) {
      idempotencyMap.set(originalKey, `KEY_${idempotencyMap.size + 1}`);
    }
    syntheticKey = idempotencyMap.get(originalKey) ?? null;
  }

  return {
    tool: "create_calendar_event",
    attendee: "attendee_1@example.test",
    startsAt: "2030-04-18T10:00:00Z",
    idempotencyKey: syntheticKey,
    outcome: raw.outcome,
    syntheticSideEffect,
  };
}

This schema does not preserve the raw error message because the example contract does not need it and free-text errors are a common leak path. A different failure may require a normalized error category such as TIMEOUT_AFTER_DISPATCH. Map that category during reviewed transformation instead of retaining an arbitrary production string.

Run secret scanning and structured validation after transformation, but keep human review. Automated detection has false negatives. The reviewer should compare the sanitized case with the restricted source trace and confirm both that sensitive content is gone and that causal meaning remains.

Convert behavior into a stable replay contract

An eval row needs an input state, controlled dependencies, expected behavior, and grading rules. Raw spans contain observed events, including the bug. They do not define what a corrected system should do.

For the duplicate invitation, a replay fixture might make the first tool attempt return a timeout-shaped result while recording that the side effect occurred. The expected behavior could allow status reconciliation or a safe recovery response, while forbidding an unguarded second create operation. The exact contract depends on the product’s supported recovery design. Do not freeze the buggy retry sequence as the expected trace.

Replace external dependencies with controlled fixtures at the boundary the test intends to exercise. If the goal is agent retry behavior, use a deterministic fake calendar service that can model “side effect committed, response timed out.” If the goal is the real calendar integration, an offline eval dataset is insufficient; use an integration environment with its own cleanup and authorization.

Remove volatile assertions. Runtime span IDs, timestamps, generated request IDs, latency, and natural-language phrasing usually change between runs. Assert tool name, normalized arguments, attempt count, side-effect relationship, terminal status, or a narrow semantic outcome when those express the defect.

The following case contract keeps both an expected rule and a forbidden event. Its validator works on a normalized replay trace and gives direct failure reasons.

type ReplayAttempt = {
  tool: string;
  outcome: "success" | "timeout" | "error";
  sideEffect: "created" | "none" | "unknown";
};

type EvalCase = {
  id: string;
  version: number;
  failureFamily: string;
  toolAttempts: ReplayAttempt[];
  expected: {
    maximumCreateCalls: number;
    allowedTerminalStates: string[];
  };
};

type ReplayObservation = {
  caseId: string;
  createCalls: number;
  terminalState: string;
};

export function gradeReplay(
  testCase: EvalCase,
  observed: ReplayObservation,
): { passed: boolean; reasons: string[] } {
  if (testCase.id !== observed.caseId) throw new Error("Case ID mismatch");
  const reasons: string[] = [];

  if (observed.createCalls > testCase.expected.maximumCreateCalls) {
    reasons.push(
      `createCalls=${observed.createCalls} exceeds ${testCase.expected.maximumCreateCalls}`,
    );
  }
  if (!testCase.expected.allowedTerminalStates.includes(observed.terminalState)) {
    reasons.push(`terminalState=${observed.terminalState} is not allowed`);
  }

  return { passed: reasons.length === 0, reasons };
}

Add a positive control. A timeout before any side effect may legitimately permit a retry, while a timeout after an uncertain dispatch may require reconciliation. Testing only the production failure can lead to a fix that disables every retry and harms recoverability. The paired case proves the boundary.

Worked example: the wrong tool succeeds perfectly

A travel agent uses search_hotels when the user asked to change an existing reservation. The search tool returns valid results, and the final response is fluent. Outcome-only grading might call the answer unhelpful without explaining why.

The trace exposes the wrong decision before execution. A dataset row should preserve the user’s modification intent, the available tool descriptions relevant to that decision, and the selected tool. The expected rule concerns tool selection or plan adherence. There is no need to replay a live hotel search.

Keep a nearby case where the user truly asks for alternatives. That case should allow search_hotels. Without it, a prompt fix could overfit by always preferring the modification tool whenever a reservation is mentioned.

Worked example: stale retrieval poisons a correct tool call

An agent retrieves an old refund policy, calculates a deadline from it, and sends a syntactically valid request to the refund tool. The tool rejects the request. A trace row focused only on the rejected call blames argument generation.

Preserve the retrieved document identity, version, and the policy fact used in the decision. The eval can pin a controlled corpus and require current evidence to appear before the tool call. This separates retrieval freshness from tool correctness. If the current document was present and the agent ignored it, the failure moves to evidence use rather than retrieval.

Worked example: a transient outage is not a model regression

An external service returns errors for every application during an incident. The agent follows the supported fallback and tells the user to try later. A trace-level failure alert may still flag the run because the original task did not complete.

Do not add the raw outage as a “must complete” model case. If recovery behavior matters, create a controlled dependency-failure case whose expected outcome is the supported fallback. Track the provider outage itself in operational reliability. This keeps the eval from training the team to hide honest failure states.

Detect duplicates and bad transformations before merge

Production volume is not dataset diversity. One retry bug can create thousands of near-identical traces. Adding all of them inflates the apparent size, slows evaluation, and lets one incident dominate aggregate results.

Create a normalized fingerprint from intent class, failure family, relevant tool sequence, expected rule, locale or policy branch when causal, and controlled dependency outcome. A matching fingerprint should trigger review, not automatic deletion. Two cases may share a tool path but cover different authorization or time boundaries.

Do not fingerprint raw text, timestamps, or production IDs. Those values make duplicates look unique. Do not use only the final answer either, because different causal paths can produce the same apology.

The script below checks stable case IDs, versions, provenance, and duplicate normalized fingerprints. It uses Node’s built-in crypto module and can run in CI.

import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";

type DatasetCase = {
  id: string;
  version: number;
  intentClass: string;
  failureFamily: string;
  toolSequence: string[];
  expectedRule: string;
  branch: string;
  provenance: { source: "production_trace"; restrictedTraceRef: string };
};

function fingerprint(row: DatasetCase): string {
  const normalized = JSON.stringify({
    intentClass: row.intentClass,
    failureFamily: row.failureFamily,
    toolSequence: row.toolSequence,
    expectedRule: row.expectedRule,
    branch: row.branch,
  });
  return createHash("sha256").update(normalized).digest("hex");
}

const path = process.argv[2];
if (!path) throw new Error("Usage: validate-trace-cases <cases.json>");
const rows = JSON.parse(await readFile(path, "utf8")) as DatasetCase[];
const ids = new Set<string>();
const fingerprints = new Map<string, string>();

for (const row of rows) {
  if (ids.has(row.id)) throw new Error(`Duplicate case id: ${row.id}`);
  ids.add(row.id);
  if (!Number.isInteger(row.version) || row.version < 1) {
    throw new Error(`Invalid version for ${row.id}`);
  }
  if (!row.provenance.restrictedTraceRef) {
    throw new Error(`Missing restricted provenance for ${row.id}`);
  }

  const key = fingerprint(row);
  const previous = fingerprints.get(key);
  if (previous) throw new Error(`Possible duplicate: ${previous} and ${row.id}`);
  fingerprints.set(key, row.id);
}

console.log(`validated=${rows.length} unique=${fingerprints.size}`);

A duplicate warning should send both cases to a curator. Keep the newer occurrence as additional incident evidence without necessarily making it another eval row. If it reveals a new locale, tool response, or recovery branch, update the normalized fields and label rationale to show why it is distinct.

Validate that placeholders resolve consistently, every expected rule has an oracle, controlled dependencies cover every referenced tool result, and no production endpoint is callable from replay. Run the new case against the known-bad product version when possible. It should reproduce the failure. Then run it against the proposed fix and its positive control. A case that never fails on the bad version may have lost the defect during sanitization.

Diagnose a case that will not reproduce

Compare the restricted source trace, sanitized fixture, and replay observation as three separate artifacts. Find the earliest point where they diverge. Looking only at the final answer encourages prompt tweaking when the fixture itself changed the input.

The command below prints the causal fields from a replay result. It assumes JSON artifacts with a normalized steps array.

jq -r '
  "case=\(.caseId) dataset=\(.datasetVersion) runner=\(.runnerVersion)",
  (.steps[]
    | "seq=\(.sequence) kind=\(.kind) name=\(.name) outcome=\(.outcome) side_effect=\(.sideEffect // "-")"),
  "terminal=\(.terminalState) grade=\(.grade.passed)",
  (.grade.reasons[]? | "reason=" + .)
' artifacts/replay-result.json

Illustrative diagnostic output for the duplicate-event case might be:

case=calendar-retry-004 dataset=agent-failures-12 runner=replay-7
seq=1 kind=tool name=create_calendar_event outcome=timeout side_effect=created
seq=2 kind=tool name=create_calendar_event outcome=success side_effect=created
terminal=reported_success grade=false
reason=createCalls=2 exceeds 1

A duplicated span can look like a duplicated side effect

The same replay-shaped log can come from a different root cause. A trace collector may receive one completed tool span twice, or an export job may append the same span again after losing its checkpoint. The normalized artifact then contains two create_calendar_event steps and the grader reports createCalls=2 exceeds 1. Nothing in that final reason proves the agent executed the tool twice.

If both copies retain one source span identity, the earlier completeness check should reject them. The deceptive form appears when normalization assigns a fresh span or row identity to each delivery. Those identities prove that two records were created, not that two tool attempts occurred.

Resolve this while the production evidence is still in its restricted system. A genuine double execution has two application attempt records that cross the tool boundary, even if both share one parent agent span. Where available, the downstream service audit trail also contains two requests or two mutation records. The attempts may have different side-effect identities, or the service may show the same logical operation applied twice. A collector duplicate points to one application attempt and one downstream mutation represented by two telemetry records.

Read identity before sequence. A healthy timeout-after-commit trace has one tool attempt, one committed side effect, and a later reconciliation or cautious terminal action. The broken agent trace has two distinct tool attempts and evidence that the second crossed the dependency boundary. The broken collector trace has the same application attempt relationship, identical timing and normalized payload, and only one dependency-side event. It may expose a repeated source identity, or it may hide that repetition behind newly assigned normalized identities. A misleading export may renumber both copies as sequence one and sequence two. Those tidy sequence values were assigned during export, so they cannot prove two executions.

The diagnostic line side_effect=created is also insufficient because it records a category, not an object relationship. Two rows that both say created could describe two calendar events, two observations of one event, or one inferred state copied into both rows. During triage, compare the restricted attempt identity, the synthetic relationship mapping, and the downstream mutation evidence. In the curated case, remove production identifiers but preserve cardinality: two real attempts become ATTEMPT_A and ATTEMPT_B; two real events become EVENT_A and EVENT_B. If telemetry duplicated one observation, collapse it or label it as collector evidence. Do not encode the duplicate observation as agent behavior.

This failure belongs to the observability pipeline, not the retry policy. The handoff should contain the restricted trace reference, collector and exporter versions, repeated source identities, the application attempt record, the dependency audit window, and the exact transformation step that introduced or retained the copy. It should not copy raw arguments into an ordinary issue. The observability owner can repair ingestion and backfill a corrected trace artifact. The eval curator then decides whether an instrumentation regression test is useful outside the behavioral dataset.

Keeping enough relational evidence to make this distinction costs schema complexity and review time. Curators must preserve which observations refer to the same attempt without retaining sensitive production identifiers. Downstream audit data may have a shorter retention period than traces, so investigation also becomes time-sensitive. Dropping all identities is cheaper, but it turns an apparently precise replay into a guess about cardinality.

If the replay shows side_effect=none for the first attempt, the fake does not model the production boundary. Fix the controlled dependency before changing the agent. If the production trace did not prove whether a side effect occurred, the case may need an unknown state and a contract for cautious reconciliation rather than asserting that creation happened.

Nondeterminism is another cause. The replay may select a different tool or phrasing on each attempt. Stabilize dependencies and grade behavior classes instead of exact text. Repeat only when variability is part of the risk, and store every attempt. Do not keep rerunning until the historical failure appears and then report that attempt alone.

Sanitization can remove a discriminator. If two accounts become the same placeholder, an authorization bug disappears. If two events receive one synthetic ID, duplicate detection becomes impossible. Review mappings for one-to-one and many-to-one relationships required by the failure.

A product change can make an old case unreachable. Decide whether the risk still exists through another path. Retire the case with a reason if the feature was removed. Update it through a new version if the contract remains but the interface changed. Do not force the runner to recreate obsolete screens or tools solely to keep the count stable.

Roll out a trace feedback loop without importing production noise

Keep intake separate from the main dataset. New failures enter a restricted quarantine queue with provenance and preliminary labels. Automated jobs can validate schema, scan for secrets, suggest duplicates, and run a sandbox replay. A human reviewer approves sanitized content and expected behavior before merge.

The CI workflow below validates only curated cases. It does not fetch production traces, which keeps credentials and restricted systems out of ordinary pull-request jobs.

name: agent-trace-eval-cases

on:
  pull_request:
    paths:
      - "evals/agent-failures/**"
      - "scripts/validate-trace-cases.ts"
      - "src/replay/**"

jobs:
  validate-and-replay:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Validate schema and duplicate fingerprints
        run: >-
          npx tsx scripts/validate-trace-cases.ts
          evals/agent-failures/cases.json
      - name: Run controlled replay
        run: >-
          npm run eval:agent-replay --
          --cases evals/agent-failures/cases.json
          --results artifacts/agent-replay.ndjson
      - name: Upload replay evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: agent-trace-replay
          path: artifacts/

Track the intake funnel: traces reviewed, rejected for policy, rejected as incomplete, merged as new cases, linked as duplicates, and routed to non-eval work. These are workflow counts, not claims about model quality. They show whether production evidence is becoming useful coverage or accumulating in a queue.

For a suite that already grades final answers, land replay support without changing the existing denominator. The result aggregator often assumes one output and one score per case. A trajectory case can instead end as dependency-fixture failure, incomplete trace, forbidden event, or valid behavioral pass. If those states are squeezed into the old pass or fail column, runner defects will appear as model regressions and missing tool evidence may appear as a clean pass.

Extend the result schema and its consumers before putting a trajectory row into the normal job. Add an evaluation-unit kind and preserve execution validity separately from the behavioral verdict. Keep the legacy answer-only view stable while a new view exposes trajectory states. Any consumer that does not understand a new state should reject it, not coerce it to pass, fail, or zero. This lands before dataset expansion because one unfamiliar status can otherwise distort every aggregate fed by the shared result export.

Artifact retention is the next compatibility problem. Existing suites may keep only final text and a scalar, while replay investigation needs controlled tool observations, relationship mappings, and grader reasons. Estimate the added artifact size, access scope, and retention period before enabling the slice. Pilot the trajectory cases in a nonblocking job, then watch invalid-state frequency, upload failures, queue time, and attempted network access. The rollout is working when legacy reports remain numerically unchanged, trajectory rows survive export and reimport without losing relationships, and removing the new slice restores the prior job without rewriting baseline results.

Cross-team ownership needs an explicit chain. The incident or product owner confirms the user-visible outcome and supported behavior. The application and observability owners establish what actually crossed the tool boundary. The privacy or data-governance owner confirms permitted use and retention. The integration owner specifies the controlled dependency states. The agent owner changes planning or retry behavior. The eval curator owns the sanitized fixture, oracle, positive control, and dataset version. One person may hold several roles, but none of the decisions should be left implicit.

A useful handoff includes the incident statement, restricted provenance pointer, access classification, causal and neighboring span identities, evidence-completeness assessment, authorized synthetic substitutions, controlled dependency transcript, expected and forbidden behavior, known-bad reproduction, positive-control result, and the named approver for the product contract. It should say whether the recipient is being asked to repair telemetry, define policy, build a fake, fix the agent, or approve a dataset row. Without that requested decision, the bundle is merely a large attachment moving between queues.

Version the dataset row when expected behavior, sanitization, controlled dependency behavior, or grading changes. Link related incidents without stuffing every occurrence into the fixture. Keep restricted trace access narrower and shorter-lived than the sanitized eval case where policy allows.

This process costs curator time, replay infrastructure, and secure storage. Minimization can reduce realism. Controlled tools can drift from production integrations. A growing regression set increases runtime and can overweight yesterday’s incidents. Review cases periodically for supported behavior and unique risk coverage.

The offline technique does not catch a duplicate created entirely inside the real dependency after one correct tool call. A calendar service might redeliver an internal command, a webhook consumer might apply an event twice, or concurrent workers might race in a way the deterministic fake never models. The agent replay can prove that the agent issued one allowed call under its controlled contract. It cannot certify the implementation behind that contract. Keep integration, concurrency, and downstream idempotency tests with the team that owns the live boundary.

Do not convert a trace that cannot be sanitized without erasing its cause. Keep the incident in its authorized system and create a synthetic case from the documented behavior if reviewers can do so safely. Do not export secrets to prove redaction works. Do not make a live destructive tool available to an eval runner.

The conversion is complete only when the new row fails for the original behavioral reason, passes for the reviewed fix, preserves no unnecessary production data, and adds a distinct boundary to the dataset. Anything less is either an incident attachment or a synthetic test with unproven provenance. Both may be useful, but neither should be mislabeled.