Find the first actionable fault in an agent trace

The last span says the final answer is wrong, so the dashboard assigns the defect to response generation. Ten events earlier, the agent sent a customer ID in the order_id field. The lookup returned an empty but valid response, and every later step merely carried that mistake forward.

Calling the last red span the cause creates bad bug reports. The useful question is narrower: which contract failed first on the causal path, and what evidence proves that its prerequisites passed? That answer can be tested without pretending a trace explains every reason behind the failure.

Localize contracts, not whichever span looks most broken

An agent trace records operations. Depending on the runtime, those operations can include model generations, tool calls, handoffs, guardrails, and custom spans. OpenAI's Agents SDK documents those categories and records parent relationships plus start and end times. None of that automatically defines your product's correctness obligations. QA still has to state what should be true at each boundary.

For a support workflow, the input-mapping contract might require an authenticated account ID to populate account_id and an order reference to populate order_id. The tool-request contract might require a currency when calculating a refund. The tool-result contract may require a success response to contain either an order or an explicit not-found status. The state-update contract can require the normalized order to be written under the key consumed by the next model turn. The final-answer contract can require the amount and status to match that state.

Evaluate each obligation as its own check and attach it to the span where the evidence exists. A check has an outcome such as pass, fail, or unknown. It also declares which checks must succeed before its result can be interpreted. The final-answer amount check depends on the state amount check, which depends on the tool result, which depends on the request mapping. That dependency chain is more informative than timestamp order.

“Earliest” means earliest in this contract graph, not the smallest wall-clock timestamp. Clocks can disagree, exporters can deliver late, and parallel branches can overlap. Parent and correlation relationships establish causality more reliably. If the normalized trace lacks enough identity to build the path, localization should stop with insufficient evidence.

“Actionable fault” is also more careful than “root cause.” A trace may prove that the request adapter put a customer ID in the wrong field. It may not show whether a developer typo, stale schema, configuration change, or model instruction caused that mapping. Assign the failing contract to the adapter owner and attach the evidence. Let code review and reproduction find the deeper implementation cause.

A span error flag is evidence, not a verdict. A network attempt can fail and be recovered by a retry. A tool can return HTTP success with semantically invalid content. A final evaluator can fail because its own reference data are stale. The localizer needs observations from contracts as well as runtime status.

Keep product failures separate from evaluation failures. If a grader times out, the run's answer does not become wrong. Record evaluation_unknown and rerun or investigate the grader. If a deterministic amount comparison fails, preserve the expected value, actual value, and source check. Do not ask a probabilistic grader to rediscover arithmetic that code can assert exactly.

The output should support several frontier findings. Two parallel tools can both receive invalid inputs. One branch may have an authorization defect while another has a stale cache. A single “root span” field forces a false choice. Return the minimal failing checks on each causal branch, then group them only when evidence shows a common failed ancestor.

Build a frontier oracle that can fail

The following code works on application-owned check records, not raw vendor spans. An adapter must normalize trace evidence into these checks. A finding is confirmed when it fails and all ancestors are known to pass. It is uncertain when no ancestor fails but at least one ancestor is unknown. A downstream failure with a failed ancestor is a symptom and stays out of the frontier.

from __future__ import annotations

from dataclasses import dataclass
from functools import lru_cache
from typing import Iterable, Literal


Outcome = Literal["pass", "fail", "unknown"]


@dataclass(frozen=True)
class Check:
    check_id: str
    span_id: str
    outcome: Outcome
    depends_on: tuple[str, ...]
    owner: str
    evidence: str


@dataclass(frozen=True)
class Finding:
    check_id: str
    span_id: str
    owner: str
    confidence: Literal["confirmed", "uncertain"]
    evidence: str


def localize_failure_frontier(checks: Iterable[Check]) -> list[Finding]:
    rows = list(checks)
    by_id = {check.check_id: check for check in rows}
    if len(by_id) != len(rows):
        raise ValueError("duplicate check id")

    for check in rows:
        if check.outcome not in {"pass", "fail", "unknown"}:
            raise ValueError(
                f"{check.check_id} has unsupported outcome: {check.outcome}"
            )
        missing = sorted(set(check.depends_on) - set(by_id))
        if missing:
            raise ValueError(f"{check.check_id} has missing dependencies: {missing}")

    visiting: set[str] = set()

    @lru_cache(maxsize=None)
    def ancestors(check_id: str) -> frozenset[str]:
        if check_id in visiting:
            raise ValueError(f"dependency cycle at {check_id}")
        visiting.add(check_id)
        result: set[str] = set()
        for dependency in by_id[check_id].depends_on:
            result.add(dependency)
            result.update(ancestors(dependency))
        visiting.remove(check_id)
        return frozenset(result)

    for check_id in by_id:
        ancestors(check_id)

    findings: list[Finding] = []
    for check in rows:
        if check.outcome != "fail":
            continue
        prior = [by_id[item] for item in ancestors(check.check_id)]
        if any(item.outcome == "fail" for item in prior):
            continue
        confidence = "uncertain" if any(
            item.outcome == "unknown" for item in prior
        ) else "confirmed"
        findings.append(
            Finding(
                check_id=check.check_id,
                span_id=check.span_id,
                owner=check.owner,
                confidence=confidence,
                evidence=check.evidence,
            )
        )

    return sorted(findings, key=lambda item: item.check_id)

The oracle does not trust an expected category stored in the same fixture. It derives the frontier from outcomes and dependencies. It traverses every check before selecting failures, so a cycle made entirely of pass or unknown outcomes is still rejected. Change an upstream request check from pass to fail and the downstream answer failure disappears from the frontier. Change a prerequisite to unknown and confidence changes. Remove a dependency record and the input is rejected rather than silently treated as complete.

These tests cover a downstream symptom, a recovered error, two independent faults, and missing evidence. Each fixture can be mutated in a way that changes the result.

import pytest

from failure_frontier import Check, localize_failure_frontier


def check(
    check_id: str,
    outcome: str,
    depends_on: tuple[str, ...] = (),
    *,
    span_id: str | None = None,
    owner: str = "support-agent",
) -> Check:
    return Check(
        check_id=check_id,
        span_id=span_id or f"span-{check_id}",
        outcome=outcome,
        depends_on=depends_on,
        owner=owner,
        evidence=f"observed {check_id}={outcome}",
    )


def test_bad_argument_is_frontier_and_bad_answer_is_a_symptom() -> None:
    checks = [
        check("input-authenticated", "pass"),
        check("order-id-mapped", "fail", ("input-authenticated",), owner="request-adapter"),
        check("tool-result-useful", "fail", ("order-id-mapped",), owner="order-service"),
        check("final-answer-correct", "fail", ("tool-result-useful",)),
    ]

    findings = localize_failure_frontier(checks)

    assert [(item.check_id, item.owner, item.confidence) for item in findings] == [
        ("order-id-mapped", "request-adapter", "confirmed")
    ]


def test_recovered_attempt_remains_on_the_failure_frontier() -> None:
    checks = [
        check("request-valid", "pass"),
        check("first-attempt-transport", "fail", ("request-valid",), owner="network"),
        check("retry-policy", "pass", ("first-attempt-transport",)),
        check("second-attempt-result", "pass", ("retry-policy",)),
        check("final-answer-correct", "pass", ("second-attempt-result",)),
    ]

    assert [item.check_id for item in localize_failure_frontier(checks)] == [
        "first-attempt-transport"
    ]


def test_returns_two_independent_frontier_failures() -> None:
    checks = [
        check("input-valid", "pass"),
        check("inventory-argument", "fail", ("input-valid",), owner="inventory-agent"),
        check("policy-version", "fail", ("input-valid",), owner="policy-cache"),
        check("final-answer", "fail", ("inventory-argument", "policy-version")),
    ]

    assert [item.check_id for item in localize_failure_frontier(checks)] == [
        "inventory-argument", "policy-version"
    ]


def test_marks_failure_uncertain_when_a_prerequisite_is_unknown() -> None:
    checks = [
        check("tool-request-captured", "unknown"),
        check("tool-result-valid", "fail", ("tool-request-captured",)),
    ]

    [finding] = localize_failure_frontier(checks)
    assert finding.confidence == "uncertain"


def test_rejects_a_missing_dependency() -> None:
    with pytest.raises(ValueError, match="missing dependencies"):
        localize_failure_frontier([
            check("final-answer", "fail", ("state-update",))
        ])


@pytest.mark.parametrize("outcome", ["pass", "unknown"])
def test_rejects_a_cycle_without_a_failing_check(outcome: str) -> None:
    with pytest.raises(ValueError, match="dependency cycle"):
        localize_failure_frontier([
            check("left", outcome, ("right",)),
            check("right", outcome, ("left",)),
        ])

The recovered-attempt test deserves careful interpretation, starting with its name. test_recovered_attempt_remains_on_the_failure_frontier asserts that the frontier is exactly ["first-attempt-transport"], so recovery does not erase the event. An earlier draft of this suite called the same test test_recovered_attempt_does_not_create_a_failure_frontier, which claimed the opposite of what the assertion checks. A name that contradicts its own body is worse than an unnamed test, because reviewers read names when they are skimming and only read bodies when something already looks wrong. Name the test after the assertion, not after the outcome you were hoping for.

The behaviour itself is deliberate. The failed transport attempt is reported as a frontier event because that contract genuinely failed, even though the workflow recovered. Release policy can classify a recovered transient attempt as non-blocking. Localization and severity are separate. Removing the event from findings would erase useful reliability evidence and make retry storms invisible.

Work the evidence through three different incidents

In the first incident, the final response says no order exists. The tool span is green because the order service accepted the request and returned a valid not-found response. The model span is also green at the runtime level because it completed normally. A final-answer evaluator fails the response.

Read backward through declared dependencies. The answer check depends on the normalized state. The state accurately contains not-found, so that check passes. The result-normalization check accurately reflects the tool response and passes. The request-contract check compares the authenticated conversation context with the dispatched arguments and finds that customer_id=C91 was placed in order_id. Its own input-context prerequisite passes. That request check is the confirmed frontier.

This conclusion does not blame the order service. An empty result was correct for the bad identifier it received. It also avoids blaming the final model turn for believing the state supplied to it. The bug report should include the request span ID, redacted expected identifier type, actual field mapping, adapter version, and the downstream symptom. It should not include the customer's raw account data.

In the second incident, a knowledge tool returns a transport error on its first attempt and succeeds on the second. The final answer cites the retrieved policy correctly. A timeline colored by error status highlights the first attempt and may label the whole run failed. The contract graph tells a more useful story.

The first transport check fails. The retry eligibility check confirms that the error category is one the application permits to retry. The backoff check passes. The next request preserves the logical operation identity while using a distinct attempt ID. Its result and the final answer pass. Localization retains the transport failure as a recovered frontier event, while release policy records success with degraded reliability.

If the retry happened on a validation error with unchanged invalid arguments, the retry eligibility check would fail too. Depending on the dependency design, that policy failure becomes another actionable finding or a downstream symptom of the bad request. Define ownership before the incident. Otherwise teams will argue about the graph when they should be fixing production.

The third incident ends with an outdated shipping date. The tool request contains the right order ID, and the tool result contains the current date. The next model turn receives the previous date. Here the tool is healthy and the final model is using the state it was given.

Compare the raw tool result with the state snapshot produced by the reducer or middleware boundary. The result-contract check passes. The state-update check fails because the new estimated_delivery value was written under a key the next prompt builder does not read. The final-answer date check fails downstream. Localization points to state update, and the evidence explains why rerunning the tool alone will not fix the defect.

An almost identical symptom can come from session contamination. The state update may be correct for session A while the next model turn reads session B. Check session or thread correlation at every boundary before declaring a reducer bug. The decisive evidence is not merely a stale value. It is which run, session, and state version produced and consumed that value.

These examples show why generic stages such as “preparation, execution, observation, decision” are too broad for a useful bug owner. Define checks around real interfaces: context-to-arguments mapping, dispatcher-to-tool transport, tool-response schema, result-to-state update, state-to-prompt projection, and final response. The exact list varies by workflow and should not be copied unchanged across unrelated agents.

Add counterfactual fixtures for each incident. Correct only the order mapping and require all downstream checks to pass. Change the second retry error category to non-retryable and require the policy finding to become blocking. Correct only the state key and require the date answer to pass. Those tests prove that the asserted fault is connected to the observed symptom.

Do not make the counterfactual unrealistically clean. When correcting the order mapping, keep the same account, tool response shape, state adapter, and answer checker. If the test replaces the entire trace with a separate successful fixture, many fields changed and the causal claim remains weak. A focused mutation identifies the contract field whose correction clears the path.

Ownership belongs to check definitions, not to generic span types. A tool span can contain a request-adapter failure owned by the agent platform, a transport failure owned by networking, or a business rejection owned by the tool service. Routing every tool span to one team recreates the same last-red-span mistake at a coarser level. Store a reviewed owner with each contract and provide a fallback queue for obsolete mappings.

State snapshots need selective capture. Saving an entire conversation after every event can expose sensitive content and consume substantial storage. Capture the version, keys relevant to the contract, and approved hashes or comparisons. For the shipping-date incident, QA needs to prove that the tool value and projected value differ. It does not need every previous customer message.

Tool results often contain partial success. A batch lookup can return records for nine identifiers and an error for one. A single green or red span status cannot represent that shape. Create per-item checks when the workflow makes per-item promises, then let the graph localize the affected answer claims. If the final response omits only the failed record, unaffected branches should remain passing.

Streaming introduces another boundary. The user may receive several tokens before a late guard or validation check fails. A final server-side state marked failed does not erase content already delivered. Add checks for what was emitted, whether the stream closed with an explicit error, and whether the client displayed partial content as complete. Localize the protocol failure separately from the semantic fault in the generated text.

Cancellation should not appear as a product error by default. If the user cancels after the tool begins, an incomplete child span may be expected under the lifecycle contract. Require a cancellation event correlated to the run, verify no prohibited side effect continued, and classify the outcome as cancelled. Without that event, the same missing end remains ambiguous.

Handoffs need a transfer contract. The source agent can choose the right destination but omit required context, after which the destination makes a poor tool call. The earliest actionable failure is the handoff payload check, not destination reasoning. Conversely, if the payload is complete and the destination ignores it, the destination request check becomes the frontier. Preserve the transferred field names and approved equality results so the test can distinguish them.

Human approval adds a strict causal obligation. A side-effecting call must be linked to the matching approval record and occur after approval under the application's policy. Merely finding any approval somewhere in the trace is insufficient. The check should compare action identity, scope, and order. A later successful tool result remains a symptom if the approval contract fails first.

For multiple frontier findings, avoid ranking by an invented severity number inside the localizer. Return evidence and policy labels, then let the release layer apply reviewed rules. An authorization failure may block immediately, while a recovered network attempt creates an operational warning. Keeping those decisions outside graph traversal prevents a policy update from changing causal facts.

Bug reports should name what the localizer did not establish. A useful ticket says the context-to-request check failed, downstream state and answer checks were suppressed as symptoms, and the trace does not reveal why the adapter selected that field. This wording is more honest and more actionable than “AI reasoning failed.” It points the owner to a reproducible boundary without inventing model intent.

When replay is possible, begin at the localized boundary. Feed the captured, redacted context into the request adapter and assert the mapped object. For a state failure, feed the approved tool result into the reducer and inspect the projected state. This reduces model variability and confirms the deterministic implementation defect. Run a full agent replay afterward only when the fix could alter broader behavior.

Distinguish a product fault from broken trace evidence

Sampling can remove the span that contains the first failed contract. A downstream answer still fails, but its prerequisite becomes unknown. Returning that downstream span as confirmed rewards incomplete traces. Mark the finding uncertain and report which required check could not be evaluated. If the release gate requires full evidence for a critical flow, block on trace completeness as a separate control.

Duplicate export can create two copies of one failed tool span. Deduplicate only with stable trace and span identity plus content checks. Two separate attempts with the same arguments are not duplicates. Preserve attempt IDs and correlation so a retry does not disappear during ingestion.

Clock skew can make a downstream symptom appear earlier than its cause. Use causal relationships for the check graph. Timestamp order remains useful for display among unrelated events, but it should not decide ancestry. If parent or call correlation is missing, say the order is unresolved.

Instrumentation can mark a handled exception as an error span even when the application contract expects fallback. Keep the span status, but add a fallback check and final outcome. Severity policy can allow the recovered path while still tracking its frequency. Erasing the error hides operational risk; blocking every recovered run makes resilience features unusable.

The inverse problem is a green span with a bad result. HTTP success, a completed Python function, or a model response without an exception only proves execution completed. Schema, business semantics, authorization, freshness, and state projection still require checks. Do not map runtime status ok directly to QA outcome pass.

Reference data can be wrong. If a final-answer evaluator expects yesterday's policy version, it will fail a correct production answer. Record the reference identifier and version as part of evaluation evidence. A disagreement between tool result and reference should route to review when neither source has established authority in the test contract.

Semantic graders introduce their own uncertainty. Save grader version, rubric, score, and rationale separately from trace facts. Repeated grader runs can disagree. A threshold crossing should not overwrite deterministic observations such as tool arguments or exact totals. When a reviewer overrides the grade, preserve both values and the reason.

Privacy redaction can also make localization impossible. If every identifier becomes the same [REDACTED] token, the adapter cannot prove that the request matched the authenticated context. Consider computing equality or a scoped digest inside the trusted service and exporting only the comparison result. Do not weaken redaction merely to make a dashboard convenient.

Some traces end abruptly. A missing final span could mean process termination, collection lag, user cancellation, or sampling. Look for an explicit lifecycle event before assigning one cause. “No final answer recorded” is evidence. “The model crashed” is an unsupported conclusion unless the trace or service logs confirm it.

A repository-owned diagnostic can render frontier findings without claiming to reproduce a vendor viewer. The exact output below belongs to the example command and can be snapshot-tested.

from __future__ import annotations

import argparse
import json
from pathlib import Path

from failure_frontier import Check, localize_failure_frontier


def load_checks(path: Path) -> list[Check]:
    raw_checks = json.loads(path.read_text())
    return [
        Check(
            check_id=str(raw["check_id"]),
            span_id=str(raw["span_id"]),
            outcome=str(raw["outcome"]),
            depends_on=tuple(raw.get("depends_on", [])),
            owner=str(raw["owner"]),
            evidence=str(raw["evidence"]),
        )
        for raw in raw_checks
    ]


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("fixtures", nargs="+", type=Path)
    args = parser.parse_args()

    for path in args.fixtures:
        print(f"TRACE {path}")
        findings = localize_failure_frontier(load_checks(path))
        if not findings:
            print("FRONTIER none")
        for finding in findings:
            print(
                f"FRONTIER {finding.confidence} check={finding.check_id} "
                f"span={finding.span_id} owner={finding.owner}"
            )
            print(f"evidence={finding.evidence}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

python failure_frontier_audit.py tests/fixtures/bad-order-mapping.json

# TRACE tests/fixtures/bad-order-mapping.json
# FRONTIER confirmed check=order-id-mapped span=span-tool-request owner=request-adapter
# evidence=authenticated identifier type did not match dispatched order_id

Keep raw evidence in a protected artifact and put only approved summaries in CI logs. Tool inputs, model messages, and function outputs may contain secrets or personal data. The tracing documentation for the Agents SDK explicitly calls out sensitive data capture for generation and function spans, which is a good reminder to make redaction part of the trace contract.

Roll out localization one contract at a time

Start with one high-value workflow and map its actual interfaces. Ask each owner what input they require, what success and failure look like, and what correlation identifiers survive the boundary. Write checks only for facts the trace can observe. A contract that depends on an unrecorded value belongs in the instrumentation backlog, not in a speculative localizer.

Capture a small set of reviewed failures and successful controls. Normalize them into committed, redacted fixtures. For each failure, record the expected frontier, downstream symptoms, unknown checks, and recovery status. Then mutate the decisive field and verify that the outcome changes. This guards against oracles that only echo fixture labels.

Run the localizer in shadow mode against current traces. Compare its findings with human incident conclusions. Disagreements are useful. They may expose a missing dependency, a check attached to the wrong span, stale ownership, or a human conclusion unsupported by evidence. Do not tune the graph merely to reproduce every historical label.

Introduce release gates only for deterministic, high-confidence contracts. Bad authorization context, duplicate side effects, schema violations, and exact amount mismatches are good candidates. Semantic relevance or tone may remain a reviewed score. An uncertain localization should never be silently converted into pass to improve the dashboard.

name: agent-failure-localization

on:
  pull_request:
    paths:
      - "trace_contracts/**"
      - "failure_frontier/**"
      - "tests/failure_frontier/**"

jobs:
  contract-frontier:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements-test.txt
      - run: python -m pip install -r requirements-test.txt
      - run: python -m pytest tests/failure_frontier -q
      - run: python failure_frontier_audit.py tests/fixtures/failure_frontier/*.json

The cache-dependency-path entry earns its line. actions/setup-python with cache: "pip" defaults to hashing **/requirements.txt, with **/pyproject.toml as a fallback, and it treats a total miss as an error rather than as a reason to skip caching. A frontier analyzer that needs no third-party runtime dependencies and keeps its test pins in requirements-test.txt matches neither default, so the workflow fails during setup and reports nothing about the contracts you meant to check. A green pipeline that never executed the analyzer is the same class of mistake this article warns about in traces: absence of a failure signal is not evidence that the check ran.

The migration costs time and maintenance. Every new contract check needs an owner. Instrumentation adds storage and privacy review. Dependency graphs can become stale when workflows change. Returning multiple findings complicates dashboards that expect one category. Those costs are preferable to confidently assigning defects to the last red span, but they need funding and review.

Version check definitions and adapters. A fixture should identify the normalized schema version and workflow version. When a state key changes, update the producer, check, and fixture together. Keep compatibility explicit for older traces rather than interpreting missing new fields as passes.

Use localization metrics carefully. A rise in request-mapping findings could mean a regression, broader trace coverage, or a new check. Do not present counts across check-version changes as a continuous measurement without qualification. Keep operational error frequency separate from the number of traces successfully localized.

Know when the trace cannot answer the question

Do not localize beyond the observed boundary. A trace can show that a tool returned stale data. It cannot prove why the database was stale unless the relevant data path is traced and checked. Route the failing result contract with its evidence and let the service investigation continue.

Avoid forcing a single root when branches are independent. Return all minimal failures and let incident management group them later. Picking the earliest timestamp or alphabetically first span throws away information.

Do not use failure localization as a substitute for reproduction. The graph narrows the failing interface. A focused replay or unit test should still demonstrate the implementation defect where possible. Trace evidence can be incomplete, and production state may have changed.

Skip semantic localization when the rubric is not stable. If reviewers disagree about whether an answer is sufficiently helpful, naming one internal span as the cause is premature. Improve the rubric, collect examples, and preserve uncertainty before building automation on top.

Finally, do not expose raw traces broadly just because they help debugging. Minimize captured content, redact at the trusted boundary, restrict artifacts, and set retention deliberately. A correct failure location does not justify leaking the customer data that happened to reveal it.