A trace taxonomy that points to the first broken contract

The support agent tells a customer, “I couldn’t complete that request.” One trace shows the model chose the wrong tool, another shows valid selection with a rejected argument, and a third shows a payment timeout after the request left your service. Giving all three runs the label agent_error destroys the information needed to fix them.

A useful taxonomy does not start with a fashionable list of failure names. It starts with contracts in your own execution path and asks which one failed first. The answer must change when the trace changes, or the classifier is decoration.

Classify the first failed contract, not the loudest symptom

Final responses are poor starting points. Orchestration often catches a low-level error, retries, rewrites it, and returns a polite sentence. Two completely different defects can produce the same words. A response grader may tell you the answer was unhelpful, but it cannot prove whether selection, arguments, authorization, or execution broke.

Follow the causal path from the beginning and stop at the earliest failed invariant supported by evidence. In a simple tool run, the contracts might be:

1. The model call completed in a form the application could parse.
2. The selected tool was allowed for this scenario.
3. Arguments matched the tool contract.
4. Authorization evaluated and allowed the requested effect.
5. Execution started only after validation and authorization.
6. The adapter returned a success or a classified failure.
7. Final output met the product's structural contract.

This list is not an industry standard or an “approved taxonomy.” It is a starter sequence for one application shape. A retrieval-only assistant may add query construction, source access, and citation verification. A code agent may add sandbox creation, command policy, and artifact validation. Name the contracts your code can actually observe.

“First” means causally earliest, not the earliest timestamp displayed by a collector. Service clocks can disagree and exporters can buffer. Prefer a sequence written by the orchestrator for one run, plus call IDs and parent event IDs. When components assign independent sequences, reconstruct causality from explicit links instead of sorting unrelated integers.

Separate failure stage from cause, effect, owner, and retry policy. tool_execution can be a stage. remote_timeout can be a code. side_effect_unknown can be an effect. payments-platform can be an owner. reconcile_before_retry can be the recovery rule. Combining all of them into retryable_tool_timeout makes the label brittle and tempts callers to retry without knowing the operation.

Expected control outcomes also need separation. A policy that denies a cross-tenant export is working. Record policy_decision with a denial code and an expected-control disposition. The scenario may still fail a product test if the user should have had access, but the authorization component did not malfunction merely because it said no.

Severity is another independent decision. A wrong weather tool and an unauthorized payment start may both sit at an orchestration stage, yet their consequences differ. Calculate severity from protected asset, action, exposure, and effect evidence. Do not bake “critical” into a category that will later be used by a harmless tool. Security and reliability teams can share a stage model while applying different escalation rules.

Ownership should follow the first broken contract, with room for contributing systems. A provider deadline can be owned initially by the model integration team, but investigation may show an application timeout shorter than the configured provider deadline. Preserve configured timeout, observed duration, provider response code, and retry attempt. A taxonomy routes the first look; it does not replace root-cause analysis.

Keep raw cause text out of high-cardinality dimensions. Exception messages can contain request IDs, URLs, customer data, and changing wording. Map them to reviewed codes at the boundary and retain a restricted reference to the original event. Dashboards stay stable, and privacy review has a smaller surface. Unknown raw values should map to unclassified_tool_error under the correct stage, not create a new category automatically.

One run can contain both a primary failure and consequences. Invalid arguments can trigger a repair loop, exhaust a retry budget, and produce an invalid final output. The argument contract is primary because later events depend on it. Retain retry exhaustion and output failure as contributing findings when they help, but do not count one run as three independent root failures in a trend chart.

There are also successful runs with concerning signals. A tool may time out once, succeed on retry, and return a valid answer. The run outcome is success, the attempt finding is tool execution error, and the recovery outcome is recovered. Removing the attempt from telemetry makes reliability look better while hiding cost and latency; treating the whole run as failed makes user success look worse. Orthogonal fields prevent that argument.

Use a small schema with orthogonal fields

Store raw vendor traces under their own retention and access rules, then normalize only the fields your classifier needs. A normalized event should keep the original trace and event references so an investigator can drill down. It should not copy entire prompts, customer records, or model responses into every finding.

For this example, each event has a run sequence, kind, optional tool call ID, and typed data. The classifier returns stage, code, disposition, and the sequence where evidence first proves the finding. It maintains per-call lifecycle state, which lets it detect a tool starting before validation or authorization.

from dataclasses import dataclass
from typing import Any, Literal, Mapping


Disposition = Literal["defect", "expected_control", "telemetry_unknown"]


@dataclass(frozen=True)
class TraceEvent:
    sequence: int
    kind: str
    call_id: str | None
    data: Mapping[str, Any]


@dataclass(frozen=True)
class Expectation:
    allowed_tools: frozenset[str]


@dataclass(frozen=True)
class Finding:
    stage: str
    code: str
    disposition: Disposition
    sequence: int


@dataclass
class CallState:
    selected: bool = False
    arguments_valid: bool | None = None
    authorized: bool | None = None
    started_at: int | None = None
    finished: bool = False


def classify(
    events: list[TraceEvent],
    expectation: Expectation,
) -> Finding | None:
    sequences = [event.sequence for event in events]
    if len(sequences) != len(set(sequences)):
        return Finding(
            "telemetry_integrity",
            "duplicate_sequence",
            "telemetry_unknown",
            min(sequences),
        )

    calls: dict[str, CallState] = {}
    control_outcomes: list[Finding] = []
    lifecycle_events = {
        "tool_selected",
        "arguments_checked",
        "authorization_checked",
        "tool_started",
        "tool_finished",
    }

    for event in sorted(events, key=lambda item: item.sequence):
        if event.kind == "model_finished" and event.data.get("status") == "error":
            return Finding(
                "model_call",
                str(event.data.get("error_code", "model_error")),
                "defect",
                event.sequence,
            )

        if event.kind in lifecycle_events and event.call_id is None:
            return Finding(
                "telemetry_integrity",
                "tool_event_missing_call_id",
                "telemetry_unknown",
                event.sequence,
            )

        if event.call_id is not None:
            state = calls.setdefault(event.call_id, CallState())
        else:
            state = None

        if event.kind == "tool_selected":
            if state.selected:
                return Finding(
                    "telemetry_integrity",
                    "duplicate_tool_selection",
                    "telemetry_unknown",
                    event.sequence,
                )
            state.selected = True
            if event.data.get("tool") not in expectation.allowed_tools:
                return Finding(
                    "tool_selection",
                    "unexpected_tool",
                    "defect",
                    event.sequence,
                )

        elif event.kind == "arguments_checked":
            state.arguments_valid = event.data.get("valid") is True
            if not state.arguments_valid:
                return Finding(
                    "tool_input_contract",
                    str(event.data.get("reason", "invalid_arguments")),
                    "defect",
                    event.sequence,
                )

        elif event.kind == "authorization_checked":
            state.authorized = event.data.get("allowed") is True
            if not state.authorized:
                control_outcomes.append(
                    Finding(
                        "policy_decision",
                        str(event.data.get("reason", "denied")),
                        "expected_control",
                        event.sequence,
                    )
                )

        elif event.kind == "tool_started":
            if state.arguments_valid is not True:
                return Finding(
                    "orchestration",
                    "tool_started_without_valid_arguments",
                    "defect",
                    event.sequence,
                )
            if state.authorized is not True:
                return Finding(
                    "orchestration",
                    "tool_started_without_authorization",
                    "defect",
                    event.sequence,
                )
            state.started_at = event.sequence

        elif event.kind == "tool_finished":
            if state.started_at is None:
                return Finding(
                    "orchestration",
                    "tool_finished_without_start",
                    "defect",
                    event.sequence,
                )
            state.finished = True
            if event.data.get("status") == "error":
                return Finding(
                    "tool_execution",
                    str(event.data.get("error_code", "tool_error")),
                    "defect",
                    event.sequence,
                )

        elif event.kind == "output_checked" and event.data.get("valid") is not True:
            return Finding(
                "output_contract",
                str(event.data.get("reason", "invalid_output")),
                "defect",
                event.sequence,
            )

    unfinished = [
        state.started_at
        for state in calls.values()
        if state.started_at is not None and not state.finished
    ]
    if unfinished:
        return Finding(
            "telemetry_completeness",
            "tool_terminal_event_missing",
            "telemetry_unknown",
            min(unfinished),
        )
    if control_outcomes:
        return min(control_outcomes, key=lambda finding: finding.sequence)
    return None

Several design choices are deliberate. The classifier never treats an event's failureCategory field as truth. It reads observable lifecycle events. It does not infer retryability from an error code. It marks missing terminal evidence as unknown rather than guessing timeout. It returns one first finding; a separate incident view can retain later consequences.

The selected field catches two selection events that reuse one call ID, but the sample does not yet require selection before arguments_checked. A stricter contract could return arguments_without_selection when state is absent. Add that only if all producers guarantee the selection event. Taxonomies fail when classifiers demand telemetry that some legitimate path never emits.

Use stable codes inside a stage. Human-readable messages can change and may include provider text. Map raw exceptions and HTTP responses at the adapter boundary, retaining a redacted original reference. MDN's HTTP status reference explains transport status semantics, but a status alone still does not reveal whether an operation is safe to retry or whether a side effect occurred.

Add an explicit schema version to every normalized event. A producer upgrade can rename allowed to decision or split one terminal event into several events. Without a version, the classifier may read a missing field as false and report a denial that never happened. Reject unsupported versions into telemetry integrity, then upgrade the producer or mapping. Silent compatibility logic is difficult to test and even harder to retire.

Call IDs must be unique within the run. If two parallel tool calls share one ID, their validation, authorization, and terminal events can merge into a lifecycle that never occurred. Add a test that selects two tools with the same call ID and require a collision finding. A globally unique event ID serves deduplication; a run-scoped call ID serves lifecycle correlation. They solve different problems.

Event producers should record decisions where they are made. The argument validator emits arguments_checked. The authorization service emits or signs authorization_checked. The tool wrapper emits start and finish. A central collector that guesses decisions from log text can misread retries and exceptions. Normalize transport format centrally if useful, but preserve the authoritative producer.

Not every data item belongs in data. Stable fields used for routing and joins deserve a defined type. Large payloads and raw responses belong behind restricted references with separate retention. This makes the classifier cheap enough to run on every trace and reduces the damage if taxonomy records are exposed.

Lifecycle state should be scoped by attempt as well as call when retries reuse a logical call ID. Either issue a new attempt ID or include an attempt number in every event. Otherwise the first failed finish can set finished=True and make a later missing terminal invisible. The run view can group attempts under the logical tool request after each attempt is classified.

Make the classifier fail when the trace changes

Build one valid control trace, then mutate one event for each contract. The control prevents a classifier that always returns a failure from passing. Every negative case drives the event list through classify and compares the complete Finding, including evidence sequence.

from dataclasses import replace

import pytest

from trace_taxonomy import Expectation, Finding, TraceEvent, classify


EXPECTED = Expectation(allowed_tools=frozenset({"customer_search"}))

VALID = [
    TraceEvent(10, "model_finished", None, {"status": "ok"}),
    TraceEvent(20, "tool_selected", "call-1", {"tool": "customer_search"}),
    TraceEvent(30, "arguments_checked", "call-1", {"valid": True}),
    TraceEvent(40, "authorization_checked", "call-1", {"allowed": True}),
    TraceEvent(50, "tool_started", "call-1", {}),
    TraceEvent(60, "tool_finished", "call-1", {"status": "ok"}),
    TraceEvent(70, "output_checked", None, {"valid": True}),
]


def change(sequence, **data):
    return [
        replace(event, data={**event.data, **data})
        if event.sequence == sequence
        else event
        for event in VALID
    ]


def remove(sequence):
    return [event for event in VALID if event.sequence != sequence]


def test_valid_trace_has_no_finding():
    assert classify(VALID, EXPECTED) is None


def test_denial_without_execution_is_an_expected_control():
    denied = VALID[:3] + [
        replace(
            VALID[3],
            data={"allowed": False, "reason": "tenant_not_granted"},
        )
    ]

    assert classify(denied, EXPECTED) == Finding(
        "policy_decision",
        "tenant_not_granted",
        "expected_control",
        40,
    )


@pytest.mark.parametrize(
    ("events", "expected"),
    [
        (
            change(20, tool="customer_export"),
            Finding("tool_selection", "unexpected_tool", "defect", 20),
        ),
        (
            change(30, valid=False, reason="missing_account_id"),
            Finding(
                "tool_input_contract",
                "missing_account_id",
                "defect",
                30,
            ),
        ),
        (
            remove(40),
            Finding(
                "orchestration",
                "tool_started_without_authorization",
                "defect",
                50,
            ),
        ),
        (
            change(40, allowed=False, reason="tenant_not_granted"),
            Finding(
                "orchestration",
                "tool_started_without_authorization",
                "defect",
                50,
            ),
        ),
        (
            change(60, status="error", error_code="remote_timeout"),
            Finding("tool_execution", "remote_timeout", "defect", 60),
        ),
        (
            remove(60),
            Finding(
                "telemetry_completeness",
                "tool_terminal_event_missing",
                "telemetry_unknown",
                50,
            ),
        ),
        (
            change(70, valid=False, reason="missing_citations"),
            Finding("output_contract", "missing_citations", "defect", 70),
        ),
    ],
)
def test_each_mutation_reaches_its_first_failed_contract(events, expected):
    assert classify(events, EXPECTED) == expected

Remove the authorization-state check from the classifier and two rows break, in two different ways. The row that deletes event 40 has no authorization evidence left anywhere, so the classifier walks the rest of a healthy lifecycle and returns no finding at all. The row that sets allowed=False still recorded a denial, so the classifier falls through to that stored control outcome and reports policy_decision at sequence 40 instead of the orchestration defect at sequence 50. Both are real failures, and the difference between them tells you which half of the invariant went missing. Change the selected tool back to the allowed one and the first row becomes clean. Replace the tool result error with success and the execution finding disappears. These are failure-sensitive oracles.

Add ordering mutations next. Put tool_started before arguments_checked and expect tool_started_without_valid_arguments at the start event. Put it before authorization and expect the authorization invariant. Do not sort the fixture into a valid lifecycle before classifying; that would erase the defect being tested.

Test duplicate and malformed telemetry separately. Duplicate sequence values should produce telemetry_integrity, not whichever event happens to win a sort. A missing run ID, invalid call ID, or unknown schema version may require quarantining the trace before classification. The classifier should not silently drop malformed events until the remaining story looks valid.

Exercise precedence explicitly. Create a trace with invalid arguments followed, incorrectly, by a tool start. The expected primary finding remains tool_input_contract at the validation event, while a secondary audit can flag that orchestration continued. Then remove the validation failure and expect tool_started_without_valid_arguments. These paired cases prove “first failed contract” is implemented rather than described.

Negative cases need positive neighbors for every optional path. If authorization is not required for a public read tool, represent that with an explicit not_required decision or a tool policy, not by omitting the event and teaching the classifier to ignore absence. Test the public path and a protected path. Otherwise someone can reclassify a protected tool as public by dropping telemetry.

Property-based generation can extend hand-written cases once the lifecycle is stable. Generate valid event sequences, then delete, duplicate, reorder, or alter one event. Assert that every tool start has earlier valid arguments and an allowed or explicitly unnecessary authorization decision. Keep a small set of readable incident fixtures even if generation finds more combinations; responders need examples they can understand without replaying a random seed.

Do not snapshot the classifier's entire output for a large trace. A broad snapshot changes whenever harmless metadata moves and encourages reviewers to approve noisy diffs. Assert the exact stage, code, disposition, sequence, call ID, and evidence references relevant to the case. Separate serialization tests can cover the full record shape.

Work through three traces that look alike

The first run ends with a generic apology after the model selects customer_export when the scenario allows only customer_search. Argument validation may pass because export has a valid schema. Authorization may later deny it. The first broken test contract remains tool selection at sequence 20. Record the later denial as a successful containment outcome, not the primary cause.

The owner is likely prompt, tool description, model configuration, or scenario expectation. Reproduce with the original tool menu and model response. Then call the classifier on normalized events. Do not send the ticket to the export adapter team unless evidence shows its schema or execution failed.

The second run selects customer_search but omits account_id. The parser emits an argument-validation event with valid=False. If orchestration correctly stops, there should be no authorization or start event for that call. The stage is tool_input_contract, and the code names the missing field. A model-selection dashboard that counts only tool name would call this successful even though the tool could not run.

The owner may be schema design, prompt guidance, model adaptation, or input mapping. Check the raw tool call under restricted access and the normalized validation error. If the raw call contains the account but an adapter rename drops it, ownership shifts from the model path to application mapping. The taxonomy gives a stage, while evidence determines the exact component.

The third run selects the right tool, passes arguments, receives authorization, starts execution, and records remote_timeout. That is a tool-execution finding. The user sees the same apology, but changing prompts will not repair a network deadline or overloaded dependency.

Do not mark it retryable from the word timeout. A read whose request never reached a server may be safe to repeat. A payment request with a lost response can have an unknown outcome. Add effect_state and recovery fields from the adapter's contract or reconciliation result. Keep them orthogonal to the taxonomy stage.

A fourth near-miss is policy denial. The trace is valid through argument checking, then authorization_checked says no for another tenant. The classifier returns an expected-control disposition. If the test scenario intentionally attacks tenant isolation, this is a pass. If an entitled user was denied because group data was stale, the user journey fails, but the executor still enforced the grant it received. Investigation moves to grant resolution.

Retries add two levels of outcome. Attempt one can have tool_execution/remote_timeout, attempt two can succeed, and the run can finish successfully. Store attempt findings rather than erasing the first error. For a release metric, decide whether recovered errors count separately from failed runs. Never label the first timeout “false” because a later attempt worked.

Parallel calls require more than one first finding. Each call has its own lifecycle; the run also has a causal outcome. One search can succeed while a payment call starts without authorization. A classifier that returns only the earliest run-wide issue may hide the higher-consequence violation. Run per-call classification first, then choose the run summary by a documented severity and causal policy.

Model failures also need a boundary between transport and output parsing. A provider connection error before any response is model_call with a transport code. A successful HTTP response whose content cannot be parsed into the expected message structure is a model-output contract or adapter-parsing finding, depending on where the contract lives. Preserve response status, SDK exception type, and parser stage. Calling both “bad model output” sends network defects to prompt engineers.

Refusal is not automatically an error. A safety refusal to a prohibited request can be an expected-control outcome. A refusal to an allowed, ordinary task can fail a scenario expectation. Record the normalized response kind, policy context, and test expectation. Avoid judging refusal from a substring in prose when the provider or application exposes structured information.

Output validation deserves deterministic and probabilistic branches. Missing a required JSON field, citation identifier, or action receipt is a contract failure with a direct assertion. Whether an answer is helpful or grounded may require an evaluator and calibrated threshold. Keep evaluator name, version, rationale, and uncertainty with that finding. Do not let a probabilistic score override proof that a required field is absent.

Retrieval introduces its own chain. A query can be malformed, the store can return no documents, the ranker can choose irrelevant sources, and the model can ignore good evidence. Labeling all of them grounding_error is the same mistake as agent_error at a smaller scale. Add stages only when traces carry evidence that distinguishes them and a team can act on the result.

Retry-budget exhaustion is usually an orchestration outcome supported by earlier attempt findings. If three tool calls fail with the same remote timeout and the loop stops at its configured maximum, keep the timeout on each attempt and retry_budget_exhausted on the run. If the loop stops after one attempt despite a remaining safe retry, the broken contract is retry orchestration. The same final state has a different cause.

Cancellation should remain distinct from failure. A user cancellation, workflow deadline, and infrastructure termination can all leave an unfinished tool span. Emit a trusted cancellation event and propagate it to the adapter where possible. If no terminal acknowledgment returns, effect state remains unknown even though cancellation was requested. Do not convert requested cancellation into proof of non-execution.

Diagnose incomplete or misleading telemetry

Start by asking whether the expected events could have been recorded. Check sampling, exporter health, queue lag, schema version, redaction rules, and process termination. An absent tool_finished can mean a hung adapter, a crash, dropped telemetry, or an event producer that was never instrumented. telemetry_unknown preserves those possibilities.

Compare the trace with external systems. A target audit entry proves a request reached the dependency. An idempotency record can show accepted, rejected, or uncertain state. A database row can show a durable effect. These sources may reveal execution even when the application trace ends early. They also prevent a complete-looking trace from hiding a side effect performed through an uninstrumented path.

Use W3C traceparent for cross-service correlation where supported, and validate its format at trust boundaries. The standard allows tracing systems to make sampling decisions and warns about security and privacy considerations. A sampled flag is not a promise that every event exists. Trace and span identifiers are not user identity, approval, or authorization.

Late ingestion can reverse display order. Preserve producer sequence, event time, ingest time, and causal parent separately. Do not overwrite the producer's order with collector arrival. If the producer clock is wrong, wall-clock duration may be unreliable while lifecycle sequence remains usable.

Redaction can break classification if it removes the very field used by an invariant. Prefer derived safe fields such as arguments_valid and stable reason codes over copying raw arguments. Generate those fields at the component that performed validation. A downstream log processor should not re-validate a redacted payload and claim the result represents execution.

Sampling creates biased failure rates when success and error traces have different retention. A platform may keep every error but sample successful runs, making raw category percentages meaningless. Store sampling policy and denominator information with aggregates. Compare like with like, or calculate rates from an unsampled decision counter. Do not present a sampled trace collection as the complete production distribution.

Exporter backpressure can correlate missing data with the very outages you are studying. When a dependency slows, spans grow, queues fill, and terminal events may be dropped. Monitor exporter queue depth, rejected batches, and delivery lag outside the agent trace path. An outage report should state when telemetry health limits confidence.

Mixed deployments complicate ordering and schema. A rolling release can produce one run with events from two producer versions. The classifier should accept documented compatible combinations or mark the run mixed-version and uncertain. Do not coerce every event to the newest shape if older producers never emitted the required evidence.

Fan-out and fan-in require causal graphs rather than one flat sequence. Two tools may start concurrently and finish in either order before an aggregator runs. Enforce lifecycle within each call and require the aggregator to reference completed inputs. A global rule that every start and finish must alternate would flag valid concurrency. Build invariants around dependency edges, not the visual order in a trace viewer.

Check for uninstrumented side channels. A tool may call a database directly while the normal adapter emits beautiful events. Reconcile protected target operations against tool call IDs or idempotency keys. A target record with no corresponding authorized start should trigger an enforcement investigation, not be squeezed into telemetry_completeness and forgotten.

When the authorization invariant is removed, two of the seven parameterized rows fail and they fail differently, which is more informative than a single red row would be. Tracebacks are trimmed below to the comparison lines:

$ python -m pytest -q tests/test_trace_taxonomy.py -k first_failed_contract
..FF...                                                                  [100%]
E       AssertionError: assert None == Finding(stage='orchestration', code='tool_started_without_authorization', disposition='defect', sequence=50)
E       AssertionError: assert Finding(stage..., sequence=40) == Finding(stage..., sequence=50)
E         Differing attributes:
E         ['stage', 'code', 'disposition', 'sequence']
E           stage: 'policy_decision' != 'orchestration'
FAILED tests/test_trace_taxonomy.py::test_each_mutation_reaches_its_first_failed_contract[events2-expected2]
FAILED tests/test_trace_taxonomy.py::test_each_mutation_reaches_its_first_failed_contract[events3-expected3]
2 failed, 5 passed, 2 deselected in 0.01s

Count the progress characters before reading the failures. Seven rows ran and two were deselected, those two being the valid-trace control and the expected-control case, which -k filtered out by name. The first failure returned None, meaning the classifier found nothing to say. The second returned the wrong finding rather than nothing, and pytest's attribute drill-down names policy_decision against orchestration as the difference. A mutation that changes which category a run lands in is easy to mistake for a passing suite if you only diff counts, so compare the complete Finding, not just the disposition.

That failure proves classifier regression, not an unauthorized production call. A separate integration test should execute the actual gate with an adapter spy. Taxonomy tests keep diagnosis honest; enforcement tests keep effects safe.

Roll out a taxonomy without rewriting history

Inventory current event producers and real support tickets. List what each component can prove today. Avoid designing categories that require fields unavailable in half the paths. Add schema versions and producer names before changing classification, so old and new traces can coexist.

Choose a small initial set of stages tied to team boundaries: model call, tool selection, input contract, policy decision, orchestration, tool execution, output contract, and telemetry integrity. Define each with entry criteria, required evidence, exclusions, and examples. Name an owner for the definition, not necessarily every incident.

Label a review sample by hand with at least two engineers. Disagreements reveal ambiguous definitions and missing events. Do not publish invented accuracy numbers. Measure agreement on your own sample, adjudicate differences, and keep uncertain when evidence cannot support a label.

Run the classifier in shadow mode and compare it with existing incident labels. Preserve original labels and classifier version. Reclassification should create a new derived record rather than edit history. Otherwise trends can change merely because the taxonomy changed.

Set an evidence threshold for automatic routing. A complete lifecycle with an explicit validation failure can open a ticket for the tool-input owner. An incomplete trace should go to observability triage or a shared queue until target evidence resolves it. Automatic certainty from partial evidence creates ticket ping-pong and teaches teams to distrust the system.

Review false merges and false splits. A false merge puts different broken contracts under one label. A false split gives two names to the same evidence and recovery. Sample both within each high-volume code. Low-frequency, high-consequence security findings need manual review even when statistical sampling would skip them.

Add mutation tests for every new code before enabling it. A fixture should contain a valid control and the smallest event change that creates the finding. Ask what production-code change would make the test fail. If nothing can, rewrite the oracle.

Wire taxonomy tests into CI without model calls. Integration checks for event production can run against an instrumented test agent after packaging. Keep trace fixtures synthetic or thoroughly redacted; production prompts and tool results rarely belong in source control.

name: trace-taxonomy

on:
  pull_request:
    paths:
      - "agent/telemetry/**"
      - "agent/taxonomy/**"
      - "tests/trace_taxonomy/**"

jobs:
  classify:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: python -m pip install -r requirements.txt
      - run: python -m pytest -q tests/trace_taxonomy

Migration costs show up in dashboards and ownership. Old broad labels split into smaller populations. Alerts may move between teams. Event volume and storage can grow when you retain lifecycle evidence. Budget for schema governance, privacy review, and responder training, not only classifier code.

Use versioned mappings for reports. A long-term chart can group detailed stages into a stable parent such as model, tool, policy, or telemetry while incident views keep precise codes. Document the aggregation so leaders do not compare unlike definitions across releases.

Every additional event adds runtime and storage cost. Recording minimal normalized decisions is cheaper than copying payloads, but it is not free. Measure event volume, exporter pressure, classification time, and retention in your environment. If you reduce telemetry, preserve the events needed for authorization and high-consequence effect reconciliation before optimizing low-value detail.

Taxonomy maintenance also consumes expert time. Tool owners need to review codes, QA needs to maintain mutation fixtures, privacy teams need to assess new evidence, and responders need updated runbooks. A smaller taxonomy with reliable evidence beats a large tree nobody can apply consistently. Expand after real ambiguous cases, not because an empty branch looks incomplete.

Define deprecation rather than deleting a code. Mark its last producer version, replacement mapping, and dashboard treatment. Old traces may remain under legal or operational retention. A classifier asked to replay them should either load the historical version or state that reclassification is unsupported.

When not to force a trace into one category

Do not choose a definitive cause when terminal evidence is missing. Use unknown, preserve candidate explanations, and gather target data. A confident but unsupported category sends responders in the wrong direction.

Avoid one run-wide label when independent parallel calls fail differently. Keep per-call findings and a separate run outcome. Summarize by causal impact only when the policy is explicit.

Do not classify a safe policy denial as a technical error solely because the user did not get what they asked for. Product entitlement, policy freshness, and agent behavior can still be investigated without calling enforcement broken.

Never use taxonomy as authorization. A post-run classifier can detect tool_started_without_authorization after the effect. The live executor must prevent that transition.

Finally, do not keep categories that nobody can act on. If two labels always share evidence, owner, recovery, and test cases, merge them. If one label contains failures with different contracts and fixes, split it. The taxonomy earns its place when a changed trace produces a changed, evidence-backed route to action.