Prove exactly where an agent handoff lost context

The billing agent asks for an invoice number that the triage agent collected two turns ago. Both agents completed their own steps, yet the handoff dropped the one field needed to continue. A final-answer score may still pass after the customer repeats the number, so the defect stays hidden until someone grades the boundary itself.

Follow the context across the handoff boundary

A handoff is not proven by two agent names appearing next to each other in a trace. The useful unit is a transition with a stable identity: one source agent, one destination agent, one route, one contract version, and the state observed on both sides. Without that identity, investigators end up matching events by timestamp and hoping adjacent spans belong to the same transfer.

Start with the destination's requirements. A billing route might require customer_id and invoice_id, while a general support route needs customer_id and locale. The source may know many more fields. Those extra fields are not automatically part of the handoff contract, and copying them all is usually the wrong fix. It expands the data exposed to another component and lets downstream code form undocumented dependencies on whatever happened to be present.

For a deterministic check, instrument four facts you control:

1. The route and the agents involved.
2. The required key names for that route and contract version.
3. The key names available before packaging, inside the envelope, and after destination ingestion.
4. Whether each observation is complete or intentionally redacted.

These are application-level trace fields. No agent framework should be assumed to emit them for you. Add them where your code creates and accepts a handoff, then document their meaning as part of your own trace schema. If an SDK later changes its internal spans, the application event can remain stable.

The comparison localizes the first broken boundary. Suppose invoice_id exists in source state but not in a completely observed envelope. Packaging removed it. If the envelope contains the key and destination state does not, ingestion lost or rejected it. When the key never existed in source state, the handoff is innocent; an earlier collection or state-update step failed. When capture is incomplete, the only defensible result is insufficient evidence.

That distinction matters because the visible symptom is often identical. In every case the billing agent says, "What is your invoice number?" The conversational output does not reveal whether the triage agent forgot to store it, an allowlist omitted it, the receiving process loaded the wrong state record, or the trace exporter hid it. The stage evidence does.

A useful verdict vocabulary stays narrow:

Do not compress these outcomes into a single score too early. A zero cannot tell a developer where to look, and a one can conceal a recovery path that repaired the conversation after a broken transfer. Preserve the categorical verdict and missing key names in the test report.

The contract also needs a version. Renaming invoice_id to invoice_reference is not a regression if the producer, consumer, and grader all moved to a new reviewed contract. Running an old expectation against a new schema creates a false failure. Store the version with every fixture and every handoff event, then reject comparisons across versions unless a migration test explicitly covers them.

Values need more care than keys. Production traces should not become a second customer database. Most regression fixtures can use synthetic identifiers, and most production diagnostics only need key presence, route, contract version, and opaque correlation IDs. If value equality is essential, design a privacy-reviewed representation for that field. A plain hash is not automatically safe for low-entropy values such as country codes or status names.

Final-answer quality and transfer integrity are separate oracles. A model-based judge can assess whether the customer received a helpful answer. A deterministic boundary check can prove whether required state survived. Run both if both claims matter, but never let a fluent answer overwrite a failed transfer verdict.

Make the handoff contract executable

The safest fix is to construct the envelope from a route-specific allowlist and fail before dispatch if a required field is unavailable. That gives the producer one place to enforce both completeness and data minimization. It also makes the accepted payload visible in code review.

The following module uses only the Python standard library. The class and function names are local examples, not framework APIs. Save it as handoff_contract.py and run it with python handoff_contract.py.

from __future__ import annotations

from dataclasses import asdict, dataclass
import json
from typing import Any, Mapping


@dataclass(frozen=True)
class HandoffContract:
    route: str
    version: str
    required_keys: tuple[str, ...]
    permitted_keys: tuple[str, ...]


@dataclass(frozen=True)
class HandoffEnvelope:
    case_id: str
    handoff_id: str
    from_agent: str
    to_agent: str
    route: str
    contract_version: str
    context: dict[str, Any]


BILLING_CONTRACT = HandoffContract(
    route="triage_to_billing",
    version="2",
    required_keys=("customer_id", "invoice_id"),
    permitted_keys=("customer_id", "invoice_id", "locale"),
)


def build_handoff(
    *,
    case_id: str,
    handoff_id: str,
    from_agent: str,
    to_agent: str,
    source_context: Mapping[str, Any],
    contract: HandoffContract,
) -> HandoffEnvelope:
    missing_at_source = [
        key for key in contract.required_keys if key not in source_context
    ]
    if missing_at_source:
        missing = ", ".join(sorted(missing_at_source))
        raise ValueError(f"required source context is missing: {missing}")

    payload = {
        key: source_context[key]
        for key in contract.permitted_keys
        if key in source_context
    }
    missing_from_payload = [
        key for key in contract.required_keys if key not in payload
    ]
    if missing_from_payload:
        missing = ", ".join(sorted(missing_from_payload))
        raise RuntimeError(f"handoff packaging removed required keys: {missing}")

    return HandoffEnvelope(
        case_id=case_id,
        handoff_id=handoff_id,
        from_agent=from_agent,
        to_agent=to_agent,
        route=contract.route,
        contract_version=contract.version,
        context=payload,
    )


if __name__ == "__main__":
    envelope = build_handoff(
        case_id="invoice-recovery",
        handoff_id="h-001",
        from_agent="triage",
        to_agent="billing",
        source_context={
            "customer_id": "customer-synthetic-4",
            "invoice_id": "invoice-synthetic-9",
            "locale": "en-IN",
            "internal_note": "must not cross this boundary",
        },
        contract=BILLING_CONTRACT,
    )
    print(json.dumps(asdict(envelope), indent=2, sort_keys=True))

This projection does two jobs. It carries invoice_id because billing requires it, and it excludes internal_note because billing is not permitted to receive it. A blanket copy would appear to solve the missing-field bug, but it would weaken the boundary and make later cleanup risky.

The early failure changes product behavior, so treat it deliberately. Instead of dispatching an incomplete envelope, the source workflow must recover, ask for the missing fact, or return a handled error. That may add a turn or prevent a partial answer. The alternative is sending a downstream agent into a state where it can only guess or ask the same question again.

Contract validation alone does not prove the receiver loaded what was sent. Capture the same handoff ID when the destination accepts the envelope. Compare source, payload, and destination stages in one grader. The diagnostic below is also standard-library Python, and its sample rows are explicitly synthetic.

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class TraceRow:
    case_id: str
    handoff_id: str
    expected_to_agent: str
    observed_to_agent: str
    required_keys: frozenset[str]
    source_keys: frozenset[str]
    payload_keys: frozenset[str]
    destination_keys: frozenset[str]
    source_capture_complete: bool = True
    payload_capture_complete: bool = True
    destination_capture_complete: bool = True


@dataclass(frozen=True)
class Grade:
    case_id: str
    verdict: str
    boundary: str
    missing_keys: tuple[str, ...]


def missing(required: Iterable[str], observed: Iterable[str]) -> tuple[str, ...]:
    return tuple(sorted(set(required) - set(observed)))


def grade(row: TraceRow) -> Grade:
    if row.observed_to_agent != row.expected_to_agent:
        return Grade(row.case_id, "routing_mismatch", "route", ())

    if not row.source_capture_complete:
        return Grade(row.case_id, "insufficient_evidence", "source_capture", ())
    source_missing = missing(row.required_keys, row.source_keys)
    if source_missing:
        return Grade(
            row.case_id,
            "upstream_context_missing",
            "before_handoff",
            source_missing,
        )

    if not row.payload_capture_complete:
        return Grade(row.case_id, "insufficient_evidence", "payload_capture", ())
    payload_missing = missing(row.required_keys, row.payload_keys)
    if payload_missing:
        return Grade(
            row.case_id,
            "packaging_loss",
            "source_to_payload",
            payload_missing,
        )

    if not row.destination_capture_complete:
        return Grade(
            row.case_id,
            "insufficient_evidence",
            "destination_capture",
            (),
        )
    destination_missing = missing(row.required_keys, row.destination_keys)
    if destination_missing:
        return Grade(
            row.case_id,
            "ingestion_loss",
            "payload_to_destination",
            destination_missing,
        )

    return Grade(row.case_id, "passed", "none", ())


if __name__ == "__main__":
    examples = [
        TraceRow(
            case_id="invoice-packaging-loss",
            handoff_id="h-002",
            expected_to_agent="billing",
            observed_to_agent="billing",
            required_keys=frozenset({"customer_id", "invoice_id"}),
            source_keys=frozenset({"customer_id", "invoice_id", "locale"}),
            payload_keys=frozenset({"customer_id", "locale"}),
            destination_keys=frozenset({"customer_id", "locale"}),
        ),
        TraceRow(
            case_id="payload-capture-redacted",
            handoff_id="h-003",
            expected_to_agent="billing",
            observed_to_agent="billing",
            required_keys=frozenset({"customer_id", "invoice_id"}),
            source_keys=frozenset({"customer_id", "invoice_id"}),
            payload_keys=frozenset({"customer_id"}),
            destination_keys=frozenset({"customer_id", "invoice_id"}),
            payload_capture_complete=False,
        ),
    ]

    for example in examples:
        result = grade(example)
        keys = ",".join(result.missing_keys) or "none"
        print(
            f"case={result.case_id} verdict={result.verdict} "
            f"boundary={result.boundary} missing={keys}"
        )

Running that file prints packaging_loss for the first case and insufficient_evidence for the second. The second row is important. Its stored payload view lacks invoice_id, but the row declares that view incomplete and the destination observed the key. Calling that context loss would blame the product for an instrumentation limitation.

The order of checks is intentional. A routing mismatch comes before context comparison because different destinations can have different contracts. Capture completeness comes before set subtraction because absence in a partial observation proves nothing. Source availability comes before envelope packaging, and packaging comes before ingestion. That is the causal order of the evidence.

The trade-off is schema work. Every production route now needs a reviewed contract, and every rename needs a versioned migration. That maintenance is real. It is still cheaper to review an explicit list than to debug a conversation where state can appear or disappear without a named boundary.

Separate three failures with the same visible symptom

Worked example one is the straightforward packaging defect. Triage stores customer_id, invoice_id, and locale. A helper builds the envelope from an outdated allowlist containing only customer_id and locale. Billing receives the correct route and handoff ID but asks for the invoice again.

The decisive trace sequence is:

• The source event lists invoice_id and marks source capture complete.
• The envelope event omits invoice_id and marks payload capture complete.
• The destination event also omits it.
• All three events share the same handoff ID and contract version.

That is enough to assign packaging_loss. You do not need a model grader to interpret the wording of the billing response. The response is supporting evidence, not the oracle. Fix the allowlist, add a negative test that would fail if invoice_id disappears again, and keep the contract scoped to the billing route.

Worked example two begins with nearly the same conversation but fails later. The payload contains both required keys. A receiving worker looks up destination state using a conversation ID while the producer wrote the state under the handoff ID. The worker initializes a new state record and billing asks for the invoice.

Here the envelope proves the producer did its job. Destination state is complete as an observation but lacks invoice_id. The first divergence sits between payload and destination, so the verdict is ingestion_loss. Adding invoice_id to the producer allowlist again would not help. It might even distract reviewers from the mismatched lookup key.

Look for correlation evidence before changing code. The envelope and destination acceptance event should carry the same opaque handoff ID. If they do not, you may be comparing different transitions. If they do match, inspect the destination's accepted key names and the state record identifier your application logs. Keep those identifiers synthetic in fixtures and opaque in production.

A pytest table makes the boundary rules executable without hiding cases inside a loop. Save the previous diagnostic as handoff_grader.py, then save this block as tests/test_handoff_grader.py. Pytest documents that parametrized values are passed as supplied, so the frozen dataclasses and frozensets also guard against accidental fixture mutation.

import pytest

from handoff_grader import TraceRow, grade


def row(**changes: object) -> TraceRow:
    values: dict[str, object] = {
        "case_id": "base",
        "handoff_id": "h-test",
        "expected_to_agent": "billing",
        "observed_to_agent": "billing",
        "required_keys": frozenset({"customer_id", "invoice_id"}),
        "source_keys": frozenset({"customer_id", "invoice_id"}),
        "payload_keys": frozenset({"customer_id", "invoice_id"}),
        "destination_keys": frozenset({"customer_id", "invoice_id"}),
    }
    values.update(changes)
    return TraceRow(**values)


@pytest.mark.parametrize(
    ("trace", "expected_verdict", "expected_boundary", "expected_missing"),
    [
        pytest.param(
            row(
                case_id="source-never-collected-invoice",
                source_keys=frozenset({"customer_id"}),
                payload_keys=frozenset({"customer_id"}),
                destination_keys=frozenset({"customer_id"}),
            ),
            "upstream_context_missing",
            "before_handoff",
            ("invoice_id",),
            id="upstream-not-handoff",
        ),
        pytest.param(
            row(
                case_id="receiver-dropped-invoice",
                destination_keys=frozenset({"customer_id"}),
            ),
            "ingestion_loss",
            "payload_to_destination",
            ("invoice_id",),
            id="destination-ingestion",
        ),
        pytest.param(
            row(
                case_id="router-selected-support",
                observed_to_agent="general_support",
            ),
            "routing_mismatch",
            "route",
            (),
            id="wrong-destination",
        ),
        pytest.param(
            row(
                case_id="destination-span-truncated",
                destination_keys=frozenset({"customer_id"}),
                destination_capture_complete=False,
            ),
            "insufficient_evidence",
            "destination_capture",
            (),
            id="partial-trace",
        ),
    ],
)
def test_grader_localizes_first_divergence(
    trace: TraceRow,
    expected_verdict: str,
    expected_boundary: str,
    expected_missing: tuple[str, ...],
) -> None:
    result = grade(trace)

    assert result.verdict == expected_verdict, trace.case_id
    assert result.boundary == expected_boundary, trace.case_id
    assert result.missing_keys == expected_missing, trace.case_id

Worked example three is the near-miss most likely to be mislabeled. The router sends the conversation to general support instead of billing. General support does not receive invoice_id because its own contract does not permit or require that field. It asks the customer for more information, and the log line looks like the previous failures.

The observed_to_agent field separates this case. Context transfer cannot be judged against the billing contract when billing was never selected. Grade the route first, report routing_mismatch, and then evaluate the actual support handoff against the support contract if that transition also matters. Otherwise the grader punishes data minimization for correctly withholding billing data from the wrong destination.

An incomplete trace is another near-miss. Sampling may retain the source event and destination response but omit the envelope event. Redaction may preserve an event while suppressing its key list. A collector failure may truncate the destination span. None of those observations prove transfer loss.

Mark completeness explicitly at the stage that produced the evidence. Do not infer completeness because a JSON object parsed successfully. A well-formed record can still be a deliberately reduced view. If the trace system cannot provide that signal, use insufficient_evidence and improve instrumentation before making the check release-blocking.

Retries can produce a subtler false match. The first handoff loses a field, the agent retries, and the second handoff succeeds. A timestamp-based query may pair the first source event with the second destination event. Stable handoff IDs prevent that splice. Also record an attempt identifier if multiple attempts can share a business case ID.

Read the failure report before blaming the model

Run the narrowest test that reproduces the transition. Pytest accepts a module, test node, or individual parametrized node on the command line. For the examples above, start with python -m pytest tests/test_handoff_grader.py -vv. The verbose report names the parameter case, which is more useful than a suite-level "context score failed."

When an assertion fails, pytest shows the expected and observed values involved in the expression. The test also attaches trace.case_id as the assertion message. A packaging test that unexpectedly returns passed should therefore identify both the mismatched verdict and the fixture case. Do not replace those assertions with one generic Boolean such as assert result.ok. Rich values are diagnostic evidence.

The trace review should answer these questions in order:

1. Does the route decision name the destination covered by the expected contract?
2. Do all stage events share the same handoff ID, attempt ID, and contract version?
3. Is capture marked complete at the source, envelope, and destination?
4. Which required keys exist at the source?
5. Which of those keys appear in the envelope?
6. Which envelope keys appear in destination state?
7. Did a retry or recovery happen after the first divergence?

Stop at the first unsupported answer. If the envelope span is missing, you cannot jump from source to destination and call the producer broken. If contract versions differ, you cannot use key comparison until you establish the intended migration rule. If the route is wrong, investigate routing first.

Specific error patterns point elsewhere. A ValueError from build_handoff saying required source context is missing: invoice_id means dispatch was prevented because upstream state was incomplete. That is not packaging loss. A RuntimeError saying handoff packaging removed required keys means the producer's projection contradicts its own contract. A pytest collection error means the test module or import setup failed before any trace was graded.

Do not use response text as the only discriminator. "Please share your invoice number" could be a recovery prompt after context loss, a legitimate prompt for a new invoice, or the intended behavior of a general support route. Bind conversational evidence to the deterministic trace row by case and handoff ID.

Tool activity can help, but only when your application records it under a documented schema. If billing calls an invoice lookup with the correct ID, that supports the claim that the value reached the point of use. Its absence does not prove the value was missing; the agent might choose another valid action. Use tool arguments as a separate consumption check with its own contract, not as a substitute for envelope and destination evidence.

Likewise, a correct final response does not repair a failed handoff. The destination may retrieve the missing invoice from an external system using customer_id. That recovery can be good product behavior and still cost an avoidable tool call. Report transfer and recovery independently: packaging_loss for the boundary, plus a successful recovery outcome if you measure recovery.

Logs need enough context to join records without exposing payload values. A practical event includes case_id for the test fixture, handoff_id for the transition, attempt_id for retries, route, contract_version, stage name, observed key names, and capture completeness. Keep the destination agent and verdict too. Raw prompts, authentication data, and customer content do not belong there by default.

If your viewer displays nested spans, use the handoff ID to filter and inspect the parent-child relationship as supporting evidence. Do not rely on color, visual adjacency, or span order alone. Concurrent agents can interleave events, and clock differences can make a later event appear earlier.

For a failed CI row, retain the exact synthetic fixture and grader version. A production trace can motivate a fixture, but copy only the minimum structure needed to reproduce the defect. Replace customer identifiers and text with synthetic values, preserve the key-presence pattern, and have someone review the sanitization before checking it in.

Wire the regression into CI without losing its evidence

Start with deterministic fixtures in the normal test process. There is no need to call a language model to prove that a required set is or is not a subset of an observed set. Keeping this layer offline removes model availability, sampling, and prompt drift from the transfer check.

This shell script runs only the handoff grader tests, writes a JUnit-style report, and saves Python logging to a separate file. Those command-line options are documented by pytest. Save it as ci/run-handoff-evals.sh and make it executable in the usual way for your repository.

#!/usr/bin/env bash
set -euo pipefail

artifact_dir="artifacts/handoff-evals"
mkdir -p "$artifact_dir"

python -m pytest \
  tests/test_handoff_grader.py \
  -vv \
  --maxfail=1 \
  --junitxml="$artifact_dir/results.xml" \
  --log-file="$artifact_dir/grader.log" \
  --log-file-level=INFO

The script does not install dependencies or assume a particular CI vendor. Your existing environment must already contain the project and pytest. Uploading the artifact directory is a separate platform-specific concern, so configure it with the CI system your team actually uses rather than copying an unverified key from an example.

Keep one test item per fixture. Parametrization gives each row a distinct node ID, and that ID appears in terminal output and reports. Use human-readable IDs such as destination-ingestion instead of a generated index. When a gate fails, the person on call should know whether to open producer code, consumer code, routing logic, or trace instrumentation before reading the full traceback.

Treat fixture validity as part of the gate. A row should fail review if it lacks a contract version, repeats a handoff ID across unrelated cases, or claims complete capture without the instrumentation needed to establish completeness. Those are data-quality errors, not passing product cases. Mixing invalid rows into the denominator of a quality score turns missing evidence into favorable arithmetic.

Keep expected verdicts in code review. If someone changes packaging_loss to passed merely to make CI green, the diff should expose that policy change. Avoid generating expectations from the implementation under test, because the same defect can then update both observed and expected values.

Negative fixtures deserve the same care as positive ones. Change one boundary at a time:

• Remove a required key from source state to exercise upstream_context_missing.
• Restore source state and remove the key only from the envelope to exercise packaging_loss.
• Restore the envelope and remove the key only from destination state to exercise ingestion_loss.
• Change only the observed destination to exercise routing_mismatch.
• Mark one capture incomplete to exercise insufficient_evidence.

Those mutations are not model-generated paraphrases. Each one challenges a branch in the causal classifier. If a fixture changes the route, payload, contract version, and trace completeness together, a failure cannot identify the cause.

Avoid a release threshold based only on an aggregate pass rate. One critical route can disappear inside a large collection of easy cases. Gate reviewed required routes individually, then use aggregate reporting for trend visibility if it helps the team. Never present an illustrative threshold as a measured reliability result.

Model-based scoring can sit after this deterministic layer when you need to judge whether the destination used context appropriately. Give that grader evidence already classified as complete, and ask one semantic question at a time. For example, a separate rubric can assess whether billing needlessly asked for a field known to be in destination state. Keep its judgment, reason, and rubric version apart from the transfer verdict.

The cost of CI evidence is artifact management. Verbose logs and XML reports occupy storage, and test matrices lengthen as routes multiply. Retention rules should match debugging needs and privacy constraints. Do not keep production payloads forever merely because a test runner can upload them.

Roll the check out without freezing the old design

Adding strict contracts to an established multi-agent system can break many tests at once, especially if agents have been reading a shared state object. Rollout needs to reveal dependencies before it blocks releases.

First, inventory actual routes from code and reviewed architecture, not from agent names seen in a handful of traces. For each route, name an owner, source, destination, and intended data purpose. Routes that cannot explain why a field crosses the boundary are not ready for an allowlist.

Second, define the smallest required and permitted key sets for one route with synthetic examples. Start with a path whose business behavior is well understood, such as triage to billing. Keep required keys distinct from permitted keys. A locale may be useful and permitted without being necessary for every billing request.

Third, emit stage events in observation-only mode. Check handoff IDs, attempt IDs, versions, and completeness flags. During this phase, report insufficient_evidence and mismatches without blocking. The purpose is to validate instrumentation and discover undocumented consumers, not to bless every behavior you observe as the desired contract.

Fourth, convert confirmed incidents into deterministic fixtures. A useful fixture preserves the causal shape: the source had invoice_id, a complete envelope did not, and billing asked again. Remove customer content and unrelated keys. Add a nearby passing control so a grader that labels everything broken cannot pass review.

Fifth, enforce the producer contract for the selected route. The explicit builder may now reject dispatches that previously limped forward. Decide the recovery behavior before enabling that check. The source might ask for the missing field, route to a human, or return a typed failure to the orchestrator. The right choice belongs to product design, but silent incomplete dispatch should not be the default.

Sixth, enforce destination ingestion and add the CI gate. Assign each verdict to a team. Packaging loss goes to the producer owner, ingestion loss to the consumer owner, routing mismatch to the router owner, and insufficient evidence to observability. Ownership prevents every failure from landing with the team that wrote the grader.

Shared-state systems need extra caution. A downstream agent may currently succeed because it can read fields that were never declared in a handoff. Moving to explicit envelopes will expose those dependencies. Decide whether the field belongs in the contract or whether the consumer should stop reading it. Copying the entire shared state into the envelope preserves the ambiguity and defeats the migration.

Contract versions should move independently of test case versions. A case may gain better evidence without changing route semantics, while a contract may rename a field without changing the incident scenario. Store both identifiers. During a rename, test old producer to migration adapter, adapter to new consumer, and the fully new path. Remove the adapter only after no supported producer emits the old version.

Retries need an explicit policy too. If a first attempt loses context and a second succeeds, decide whether the case should fail transfer integrity, pass recovery, or both. A sensible report records the first attempt's defect and the later recovery as separate facts. Replacing the whole case with the final successful attempt trains the suite to ignore wasted turns and hidden instability.

Do not backfill historical traces with assumptions. If older records lack capture-completeness flags, they are not equivalent to complete records. Use them to find candidate scenarios, not to calculate a definitive context-loss rate. New instrumentation can establish evidence prospectively.

Before expanding to another route, review the first route's false failures. Common causes include contract-version mismatches, retry events joined by case ID instead of handoff ID, and redacted payload views marked complete. Fix the evidence model rather than adding exception lists around each symptom.

The migration pays off when a failure points to a boundary and an owner without replaying a whole conversation. It does not remove the need for end-to-end tests. It gives those tests a sharper internal oracle so the team can separate a broken transfer from a bad route, an upstream omission, or a weak final answer.

Know the trade-offs and the cases this check cannot settle

Explicit handoff events increase instrumentation and storage. Each route needs contracts, each producer and consumer needs stage capture, and each retry needs a distinct transition identity. On a busy system, key-name events still add volume. Sampling can control cost, but sampled-out stages must become insufficient evidence rather than presumed passes.

Data minimization can conflict with debugging convenience. Storing every value would make comparison easy, but it creates privacy and access risks. Key-only evidence cannot detect a value that changed while retaining the same key. When value integrity matters, use synthetic end-to-end fixtures or a separately reviewed protected signal. Do not quietly widen production trace capture.

Strict producers can add user-visible friction. Rejecting an incomplete handoff may force another question or a fallback route. That cost is often preferable to downstream guessing, but the product team should choose the recovery. Tests can prove the contract was enforced; they cannot decide whether asking again is acceptable.

Route contracts also reduce flexibility. An experimental agent may legitimately discover and transfer new context that was not anticipated. A hard allowlist will drop it until reviewed. In an exploratory environment, run the grader as a diagnostic and log proposed contract additions instead of blocking every novel field.

Do not use this method when there is no actual handoff boundary. A single agent changing an internal plan is not a source-to-destination transfer, even if the trace displays multiple reasoning steps. Grade state continuity or tool usage with a contract suited to that architecture.

Do not call a tool outage context loss. If the destination receives invoice_id and a billing lookup times out, the handoff passed. The end-to-end task failed for another reason. Preserve the tool error and terminal status in a separate execution oracle.

Do not infer transfer loss from an agent's repeated question alone. The question may be required confirmation, a policy rule, or a response to conflicting values. Prove stage divergence first. If all required keys reached destination state, investigate instruction following or context use instead.

Do not grade against a destination contract after routing selected a different destination. The first failure is routing. Evaluating the wrong route's required fields can reward broad data sharing and punish correct isolation.

Do not block a release with partial captures. A missing envelope event, redacted key list, truncated span, or unmatched handoff ID makes the result inconclusive. Improve trace coverage, replay a synthetic case, or test the builder and consumer directly.

Do not require every available field to survive. The point is preserving required context while excluding unrelated data. An allowlist that intentionally drops an internal note is working. A grader based on full object equality would report that safety property as a defect.

Finally, do not ask a probabilistic judge to perform set arithmetic that normal code can perform exactly. Use model judgment for semantic questions such as whether the destination used available context sensibly or recovered helpfully. Use deterministic checks for routes, versions, required keys, capture completeness, and the first stage where evidence diverged.

When a case falls outside those boundaries, choose the oracle that matches the claim. Routing tests answer where work went. Contract tests answer what was transferred. Ingestion tests answer what the destination accepted. Trace-pipeline tests answer what was observed. Conversation-quality graders answer whether the user received a good result. Keeping those claims separate is what makes a failure actionable.