Evaluating Agent Handoff Routing and Context Transfer

Agent handoff evaluation has two independent jobs: decide whether control moved to the right owner, then prove that the new owner received the right evidence. A correct route with missing constraints can fail the user; a complete transcript sent to the wrong specialist can become a privacy incident.

The OpenAI Agents SDK handoff guide describes a handoff as a transfer to a specific agent represented to the model as a tool. By default, the receiving agent sees prior conversation history; input_filter can change that view. Structured input_type metadata is validated for the handoff callback but does not replace the specialist's main input or dispatch among destinations.

Freeze the handoff contract

List every legal destination and the authority it owns. For each route, specify positive intents, hard exclusions, required clarifying facts, permitted tools, escalation targets, and terminal outcomes. Include answer_here, ask_clarification, and human_escalation as first-class decisions rather than scoring only specialist labels.

Define transfer semantics separately. A support specialist may need account tier, issue summary, steps already attempted, and the user's requested outcome. It may not need authentication secrets, internal policy prompts, unrelated transcript turns, or data from another account. Make those fields machine-checkable in fixtures.

Also define ownership after transfer. Does the specialist answer the user directly, return a result to a manager, or hand back? Who may invoke a side-effecting tool? Without an ownership rule, a route can appear correct while both agents act or neither agent closes the task.

Build a routing corpus around boundaries

Easy, single-intent prompts establish basic reachability but do little for handoff risk. Concentrate cases near route boundaries: billing language inside a technical incident, account access mixed with cancellation, policy questions that mention a prohibited action without requesting it, and vague requests where clarification is safer than guessing.

Include multilingual phrasing, misspellings, short follow-ups whose meaning depends on history, conflicting user statements, tool outages, unavailable specialists, and attempts to name or instruct an internal agent. Add adversarial content that asks triage to ignore routing policy or smuggles a false specialist summary in quoted text.

Each example should carry allowed decisions, not always one forced label. Two specialists may both be valid for a mixed request if the policy permits either route. Record forbidden destinations and the expected next action after routing. This supports partial credit without forgiving a route that lacks authority.

Score decisions without hiding abstention

Report a confusion matrix across all decisions, including direct response, clarification, specialist routes, and human escalation. Aggregate accuracy alone can conceal a rare but dangerous route from a low-authority agent into an agent with destructive tools. Review results by intent, risk, language, ambiguity, and required permission.

Distinguish over-routing from under-routing. Over-routing adds latency, cost, repetition, and specialist load. Under-routing can produce unsupported answers or unsafe actions. A wrong-specialist error differs from a needless-specialist error and needs a different fix.

When a routing score or model confidence exists, calibrate it against observed correctness before using it as an automatic threshold. Thresholds are product policies, not universal constants. Keep an explicit ambiguity region where the agent asks a question or uses human review.

Inspect the exact context received

Capture three artifacts for every test: source conversation and trusted application state, the actual receiving-agent model input, and the locally available run context. Compare them structurally. A polished summary is not proof that dates, negations, units, identity, and user constraints survived.

Use required facts with unique fixture markers and prohibited facts with canary markers. Assert presence, absence, and provenance. Then vary transcript length and place a critical correction late in the conversation: "not account A; use account B." The specialist must receive the correction without preserving the contradicted fact as current truth.

The SDK guide separates several channels. input_type provides small model-generated handoff metadata such as reason or priority to on_handoff. RunContextWrapper.context carries local application state and dependencies. input_filter controls the history and generated items forwarded to the next agent. Tests should not assume one substitutes for another.

Minimize transfer while preserving provenance

Full-history transfer maximizes recall but also increases token use, distraction, and exposure. A generated summary is compact but can omit qualifiers or invent certainty. A structured envelope is easier to validate but requires a maintained schema. Many systems need a hybrid: bounded source excerpts plus structured current facts and references to authorized records.

Mark every transferred fact with source, observation time, and status such as user-claimed, tool-verified, or inferred. Specialists should not receive an inference dressed as a verified account fact. When context is trimmed, preserve unresolved questions and rejected options so the specialist does not repeat work.

Privacy filtering happens before model input. Redaction after tracing or after the specialist responds is too late. Test tenant isolation, role-based field suppression, credential removal, hidden system text, attachments, and tool outputs. A specialist should be able to request missing authorized evidence through a controlled tool instead of receiving an entire record by default.

Exercise the transfer lifecycle

Observe the sequence from route exposure to terminal response. The handoff call should be schema-valid, name one eligible destination, log the reason when required, transfer control once, and establish the receiving owner. Concurrent handoff calls need deterministic rejection or documented arbitration.

Test malformed metadata, callback failure, input-filter exception, destination timeout, destination cancellation, and a specialist that requests another handoff. Set a loop budget and retain the complete route path. A cycle such as triage to billing to triage must terminate in a useful escalation rather than consuming the entire run budget.

Input and output controls need boundary-aware tests. The handoff documentation notes that agent-level input guardrails apply to the first agent in the chain and output guardrails to the final-output agent. Do not assume each specialist automatically reruns them. Put checks around the actual risky operation or implement explicit specialist validation where the workflow requires it.

Attribute traces to the responsible agent

The Agents SDK tracing documentation describes traces composed of parent-linked spans and includes agent and handoff spans in default tracing. Use one stable evaluation case ID in trace metadata, then retain agent identity, route name, handoff input validation, parent span, start and end status, tool attempts, and final owner.

Trace attribution should answer: which agent selected the route, which filter produced the context, which agent called each tool, and which agent produced the user-facing output? Do not blame the specialist for a fact the triage filter removed. Conversely, a correct envelope followed by specialist misuse belongs to the specialist stage.

Tracing may contain model and function inputs or outputs. Apply the product's sensitive-data capture policy before evaluation export. Test reports should reference sanitized artifact IDs, not replay complete customer conversations.

Automate context and route assertions

This Python example models an evaluation record without depending on a particular router implementation:

from dataclasses import dataclass

@dataclass(frozen=True)
class HandoffExpectation:
    allowed_destinations: set[str]
    required_markers: set[str]
    prohibited_markers: set[str]
    maximum_handoffs: int

def assert_handoff(run: dict, expected: HandoffExpectation) -> None:
    assert run["destination"] in expected.allowed_destinations
    transferred = run["receiving_model_input"]
    assert all(marker in transferred for marker in expected.required_markers)
    assert all(marker not in transferred for marker in expected.prohibited_markers)
    assert len(run["handoff_path"]) <= expected.maximum_handoffs
    assert run["final_owner"] == run["destination"]

Markers work for controlled fixtures, while production-derived cases need field-level or semantic checks with human review. Add an outcome evaluator only after deterministic route, privacy, schema, and ownership checks. A fluent final answer must not override a prohibited transfer or duplicate side effect.

Illustrative policy configurations can make arbitration explicit:

{
  "label": "illustrative thresholds only",
  "automaticRouteMinimum": 0.82,
  "clarificationBand": [0.55, 0.82],
  "maximumHandoffs": 2,
  "criticalRouteErrorsAllowed": 0,
  "requiredReviewSlices": ["destructive-action", "identity-uncertain"]
}

Calibrate these values on adjudicated data and replace them when traffic or routing policy changes.

Use a concrete failure model

Inject each condition rather than waiting for stochastic occurrence. Repeat non-deterministic route cases enough to expose instability, but report repeated runs separately from unique corpus coverage.

Ship with a decisive handoff plan

1. Approve the route catalog, authority boundaries, clarification behavior, ownership model, and context schema.
2. Build adjudicated normal, ambiguous, mixed-intent, multilingual, adversarial, unavailable-destination, and high-risk cases.
3. Score routing, context completeness, context minimization, lifecycle integrity, and final resolution as separate dimensions.
4. Capture receiving input and local context with sanitized lineage so failures can be assigned to router, filter, specialist, or tool stage.
5. Fault-inject malformed metadata, filter errors, timeouts, cycles, duplicate actions, and guardrail-boundary mistakes.
6. Block release on prohibited transfer, unauthorized destination, duplicate side effect, unbounded loop, or missing trace ownership even when aggregate resolution looks strong.

A dependable handoff makes responsibility clearer, not murkier. The specialist should know what must be done, why it owns the task, which evidence is trustworthy, and which information it was never entitled to receive.