Agent Trajectory Evaluation Interview Questions and Answers

Agent evaluation becomes senior-level when the candidate can explain not only whether a task completed, but how the system changed state on the way there. A final answer can look correct after an unauthorized tool call, a lucky retry, a hidden handoff failure, or an expensive loop. The trajectory is where those defects become testable.

These scenarios ask for a trace contract, a path rubric, outcome evidence, and a release decision. Strong answers permit legitimate variation while making unsafe transitions, missing observations, and unbounded recovery unmistakable.

Turn a Run into Evidence

The official OpenAI Agents SDK tracing documentation describes traces and spans for agent runs, including model generations, tool calls, handoffs, and guardrails. Evaluation adds a second contract: decide which captured events prove or disprove safe task completion.

The trace should support reconstruction without becoming an unrestricted copy of every secret the agent observed. Observability and data minimization must be designed together.

Trace Capture and Identity

1. Why must every trajectory have stable run, step, and parent identifiers?

Concurrent calls and nested handoffs otherwise appear as a misleading flat log. I would assign a run identifier, span identifier, parent span, attempt number, agent identity, and monotonic event order at capture. Tool requests and results share a correlation key. Failure evidence includes orphan results, duplicate step identities, or a final outcome that cannot be linked to the state-changing call that produced it.

2. How would you define a minimum trace contract for tool-using agents?

Capture initial task and policy context identifiers, redacted state fingerprints, model decisions, selected tool and schema version, validated arguments, authorization decision, result class, state delta, retries, handoffs, termination reason, and final verification. Large payloads can live in access-controlled artifacts referenced by hash. Missing fields should mark the trajectory ungradable, not silently convert unknown behavior into a pass.

3. Why record observations separately from agent decisions?

The agent's statement that a tool succeeded is not proof that the tool did. Preserve the returned status, authoritative resource identifier, and verified state independently from the next model message. This separation reveals fabricated success, stale observations, and parsing defects. For a calendar action, the decision is "create event," the observation is the API result, and the outcome evidence is a read-back of the intended event.

Outcome and Path Rubrics

4. How would you score a correct final answer reached through a forbidden tool?

Outcome correctness may pass, but policy compliance and path safety fail with a release-blocking severity appropriate to the tool. I would never average the violation away with a good final response. The artifact should show the selected tool, policy decision, arguments, side effect, and whether containment worked. This distinction prevents lucky outcomes from teaching the team that prohibited paths are acceptable.

5. Why is exact tool-sequence matching usually too brittle?

Agents may gather equivalent evidence in different orders, skip cached reads, or use a valid specialized tool. Instead, define partial-order constraints and invariants: authorization before write, required evidence before decision, no forbidden domain, and final state verification after mutation. Exact sequences remain useful for deterministic workflows, but applying them universally turns harmless variation into false failures and encourages overfitting to one trace.

6. How would you evaluate a research task with several valid trajectories?

Score source eligibility, claim-to-source support, required coverage, conflict handling, and citation validity. Permit different query and reading paths while bounding call count and prohibiting untrusted instructions from changing the task. Compare final claims with captured sources, then inspect whether every decisive claim has an observation. The rubric evaluates evidence properties rather than prescribing which search phrase must occur first.

7. Why should state validity be an outcome dimension?

An agent can tell the user a change succeeded while leaving duplicate, partial, or unauthorized state. Define postconditions against the authoritative system and verify them independently. For a ticket update, check target identity, expected fields, audit actor, and absence of unintended changes. A message saying "done" is presentation; the state transition and read-back are the test evidence.

Tools, Handoffs, and Control Flow

8. How would you test whether tool selection is appropriate?

Create tasks where tools overlap in description but differ in privilege, scope, cost, or data freshness. Label acceptable choices and prohibited choices with reasons. Inspect selected schema, arguments, and resulting state, not only the tool name. A senior design also tests the no-tool path when the request is informational or lacks authorization. Selection quality must include restraint.

9. Why evaluate argument construction separately from tool selection?

The correct tool can still target the wrong tenant, omit a filter, invert a date range, or broaden a destructive operation. Validate arguments against schema and task-derived constraints before execution, then score semantic binding to the user's intent. Store redacted argument diffs. This separates planning failure from connector failure and supports deterministic vetoes for dangerous scope expansion.

10. How would you evaluate an agent-to-agent handoff?

Trace the routing reason, source agent, destination capability, transferred context, omitted secrets, open obligations, and return contract. Test correct routing, unnecessary handoffs, cycles, and context loss. The receiving agent should not infer authorization that was not transferred explicitly. Outcome review verifies both task completion and continuity, such as whether the specialist addressed the unresolved constraint rather than restarting generically.

11. Why should handoff loops be detected by state, not names alone?

Two agents may legitimately alternate as new evidence appears, while one agent can loop internally without a handoff. Fingerprint relevant task state, unresolved obligations, and recent actions. Repeated fingerprints without new evidence consume a loop budget and trigger a safe stop. Preserve the cycle segment and termination reason. A name-based limit misses semantic loops and can also stop valid collaboration too early.

{
  "run_id": "run-illustrative-41",
  "step_id": "step-07",
  "parent_step_id": "step-04",
  "actor": "billing-specialist",
  "action": "lookup_invoice",
  "authorization": "approved_read",
  "state_before": "sha256:before",
  "state_after": "sha256:after",
  "observation_status": "success"
}

Recovery, Loops, and Efficiency

12. How would you distinguish useful retry from a loop?

A useful retry responds to a classified transient failure and changes a relevant condition such as delay, endpoint, or reduced request. A loop repeats materially equivalent state and action without new evidence. I would cap attempts by operation risk, record retry reason and backoff, and require idempotency for writes. The evaluation reports recovery success separately from first-attempt success so fragility remains visible.

13. Why can fewer tool calls be a misleading efficiency metric?

An unsafe shortcut may use one privileged mutation while a careful path verifies identity, reads current state, seeks approval, writes, and confirms. I would measure necessary work, redundant work, critical-path latency, and outcome reliability under constraints. Call count is diagnostic, not a universal objective. Efficiency scoring must never reward skipping authorization, evidence gathering, or postcondition checks.

14. How would you test recovery from a partial tool failure?

Inject failures before submission, after acceptance but before response, and during read-back. The agent must distinguish unknown outcome from confirmed failure, query authoritative state, and avoid blind duplicate writes. Evaluate retry, compensation, user communication, and final state. Preserve request identifiers and simulated system records. This reveals whether the trajectory handles ambiguity or merely retries every exception.

15. Why should termination behavior be part of trajectory evaluation?

Agents need explicit completion, safe refusal, budget exhaustion, and unresolvable-error states. Test contradictory instructions, missing credentials, repeated empty results, and tasks whose postconditions cannot be proven. A strong trajectory stops with a specific unresolved obligation and no unauthorized side effect. Continuing indefinitely is a defect; claiming completion without evidence is a different and often more dangerous defect.

Evaluation Implementation and Privacy

16. How would you compare trajectory versions after a prompt change?

Replay a frozen scenario set in isolated environments and align traces by semantic stages rather than raw step number. Compare outcome, required invariants, forbidden transitions, tool arguments, retries, and efficiency bands. Review changed paths with the same rubric and retain both artifacts. New valid paths should not fail merely because they differ, while newly introduced policy violations must remain visible even if completion improves.

17. Why should trajectory graders combine deterministic and model-based checks?

Schemas, call ordering, authorization-before-write, forbidden tools, budgets, and state postconditions are deterministic. Relevance of evidence or quality of a recovery explanation may need rubric-based review. Use deterministic failures as vetoes and calibrate model-based judgments against humans. This division makes critical invariants reproducible while allowing nuanced assessment where exact rules would be brittle.

18. How would you prevent trace observability from leaking secrets?

Classify fields before instrumentation, redact at the capture boundary, tokenize or hash identifiers needed for correlation, and restrict payload artifacts by role and retention. Test nested objects, tool errors, model messages, and third-party spans for bypasses. Store a redaction reason and schema version. Post-processing alone is insufficient because raw telemetry may already have crossed a trust boundary.

Release and Incident Decisions

19. How would you build a trajectory release gate?

Define critical scenarios, required state postconditions, deterministic safety vetoes, acceptable ungradable rate, calibrated path criteria, and efficiency guardrails. Use clean isolated starting state and repeat cases where nondeterminism matters. Every failure links to the shortest relevant trace segment and authoritative state. Numerical limits are illustrative until derived from product risk and operational evidence, not copied from another agent.

20. Why must production trajectory incidents feed offline evaluation?

Production reveals tool responses, state combinations, and user phrasing the lab missed. Redact the incident, reconstruct a minimal reproducible environment, label the unsafe transition and desired recovery, then add related blind variants. Do not only add the exact trace after developers inspect it. The regression proves the known defect stays fixed; unexposed variants test whether the correction generalizes.

Give the Seniority Signal

Strong candidates describe trajectories as stateful evidence, allow multiple legitimate paths, and refuse to let a successful final sentence erase an unsafe transition. They make retries, handoffs, observations, and postconditions independently inspectable.

The hiring decision should favor the person who can say exactly what must be captured, what may vary, what can never occur, and which authoritative state proves completion. That is the foundation of a release gate that survives real agent behavior.