OpenAI Grader and Agent Trace Interview Scenarios

A senior OpenAI eval interview should connect three layers: the eval definition that specifies evidence, the grader that applies a criterion, and the agent trace that explains execution. Candidates who discuss only final-answer scoring miss unauthorized tools, duplicate side effects, broken guardrails, and recovery paths that happened to end well.

These 23 scenarios test data contracts, grader selection, calibration, trace interpretation, and release judgment. Strong answers choose deterministic evidence whenever possible, treat model grading as a measurement system that needs validation, and preserve run and trace identifiers for failed-case diagnosis.

Connect Eval, Grader, and Trace Evidence

The OpenAI Evals API reference describes eval definitions, data-source schemas, runs, and run output items. The Graders API reference includes deterministic string checks and text-similarity graders among the available grader structures. Agent execution evidence belongs beside those results, not hidden behind the aggregate.

The interviewer's recurring probe should be, "What artifact would let another engineer reproduce that verdict?" This prevents a dashboard status from standing in for an evidence chain.

Define the Eval Data Contract

1. Why should an eval data schema be designed before a grader prompt?

The schema determines which facts the grader can inspect and whether cases remain reproducible. I would define user input, reference properties, candidate output, risk tags, context or tool expectations, and provenance before writing criteria. Validate required fields at ingestion. A clever grader cannot recover a missing authorization state or obsolete reference. Schema discipline adds setup work, but it keeps rubric changes from quietly inventing evidence that the run never captured.

2. How would you represent multiple acceptable answers in an eval item?

Store acceptable properties, reference facts, or a set of allowed outcomes rather than one canonical string when wording is open. Deterministic graders can check structural properties, while a semantic grader can judge satisfaction of the explicit facts. Preserve prohibited claims separately. This reduces false failures from paraphrase without declaring every plausible answer correct. The tradeoff is more label design and adjudication at dataset creation time.

3. What would you do when an eval run contains malformed data items?

Reject or quarantine them before application generation and report item IDs, schema errors, and affected slices. Do not score missing fields as model failures or silently drop rows from the denominator. Fix the dataset through a versioned change, then create a new run. If malformed items cluster in one source, investigate the ingestion pipeline. Data reliability and application quality need separate dispositions even when they share one job.

Choose the Narrowest Valid Grader

4. Why use a string-check grader for an exact classification contract?

If the output must equal one of controlled labels, semantic judgment adds unnecessary ambiguity. Normalize only transformations the product allows, then apply exact or case-aware string checks and report unexpected labels. This is deterministic, fast, and easy to reproduce. It is inappropriate when explanatory text or multiple semantic equivalents are allowed, so the candidate must connect the grader to the actual interface contract rather than default to exact matching.

5. When can text similarity misrepresent answer correctness?

A response can share words with a reference while reversing a fact, or express the same correct meaning with different language. I would test the similarity measure on negation, reordered facts, concise paraphrases, and templated filler before using it. Similarity may support retrieval or wording analysis, but high-consequence factual decisions need property checks, source-grounded judging, or human review. A threshold cannot repair a metric that measures the wrong relationship.

6. How would you grade a structured tool-call decision?

Parse the output into a schema and deterministically validate tool name, required arguments, authorization scope, value bounds, and forbidden fields. A semantic grader can assess whether the selected action addresses the intent, but it should not replace structural safety checks. Preserve parse errors as their own failure. This layered approach costs more implementation than one broad judge, while providing precise ownership and protecting hard invariants.

7. Why should several graders not be averaged automatically?

The graders may represent incomparable criteria and asymmetric harm. A policy violation should not be canceled by high style or similarity. Define hard vetoes, required criteria, advisory diagnostics, and any weighted combination before seeing results. Keep component outputs in the artifact. Multi-criterion policy is useful, but an unexplained mean launders severe failures into a smooth score and makes remediation unclear.

{
  "grading_policy": "illustrative_layered_policy",
  "veto": ["unauthorized_tool", "sensitive_data_exposure"],
  "required": ["schema_valid", "reference_facts_supported"],
  "advisory": ["clarity", "response_similarity"],
  "thresholds": "derive_from_calibration_not_this_example"
}

Validate and Defend Graders

8. How would you calibrate a model-based grader for a domain rubric?

Create independently labeled clear passes, clear failures, and boundary cases across product slices. Run the grader blind, compare decisions and reasons with adjudicated labels, and inspect severe false passes separately from broad agreement. Revise the rubric on a development subset and retain a holdout. Record evaluator and rubric identities. A model grader is fit for a release only to the extent this evidence supports its intended criterion.

9. What adversarial outputs would you use to test a grader?

Include outputs that address the grader directly, claim they deserve a high score, embed rubric-like instructions, hide a prohibited statement in verbose text, copy reference phrases while contradicting them, and exploit malformed structure. Treat candidate output as untrusted data and delimit it from grader instructions. The grader should either judge correctly or fail closed into review. Red-team cases test measurement integrity, not the application alone.

10. How would you handle a grader that changes verdict on identical input?

Preserve every result and reason, then assess repeatability on the calibration set under a pinned configuration. Separate infrastructure errors from valid but unstable judgments. Clarify the rubric, constrain output structure, or route boundary cases to a second judgment or human review. Never retry until the desired answer appears. Variability can be acceptable for exploratory triage, but a blocking policy needs a documented rule for uncertain cases.

Read Agent Traces as Behavioral Evidence

11. Why can final-output grading miss a critical agent defect?

An agent may call an unauthorized service, expose data in tool arguments, repeat a purchase, ignore a guardrail, or recover from a wrong handoff before producing a correct sentence. I would define trajectory invariants and inspect the trace alongside outcome graders. The Agents SDK tracing guide covers generation, tool, guardrail, and handoff spans, which are the operations QA must connect to product rules.

12. How would you grade whether an agent chose the correct tool path?

Represent required, allowed, and forbidden operations plus ordering or precondition constraints. Evaluate the trace as a structured sequence, ignoring harmless planning variation while enforcing safety and task invariants. For example, authorization must precede a write and a read-only request must not invoke a mutation. Exact full-path matching is brittle; invariant-based grading retains flexibility while still detecting dangerous trajectories.

13. What trace evidence would you inspect after a duplicate side effect?

Follow the request through orchestration and tool attempt spans, checking operation or idempotency keys, retry causes, timing, tool results, and checkpoint or handoff boundaries. Correlate with the external service ledger. A duplicated effect may come from retry after ambiguous commit rather than two planner decisions. The trace proposes the mechanism; external records confirm it. Fix verification must inject the same boundary failure and prove one logical operation.

14. How would you evaluate a handoff that reaches the right specialist too late?

Separate routing correctness from efficiency and user impact. Check preceding agent turns, tool calls, guardrail outcomes, handoff span, and the information transferred. Grade whether escalation conditions were met at the first eligible point and whether context was minimized but sufficient. A late correct handoff can increase cost, latency, and exposure. Requiring immediate handoff may be too strict when clarification is necessary, so encode the allowed preconditions.

Join Grader and Trace Results

15. How would you correlate an eval output item with its trace?

Carry a stable run-item identifier into application trace metadata and return the trace identifier as a safe artifact reference. Validate one-to-one or documented one-to-many relationships and authorize access separately. The join should survive asynchronous execution and retries. Avoid putting raw prompts or secrets into metadata. Without this correlation, a low grader score cannot reliably identify which tool path and generation produced the output.

16. What if the grader fails but the trace satisfies every trajectory invariant?

Inspect the output criterion, reference data, and grader reason. The agent may execute safely yet answer incorrectly, incompletely, or unclearly; trajectory success does not imply outcome quality. Alternatively, the grader or label may be wrong. Replay the immutable output through calibration and obtain human review for critical disagreement. Keep the two contracts separate so a clean path cannot overrule a bad result, and a good result cannot excuse a bad path.

17. What if the final answer passes but the trace grader fails?

Classify the violated trajectory rule and its consequence. Unauthorized calls, leaked content, or duplicated writes are blockers even if the answer is useful. For an efficiency-only rule, quantify cost and latency impact and apply the declared policy. Review whether the invariant is overly prescriptive about harmless planning variation. The release decision follows the risk represented by the failed path, not the visual success of the final response.

Operate Eval Runs and Release Gates

18. How would you make an eval run reproducible?

Pin dataset or data-source revision, application configuration, prompts, tool schemas, retrieval corpus identity, grader definitions, and environment metadata. Preserve run and output-item identifiers plus trace links. If external dependencies cannot be frozen, record responses or declare the limitation. Reproducibility does not require every generation to be identical; it requires enough lineage to distinguish application behavior from changed evidence or measurement.

19. What should a release report show beyond aggregate pass rate?

Show item completion and grading errors, hard-veto counts, per-grader results, risk slices, paired baseline deltas, disputed cases, and representative trace diagnoses. Include dataset and grader versions and accepted exceptions. Do not invent a universal numeric threshold; mark any design numbers illustrative until calibrated. The report should let an approver see which product claims were tested and where evidence remains missing.

20. How would you stop a long eval run without losing decision integrity?

Cancel through the supported run control, preserve completed output items, and mark the run incomplete rather than extrapolating a pass. Record why it stopped and which slices are underrepresented. A later run may resume operational work, but its results need a clear relationship to the original dataset and configuration. Partial evidence can support debugging; it should not silently satisfy a gate that required full critical-slice coverage.

Protect the Measurement System

21. Why should grader inputs be treated as a trust boundary?

Candidate outputs, retrieved text, and tool results can contain instructions aimed at the grader or data intended to escape delimiters. Build structured templates, isolate untrusted fields, validate lengths and types, and test injection cases. Restrict secrets in evaluator context. A grader compromise can approve unsafe behavior across many runs, so its prompt and parser deserve security tests comparable to the application being measured.

22. How would you control sensitive data in agent traces used for eval diagnosis?

Classify fields before capture, redact at span producers and again before export, use safe identifiers for joins, limit retention, and require audited access to restricted payloads. Inject canary secrets through tool and error paths to test leakage. Structural spans can often diagnose ordering without raw content. More redaction reduces semantic evidence, so document unresolved hypotheses instead of broadening capture informally during an incident.

23. How would you score a candidate who proposes one model grader for every criterion?

Score below senior level until they separate exact contracts, semantic judgments, trajectory invariants, and human escalation. Ask how they calibrate severe false passes, defend against grader injection, and diagnose a failed output with its trace. One judge is convenient but creates a broad correlated failure domain. Senior design uses the narrowest valid evidence and preserves disagreement where no single grader has earned authority.

Require Agreement Across the Evidence Chain

OpenAI eval resources can organize data, runs, graders, and output items, while agent traces reveal execution. Their value comes from a QA policy that defines which grader proves which criterion and how trajectory failures affect release status.

The decisive close is simple: never let a passing answer hide an unsafe path, and never let a sophisticated grader outrank calibrated source truth. Preserve the failed item, its grader reason, and its trace; test the proposed cause; then release only when outcome and execution evidence satisfy their separate contracts.