LLM-as-a-Judge Calibration Interview Questions and Scenario Answers

An LLM judge is a measurement instrument with failure modes, not an oracle attached to a test runner. Senior candidates should be able to define the construct being measured, calibrate the grader against adjudicated human evidence, expose bias, and decide where automation must yield to review.

These scenarios reward operational answers. Name the comparison set, rubric fields, randomization, disagreement artifacts, monitoring slices, and release consequences. A vague promise to "use another model" is not a calibration plan.

Establish the Calibration Contract

The official OpenAI Graders API reference documents grader configurations and validation. The interview challenge is broader: prove that a configured grader measures the intended quality criterion reliably enough for a particular decision.

The candidate should state what a false pass and false fail cost. That decision determines calibration depth, abstention policy, and whether a judge can gate a release or only prioritize human review.

Rubric and Construct Design

1. Why must the rubric name observable evidence rather than broad quality?

Terms such as "helpful" invite the judge to substitute its own preferences. I would decompose the construct into claims supported by the reference context, required task completion, prohibited content, and explicit presentation constraints. Each criterion includes positive and negative examples plus a not-applicable rule. Failure evidence includes rationales that cite no input fact or scores that change when irrelevant style is altered.

2. How would you calibrate a judge for policy-grounded customer support answers?

Build an independently adjudicated set spanning clear compliance, clear violations, partial answers, conflicting policy excerpts, missing evidence, and acceptable wording variants. Give the judge only the evidence available to the product. Compare criterion-level decisions, not just a total. Policy experts review disagreements and update either labels or rubric. A high-risk violation remains a deterministic veto even if the composite judge score passes.

3. Why should correctness and style be scored separately?

A polished unsupported answer can look preferable to a terse grounded one. Separate scores let the release policy prioritize factual support while still tracking readability. I would prevent style from compensating for a critical correctness failure, then inspect correlation between dimensions. If the judge repeatedly penalizes concise valid answers, transformed style-pair probes will reveal that the construct is contaminated by presentation preference.

4. How would you handle multiple valid reference answers?

Represent acceptable facts, constraints, and forbidden claims rather than requiring one surface string. The judge receives source evidence and a rubric describing permissible alternatives. Calibration cases deliberately use different wording, ordering, and levels of detail. If human experts disagree because the task itself is underspecified, mark it for abstention or revise the product requirement instead of training the judge to enforce accidental phrasing.

Human Evidence and Sampling

5. How would you create a trustworthy human comparison set?

Sample across risk and behavior slices, hide system identity, randomize output order, and obtain independent labels before adjudication. Reviewers record criterion decisions and evidence, not only a final preference. Include boundary cases and known judge traps. Keep the calibration set separate from prompt tuning, and preserve reviewer and rubric versions. The comparison is only as trustworthy as the independence and traceability of its labels.

6. Why is majority vote not always adequate ground truth?

Reviewers can share the same ambiguous instruction or lack domain authority. I would inspect disagreement reasons and route safety, legal, or domain questions to qualified adjudicators. Some cases should remain plural or unresolved. Majority vote is useful for clear subjective preferences, but it cannot manufacture source truth. A senior candidate recognizes when disagreement is measurement evidence rather than annotation noise to suppress.

7. How would you sample calibration cases when failures are rare?

Combine representative traffic with enriched known failures, adversarial constructions, and near-boundary outputs. Keep source proportions as metadata and report performance separately by sampling stratum. Otherwise an enriched set can be mistaken for production prevalence. I would refresh the rare-failure pool from incident reviews and disagreement mining while retaining a stable bridge subset for comparisons across judge changes.

8. Why inspect confusion matrices by criterion and slice?

One agreement figure can conceal that the judge misses unsupported citations while matching humans on easy formatting checks. For each criterion and risk slice, examine false passes, false fails, abstentions, and human uncertainty. Attach representative rationales and source evidence. The release decision should focus on costly error modes, not whichever aggregate is easiest to present.

Bias and Robustness Probes

9. How would you test position bias in pairwise grading?

Run both A/B and B/A orders with stable inputs, then compare preferences and ties. Randomize labels and prevent rationales from naming a position as evidence. Inconsistent pairs are reviewed by slice, especially near rubric boundaries. The official LangSmith pairwise evaluation guide provides a workflow for comparing experiment outputs; calibration still requires deliberate order counterbalancing and a tie policy.

10. Why use metamorphic transformations against a judge?

Transformations create cases where the expected judgment should remain stable: reorder equivalent bullets, change names unrelated to the criterion, normalize formatting, or shorten redundant prose without removing facts. A score shift supplies direct failure evidence because task quality did not change. Some transformations legitimately alter readability, so each probe declares which rubric dimensions must remain invariant and which may move.

11. How would you detect verbosity bias?

Construct paired outputs with identical supported claims but different redundant detail, and a second set where the longer answer adds an unsupported claim. A calibrated judge should not reward length automatically and must penalize the new error. Compare criterion rationales and preference consistency. I would also slice production disagreements by token or character bands, using them diagnostically rather than asserting a universal ideal length.

12. Why test adversarial candidate outputs against the judge prompt?

The evaluated output is untrusted data and may contain instructions aimed at the grader. Include outputs that request a high score, mimic rubric text, hide claims in formatting, or claim evaluator authority. The harness must delimit content, demand evidence from approved fields, and apply deterministic checks where possible. A successful attack is a security defect in the evaluation system, not an impressive model answer.

Thresholds, Ties, and Abstention

13. How would you select a release threshold?

Choose it from the cost of false passes and false fails on adjudicated calibration data, then validate by risk slice. Do not select the number that makes the current release pass. Document the tradeoff and add deterministic vetoes for critical criteria. Any numerical threshold shown in an interview should be labeled illustrative until measured against the organization's own evidence and decision costs.

{
  "policy": "illustrative_only",
  "overall_review_below": 0.74,
  "critical_grounding_veto": true,
  "abstain_on_missing_source": true,
  "required_slices": ["policy_conflict", "unsupported_claim", "valid_alternative"]
}

14. Why should a grader have an explicit tie policy?

Pairwise outputs can be genuinely equivalent, incomparable across dimensions, or unsupported by available evidence. Forcing a winner creates false precision and can amplify position bias. Define when to return tie, both-fail, or abstain, then calibrate those outcomes against human decisions. Release logic must know how each state aggregates; silently converting ties to one side corrupts experiment comparisons.

15. How would you design abstention for missing evidence?

Require the judge to identify the source fields needed for each criterion and return a structured abstention reason when they are absent or contradictory. Test malformed, truncated, and conflicting contexts. The pipeline routes high-risk abstentions to humans and reports their rate separately from failures. Treating an abstention as a pass rewards missing telemetry; treating every abstention as failure may block valid but ungradable cases.

16. Why might a judge be suitable for triage but not gating?

Calibration may show that it ranks obvious failures well but has unstable boundary decisions or unacceptable false passes in a critical slice. It can still prioritize human queues, cluster regressions, or generate hypotheses. The operating policy should name that limited purpose and prevent downstream consumers from reusing the score as a release gate. Capability claims must match measured reliability and consequence.

Implementation and Reproducibility

17. How would you make judge runs reproducible enough to investigate?

Persist input identifiers, candidate output, evidence bundle, grader configuration, rubric version, execution parameters, structured result, rationale, and retry history. Hash immutable artifacts and keep access controls around sensitive prompts. Reproduction may still encounter nondeterminism, so rerun policies should preserve every attempt instead of replacing the first result. Investigation needs the distribution of outcomes and exact context, not a dashboard average.

18. Why validate structured grader output before aggregation?

Malformed fields, out-of-range values, missing criteria, or rationale-score contradictions can otherwise become zeros or passes through permissive coercion. Validate against a schema, reject unknown critical states, and route invalid records to an error lane. Track invalidity as grader reliability evidence. A retry can recover transient formatting, but retaining the failed attempt distinguishes grader instability from application quality.

Monitoring and Change Control

19. How would you detect judge drift after deployment?

Continuously sample judged cases for blinded human review, preserve a stable sentinel set, and monitor disagreement by rubric criterion, language, task family, and output form. Investigate changes in abstention, invalid output, and rationale evidence. Drift can originate in the judge, rubric, application distribution, or reference context, so diagnosis compares all four lineages before recalibration.

20. Why rerun calibration after a judge prompt or model change?

The grader itself is part of the measured system. A new prompt or underlying model may shift severity, bias, formatting, or interpretation even when the application outputs are unchanged. Run old and new judges on the same blinded bridge set, adjudicate changed decisions, and version the result. Promote only when critical errors remain acceptable for the intended use; lower cost alone does not establish equivalence.

Make the Hiring Decision

Strong candidates make the judge falsifiable. They specify cases that should reverse a score, cases that should leave it invariant, and evidence that sends automation to a human. They distinguish grader invalidity from product failure and pair statistical summaries with inspected disagreements.

The decisive senior answer limits the judge to the role its calibration supports. Hire the candidate who will reject an attractive automated score when critical slice evidence, adversarial probes, or human adjudication shows the instrument is not fit for that decision.