Building Human-Adjudicated Gold Sets for LLM Judge Calibration

An LLM judge is not calibrated because its rationale sounds thoughtful. Calibration requires a dataset whose labels came from independent human judgment, explicit conflict resolution, and preserved evidence. Without that foundation, agreement with the judge can mean nothing more than agreement with one hurried annotator.

Build the gold set as a measurement asset, not a folder of favorite examples. It needs scope, sampling rules, reviewer qualifications, blind labeling, adjudication, versioning, and a decision policy for judge errors. The OpenAI graders API reference describes structured grader configurations, but the validity of the labels used to evaluate any grader remains an editorial and domain responsibility.

Specify the judgment task first

Define exactly what the judge will score and what evidence a human may use. Correctness against a supplied policy excerpt is a different task from helpfulness without a reference. A pairwise preference label is different from a four-level safety classification. Mixing them in one gold field makes disagreement impossible to interpret.

Write a label ontology with observable criteria and precedence. For example, pass, minor_error, major_error, and not_applicable need concrete boundaries. Explain whether a missing caveat is minor or major, when a safe refusal is correct, and when evidence is too weak to decide. Add counterexamples for neighboring labels rather than long lists of adjectives.

Define the intended use. A judge calibrated for low-risk writing tone must not automatically approve medical correctness or policy compliance. Store an applicability statement with the dataset: supported task, languages, input formats, risk classes, and exclusions. Calibration is conditional on that scope.

Sample for coverage, not convenience

Start with a coverage matrix derived from production intents, known defects, policy risks, and output forms. Include routine cases so the judge is not calibrated only on spectacular failures. Deliberately add close boundaries: fully correct versus correct-with-omission, acceptable refusal versus excessive refusal, and grounded inference versus unsupported assertion.

Preserve natural prevalence in one reporting view, but oversample rare critical failures for diagnosis. Tag the sampling stratum so a deliberately balanced calibration set is not later presented as production incidence. Keep prompt authors from selecting only examples that make their preferred judge look good.

Avoid duplicates and near-duplicates across development and held-out calibration partitions. Outputs generated from the same source prompt with superficial paraphrases can leak the answer pattern. Group related cases before splitting. Record source, consent or privacy treatment, generation configuration, and any transformation.

Design blind independent review

Each item should receive independent preliminary labels. Reviewers must not see the model name, candidate status, expected release outcome, or other reviewers' choices unless one of those facts is explicitly part of the rubric. Randomize display order for pairwise items and remove stylistic identity markers when feasible.

The LangSmith annotation queue guide documents prescribed rubrics, multiple reviewers, pairwise review, and reviewer workflows. Whatever platform you use, capture reviewer ID or pseudonymous code, rubric version, label, confidence category, rationale, evidence spans, start and completion state, and escalation request.

Train reviewers on a separate practice set. Discuss disagreements there, revise confusing rubric text, and repeat until the team can explain boundaries consistently. Do not consume the held-out gold candidates during training, because remembered adjudications will make later independence artificial.

Store evidence in a reviewable record

The record should distinguish a reviewer's preliminary judgment from the final adjudicated label. A compact JSON shape makes that lineage inspectable:

{
  "caseId": "refund-policy-017",
  "rubricVersion": "refund-correctness-v3",
  "inputRef": "dataset://support/refund-policy-017",
  "outputHash": "sha256:example-placeholder",
  "reviews": [
    {"reviewer": "r12", "label": "major_error", "confidence": "high", "evidence": ["policy:14-18"]},
    {"reviewer": "r27", "label": "minor_error", "confidence": "medium", "evidence": ["policy:14-18"]}
  ],
  "adjudication": {
    "label": "major_error",
    "reason": "The answer promises eligibility that the supplied policy excludes.",
    "status": "resolved"
  }
}

The hash detects accidental output changes without putting sensitive text into every report. Evidence references should resolve only for authorized reviewers. Store tie, multiple_valid, insufficient_evidence, or rubric_gap when those are the honest outcomes. A fabricated certainty is worse calibration data than an explicit unresolved item.

Measure agreement before adjudication

Pre-adjudication agreement reveals whether reviewers can apply the rubric. Report the raw label matrix and per-class disagreements. A summary statistic can help comparison across rubric revisions, but always include prevalence and confusion details because chance-corrected values can be difficult to interpret when one label dominates.

Examine disagreement by intent, language, response length, evidence type, severity, and reviewer experience. A rubric may be stable for direct factual errors yet unclear for omission severity. Do not average that boundary away. Review rationales to separate annotation mistakes from missing guidance.

Use confidence as triage metadata, not a vote multiplier invented after labels arrive. A high-confidence conflict is especially informative: two reviewers believe the rubric supports opposite labels. That usually deserves adjudication and perhaps a rubric amendment.

Adjudicate conflicts without rewriting history

An adjudicator should see the original item, rubric, evidence, and independent rationales after preliminary review closes. The goal is to determine the defensible label or document why no single label is warranted. The adjudicator should not be told which result will help a model pass.

For domain-critical cases, use a qualified specialist and define when a panel is required. Adjudication outcomes should include final label, rationale, evidence, participants, date, and whether the rubric changed. Preserve the preliminary labels; they are valuable for measuring ambiguity and reviewer drift.

If adjudication exposes a rubric gap, update the rubric as a new version and identify every affected item. Re-review those items under the new wording. Changing a label in place while retaining the old rubric version creates an internally inconsistent gold set.

Calibrate the judge by error class

Freeze the judge prompt or configuration before running the held-out set. Compare its structured verdict with the gold label, and capture invalid output, refusal, timeout, and parsing failure separately. Build a confusion matrix and inspect error cost. A false pass on a critical policy violation is not equivalent to calling a clean response minor_error.

Measure exact agreement, class recall where missing a class matters, and false-pass or false-fail rates at the operational decision boundary. Add intervals over cases so small slices are visibly uncertain. If repeated judge calls are part of production policy, run the same repetition and aggregation rule during calibration rather than testing an easier one-shot version.

Do not tune on the held-out results repeatedly. Judge prompt edits, rubric edits, or model changes return to the development partition, then receive a fresh held-out evaluation. When the held-out set becomes familiar through repeated error analysis, retire or rotate a documented portion.

Define use boundaries and vetoes

Map calibration evidence to permitted automation. A well-performing judge on supported low-risk slices may auto-label monitoring samples. Borderline labels can route to human review. Critical decisions can require deterministic checks or specialist approval regardless of model-judge agreement.

The following is an illustrative policy, not a recommended universal threshold:

{
  "policyLabel": "illustrative-only",
  "supportedLabels": ["pass", "minor_error", "major_error"],
  "unsupportedSlices": ["new-language", "legal-advice"],
  "criticalFalsePassLimit": 0,
  "minimumHeldOutCasesPerReportedSlice": 40,
  "uncertainDisposition": "human-review",
  "invalidVerdictDisposition": "evaluation-failure"
}

The zero critical false-pass rule is an example of a veto, not a claim that a finite sample proves zero future risk. State what the test observed and retain monitoring. If a slice lacks evidence, mark it unsupported rather than borrowing confidence from unrelated cases.

Govern the gold set as a versioned asset

Release a dataset version with checksums, rubric version, label schema, sampling notes, adjudication log, supported scope, and partition policy. Corrections should produce a changelog that states whether historical calibration results remain comparable. Keep access controls appropriate to production-derived content.

Monitor a fixed sentinel subset for reviewer and judge drift, then add newly adjudicated production failures through a controlled intake. Do not let every incident immediately rewrite the release benchmark. First establish provenance, review it independently, adjudicate it, and decide which partition it belongs to.

Track label appeals. A product or policy owner may discover that an old criterion no longer matches desired behavior. That is a specification change, not a judge improvement, and reports should show the boundary between old and new series.

Follow a decisive build plan

1. Write the task scope, label ontology, evidence rules, exclusions, and operational use before collecting labels.
2. Build a stratified candidate pool with routine, boundary, critical, and known-failure cases; group related examples before partitioning.
3. Train reviewers on separate practice data, then collect blinded independent labels and rationales on gold candidates.
4. Measure pre-adjudication disagreement, route conflicts and low-confidence items, and preserve uncertainty where the rubric cannot decide.
5. Freeze adjudicated labels with lineage, checksums, rubric version, and supported slices; protect a held-out calibration partition.
6. Evaluate the frozen judge by class, risk, and slice with uncertainty, then apply predeclared human-review and veto boundaries.
7. Version every change and maintain sentinels plus controlled production intake so calibration remains current without becoming mutable evidence.

A gold set is trustworthy when another qualified reviewer can reconstruct why each consequential label exists. That traceability is what turns a model judge from a convenient opinion generator into a bounded measurement instrument.