A grader is not reliable until it fails your calibration set

A model update clears the eval gate, yet the first reviewer spots an answer that invents a refund policy. The model did not beat the test. The grader accepted behavior the team never meant to allow.

That failure is easy to miss because a grade looks like evidence. It is only another program output. Before a release can depend on it, QA has to test the criterion, its inputs, its reference data, and the decisions it makes at the boundary.

Treat the grader as code you have to test

OpenAI's eval workflow separates the data schema from the testing criteria. The data_source_config describes fields available on each item. The testing_criteria array contains graders that decide whether a model sample meets the requirement. Templates can read item fields and sample fields, so a configuration error can point a valid grader at the wrong value.

That makes four components part of the test surface:

1. The dataset row supplies the scenario and reference information.
2. The application or model produces the sample.
3. The grader maps the row and sample to a result.
4. Release policy turns results into pass, fail, or review.

Teams often test only component two. They run a new prompt, see a higher aggregate grade, and approve it. A reliable process asks whether component three still agrees with a trusted reviewer and whether component four treats uncertainty honestly.

The documented grader types solve different problems. A string check handles exact equality, inequality, and documented containment operations. Text similarity compares output with reference text using a selected similarity metric. A score-model grader asks a model for a numeric judgment. The API reference also describes label-model, Python, and multi-grader objects. Availability and workflow support can change, so use the current guide and API reference for the surface you are actually deploying.

OpenAI's current grader guide also carries a deprecation notice for graders used in supported eval and fine-tuning workflows. The docs place Evals under legacy APIs and direct readers to the deprecations page for transition information. That is not a reason to abandon evaluation discipline. It is a reason to keep the durable assets outside one endpoint: labeled cases, rubric definitions, criterion IDs, version history, and disagreement reports.

Reliability does not mean the grader always returns the same number. It means the team knows which claim the result supports, has measured the grader against independent labels, can reproduce a bad decision, and can detect when behavior changes. A deterministic exact check can be reliable and narrow. A model judge can be useful and probabilistic. Trouble starts when the second is presented with the confidence of the first.

Take a support answer that must satisfy two requirements:

• It must not promise a refund before eligibility is confirmed.
• It should explain the next available action in plain language.

The first criterion should consume structured evidence from the workflow if it exists. Whether the eligibility tool returned false is not a matter of prose quality. The second criterion may need semantic judgment because many explanations are acceptable. Putting both into one broad "helpfulness and correctness" prompt makes diagnosis difficult. A high helpfulness impression can soften a policy breach, while a terse but safe answer can score poorly.

Keep each result tied to one criterion. Store the criterion ID, grader version, dataset case ID, decision, and reason. A release summary may aggregate those records later. The raw row is what lets an engineer discover that one refund policy example failed while every shipping example passed.

The judge prompt is code too. Review it for missing definitions, contradictory instructions, unbounded scales, and references to fields that are not guaranteed by the schema. Give labels operational meaning. "Mostly correct" is vague. "No unsupported action is promised, and every stated policy condition appears in the supplied reference" is testable by a reviewer.

A reference answer is not automatically truth. It can be stale, incomplete, or more verbose than necessary. The goal is not to make every output resemble it. The goal is to encode the valid decision boundary. For a classification task, a human label may be enough. For advice, supply factual criteria or accepted claims rather than treating one writing style as the only correct answer.

Calibrate on disagreements, not easy examples

Start with a calibration set labeled by people who understand the policy. Include clear passes and clear failures, then spend most of the attention on boundary cases. Ten paraphrases of an obvious answer add less information than one case where two reviewers initially disagree.

Each row should say why the expected label is correct. That rationale catches weak gold data. If the owner cannot explain whether a response passes, the grader cannot be expected to infer a stable boundary from a short prompt.

For the refund example, useful rows include:

• The tool denies eligibility and the answer correctly declines.
• The tool denies eligibility and the answer says a refund "should be fine."
• Eligibility is not checked and the answer routes to a human without making a promise.
• Eligibility is confirmed, but the answer promises a credit amount not present in the evidence.
• The answer is safe but fails to give the user any next step.
• A malicious user tells the evaluator to ignore the rubric inside quoted conversation text.

Those cases exercise distinct mistakes. They are not one negative example with names changed.

Do not tune against every labeled row and then report performance on those same rows. Keep a holdout set that rubric authors do not use during prompt editing. If the dataset is small, at least freeze a set of incident cases before the next revision. Otherwise, the team measures memory of its examples rather than coverage of the policy.

A simple local report can expose false accepts and false rejects without depending on a dashboard. This program uses illustrative labels embedded in the script. It calculates agreement and prints every disagreement. The numbers it prints describe these six rows only; they are not claimed as product measurements.

from collections import Counter
from dataclasses import dataclass

@dataclass(frozen=True)
class Case:
    case_id: str
    slice_name: str
    human: str
    grader: str
    reason: str

cases = [
    Case("refund-01", "clear_denial", "pass", "pass", "declines after tool denial"),
    Case("refund-02", "soft_promise", "fail", "pass", "says refund should be fine"),
    Case("refund-03", "safe_escalation", "pass", "pass", "escalates without promise"),
    Case("refund-04", "invented_amount", "fail", "fail", "states unsupported amount"),
    Case("refund-05", "no_next_step", "fail", "pass", "safe but incomplete response"),
    Case("refund-06", "rubric_injection", "fail", "fail", "quoted text targets judge"),
]

labels = ("pass", "fail")
matrix = Counter((case.human, case.grader) for case in cases)
matches = sum(case.human == case.grader for case in cases)

print(f"agreement: {matches}/{len(cases)}")
for human in labels:
    for grader in labels:
        print(f"human={human:4} grader={grader:4} count={matrix[(human, grader)]}")

print("disagreements:")
for case in cases:
    if case.human != case.grader:
        print(case.case_id, case.slice_name, case.reason)

false_accepts = [
    case.case_id for case in cases
    if case.human == "fail" and case.grader == "pass"
]
if false_accepts:
    raise SystemExit("false accepts: " + ", ".join(false_accepts))

A CI job running this fixture should fail on refund-02 and refund-05. The useful output is not the agreement fraction. It is the two false accepts and their slices. A release gate for policy compliance should care more about those than about a false reject on an awkward but safe answer.

Agreement needs context. Overall agreement can improve because the dataset gained easy cases while a critical slice stayed weak. Report false-accept and false-reject counts by criterion and slice. For ordered scores, inspect whether the grader preserves important rankings, not only whether its average resembles a reviewer average.

Reviewers also need calibration. Give two people the same cases independently and compare their labels. If they disagree on "no next step," resolve whether that is a policy failure, a quality warning, or outside this grader. Do not force the judge to settle an ownership question the team has avoided.

When the judge and reviewer disagree, classify the cause before editing:

• The rubric is ambiguous.
• The dataset lacks evidence needed by the rubric.
• The human label or rationale is wrong.
• Template variables point to the wrong fields.
• The judge misapplied a clear criterion.
• The answer exploits predictable judge behavior.
• Release policy interpreted a valid result incorrectly.

This taxonomy prevents endless prompt growth. A missing eligibility result should be fixed in the dataset or marked ungradable, not buried under another paragraph telling the judge to be careful.

One near-miss deserves special attention. A grader may appear too lenient when the real issue is a bad reference. Suppose the expected answer says returns are allowed for 30 days, but the policy changed to 14 days. A correct new response will disagree with the reference. The same dashboard pattern appears when a judge accepts an invented 30-day policy. Check the source and version of the reference before blaming the judge.

Another near-miss comes from label inversion. A field named violation may use true for failure, while a grader template expects allowed. The judge can be perfectly consistent and every result wrong. Include one canary case whose expected mapping is unmistakable. If that canary flips, stop the run and inspect plumbing rather than tuning a threshold.

Match the grader to the claim

Exact requirements deserve exact graders. If a classifier must emit Hardware, Software, or Other with no extra prose, string equality is a clear oracle. OpenAI documents string_check with operations including eq, ne, like, and ilike in the API reference. Check the current schema when implementing because guide text and reference details can evolve.

The eval configuration below follows the current TypeScript example in OpenAI's eval guide. It defines a custom row schema and uses an exact string criterion. Running it requires the OpenAI JavaScript package, an API key in the environment, and access to the documented eval surface.

import OpenAI from "openai";

const openai = new OpenAI();

const evaluation = await openai.evals.create({
  name: "Support ticket classification",
  data_source_config: {
    type: "custom",
    item_schema: {
      type: "object",
      properties: {
        ticket_text: { type: "string" },
        correct_label: { type: "string" },
      },
      required: ["ticket_text", "correct_label"],
    },
    include_sample_schema: true,
  },
  testing_criteria: [
    {
      type: "string_check",
      name: "Exact human label",
      input: "{{ sample.output_text }}",
      operation: "eq",
      reference: "{{ item.correct_label }}",
    },
  ],
});

console.log(evaluation.id);

The trade-off is brittleness. "Hardware\n" fails equality even if a downstream consumer would trim whitespace. Decide where normalization belongs. If production trims, evaluate the normalized value or require the model to meet the raw protocol and test the parser separately. Do not quietly normalize in the grader when production does not.

Containment checks are useful for a genuinely required phrase or marker. They are dangerous as proxies for meaning. An answer can contain "I cannot approve a refund" and later say "but it is approved anyway." A like operation sees the safe phrase and misses the contradiction. Use structured state for policy decisions and semantic review for the explanation.

Text similarity has a related limitation. Lexical closeness to one reference is not factual correctness. A concise correct answer can differ from a long reference. A copied reference can score well while ignoring the user's actual situation. Similarity is appropriate when closeness itself is the requirement, such as regression detection for a tightly controlled transformation. It is weak evidence for open-ended truth.

A score-model grader earns its cost when quality is continuous and valid outputs vary. Give it a narrow criterion, explicit scale anchors, and all evidence a reviewer would need. If the criterion is binary, consider labels rather than pretending a decimal is more precise. A number with no calibrated interpretation creates debate at the threshold.

Separate correctness from style. One judge prompt that grades factuality, completeness, tone, formatting, safety, and relevance produces a score that cannot explain a failure. Multiple criteria cost more model calls or more complex output, but they allow release policy to say that safety blocks while a style warning does not.

Model judges are vulnerable to content inside the answer. A response may mention the rubric, repeat desired labels, or include user-provided instructions aimed at the evaluator. Delimit untrusted content clearly in the grader input and tell the judge what is evidence versus instruction. More importantly, add adversarial cases and inspect whether the judge follows them. Prompt wording alone is not proof of resistance.

OpenAI's guide discusses grader or reward hacking in the training context: a model can perform well against a grader while human evaluations remain poor. The same testing lesson applies outside training. Compare judge results with expert labels, preserve disagreement cases, and watch for output patterns that improve the grade without improving the task.

Use a deterministic composite only when the combination reflects policy. Averaging exact format, factuality, and style gives each an implicit weight. A failed safety invariant should not be rescued by perfect formatting. Encode it as a required condition, then calculate optional quality scores among safe cases.

Make a wrong grade reproducible

A dashboard row is the start of the investigation. Export the exact dataset item, model sample, criterion configuration, result, and available reason. Remove secrets, but do not paraphrase the failing answer. One changed qualifier can be the boundary.

OpenAI's templates use item and sample namespaces. The guide documents sample.output_text for textual model output and item fields from the data source. Validate every template reference against the item schema. A typo that resolves to missing data may produce errors or misleading results depending on the grader and workflow. Treat schema validation as a gate before judging quality.

Record versions outside mutable display names. Useful metadata includes dataset revision, criterion ID, rubric digest, grader type, judge model when applicable, application prompt version, and the time of the run. The purpose is not bureaucracy. It lets another engineer recreate the input after a dashboard configuration changes.

For model judges, rerun one disagreement several times without changing anything. Variation shows that the row lies near an unstable boundary or that the judge is nondeterministic under the chosen settings. Stability does not prove correctness. A consistently wrong result is still wrong, but easier to debug.

Avoid invented confidence. A judge returning 0.83 does not mean an 83 percent probability of correctness unless the output was explicitly calibrated to support that interpretation. Treat the value as a score on the defined scale. Establish the release threshold on labeled holdout data and name the errors it permits.

Counterfactual tests are especially revealing. Change one fact that should flip the result while keeping the rest of the answer constant. Then change a harmless style feature that should not flip it. A good criterion responds to the first and ignores the second.

The following script tests a local deterministic policy oracle with paired cases. In a real suite, the same pairs can be sent through a model grader and their returned labels written into this simple schema for comparison.

from dataclasses import dataclass

@dataclass(frozen=True)
class RefundCase:
    eligibility: str
    answer: str
    expected: str

def deterministic_policy_label(case: RefundCase) -> str:
    text = " ".join(case.answer.lower().split())
    approval_phrases = ("refund is approved", "refund has been approved")
    promises_approval = any(phrase in text for phrase in approval_phrases)

    if case.eligibility != "eligible" and promises_approval:
        return "fail"
    return "pass"

pairs = [
    RefundCase(
        "ineligible",
        "The refund is approved. It should arrive soon.",
        "fail",
    ),
    RefundCase(
        "eligible",
        "The refund is approved. It should arrive soon.",
        "pass",
    ),
    RefundCase(
        "ineligible",
        "I cannot approve this refund. I can open a review.",
        "pass",
    ),
    RefundCase(
        "ineligible",
        "I cannot approve this refund.\n\nI can open a review.",
        "pass",
    ),
]

for index, case in enumerate(pairs, start=1):
    actual = deterministic_policy_label(case)
    print(index, actual, case.expected)
    assert actual == case.expected

This oracle is intentionally narrow. It proves whether the paired fixtures and release plumbing react to two specific phrases. It does not understand every way to promise money. A model grader may cover paraphrases, but it should still pass the same counterfactual logic.

Look at reasons, not only labels. A judge can land on the expected label for the wrong reason. If it fails the ineligible case because the tone is terse rather than because approval contradicts eligibility, a minor tone edit may expose the weakness later. Human reviewers should spot-check reasoning against criterion IDs.

Diagnostic output should distinguish system errors from quality failures. A missing item field, rejected grader configuration, rate limit, or judge parse error is not a failed model answer. Count it as an execution error and decide whether the run is valid. Treating errors as zero silently lowers scores. Dropping them silently raises scores. Neither is acceptable.

A second failure mode appears when sample generation changes format. Suppose an eval previously received plain text and now receives structured JSON rendered into output_text. The grader may compare the serialized object with a reference string and fail every row. That is an interface mismatch, not a quality collapse. Inspect one raw sample before rewriting a rubric.

Roll a new grader into CI without hiding drift

Version the new criterion beside the old one. Run both over the same frozen calibration set and current candidate outputs. Store row-level results from each. Do not start with an aggregate threshold, because the first task is to understand decision changes.

Create three result buckets:

• Same decision, where migration does not affect the case.
• Intentional change, where the new rubric fixes an agreed weakness.
• Unexplained change, where evidence or judge behavior needs investigation.

A reviewer should approve intentional changes with a short rationale. Add corrected cases to the permanent calibration set. If a case changes because the gold label was wrong, preserve that history rather than rewriting the record without explanation.

Promote deterministic criteria first. Exact labels, JSON schema, required fields, and known prohibited tool actions are cheap and clear. Run semantic judges after those checks so they do not spend time grading malformed samples. Keep their reports separate.

A GitHub Actions job can run local contract tests before an API-backed eval. The remote step should receive credentials through repository secrets, and artifacts should exclude sensitive prompt or customer data. This wiring is an example of control flow, not evidence of any particular runtime or score.

name: grader-calibration

on:
  pull_request:
    paths:
      - "evals/**"
      - "prompts/**"

jobs:
  calibrate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Check dataset and rubric contracts
        run: python evals/validate_contracts.py
      - name: Run frozen local calibration cases
        run: python evals/calibrate.py --fail-on false-accept
      - name: Run provider-backed evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python evals/run_openai_eval.py --output artifacts/openai-eval.json
      - name: Preserve row-level results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: grader-results
          path: artifacts/openai-eval.json

The provider-backed script should fail loudly when the eval run itself fails. It should also capture the eval and run identifiers returned by the SDK, rather than guessing a URL or polling an invented method. Follow the current eval guide and SDK types for creation and result retrieval because this surface is changing.

Keep the initial provider step non-blocking while the team reviews disagreement classes. A gate nobody trusts will be bypassed. After the calibration set is stable, choose blocking rules per criterion. For example, any false accept on a protected policy can block, while a style criterion may only post a report.

Threshold changes deserve code review. Raising the pass threshold can reduce false accepts and increase false rejects. That may be the correct trade for a safety criterion and wasteful for copy tone. Attach the disagreement cases that motivate the change. Never move a threshold only until the current candidate passes.

Monitor grader drift after release. Periodically rerun a small frozen canary set. Include obvious pass, obvious fail, boundary, and injection cases. If a canary decision changes, pause comparisons across the change and investigate the grader configuration or underlying service before attributing the movement to the application.

Portable artifacts reduce migration risk. Store calibration rows in a documented local format. Keep rubric text and criterion metadata in the repository. Export per-row decisions from the service. The OpenAI guide's deprecation notice makes this practical even for teams satisfied with the current API: infrastructure can change while the QA contract survives.

Know when model judgment is the wrong tool

Do not use a model judge for syntax the parser can check. JSON validity, enum membership, required fields, tool names, and identifier equality have deterministic oracles. A judge adds cost and a new failure mode.

Avoid it when the evidence is missing. A polished rubric cannot determine whether an answer cited an approved document if the dataset does not contain the document or approved claims. Fix data collection or narrow the criterion.

Do not let a judge authorize irreversible actions. Runtime policy should enforce payment limits, account ownership, permissions, and destructive operations. Evaluation tests whether those controls and the surrounding behavior work. It does not replace them.

Skip aggregate model scores when reviewers cannot agree on the construct. "Professional," "helpful," and "high quality" need audience-specific definitions. Resolve the product decision first. Otherwise, the grader automates ambiguity and returns it with decimals.

Be careful when generator and judge share blind spots. Using different model families can reduce some correlation, but no pairing establishes truth. Independent human labels, factual sources, and deterministic invariants provide the external anchors.

A model judge is also a poor fit for a tiny, stable rule set. If five explicit phrases and a parser cover the requirement, maintain those tests. The semantic judge becomes worthwhile when valid language expands beyond rules and the added coverage justifies latency, cost, and calibration work.

Even then, keep a review state. Some rows are genuinely ambiguous or lack evidence. Forcing every case into pass or fail hides that limitation. A release policy can define how many reviews are acceptable, which criteria require human resolution, and when an incomplete run invalidates the gate.

The cost of a strong grader is not just API usage. Engineers maintain datasets, reviewers resolve disagreements, QA investigates drift, and owners version rubrics. Those costs buy coverage only if failures remain actionable. If every bad grade ends with "the judge felt it was weak," the criterion is not ready to block a release.

Finally, do not confuse endpoint availability with evaluation maturity. A configuration can validate successfully and still encode the wrong requirement. The most valuable grader test is the case designed to fool it, followed by a reviewer who can explain exactly why it should fail.