Why an exact-match grader rejects the answer you meant

The answer in the report looks right, but the grader gives it a zero because the model added a newline. Another row passes even though it contains a paragraph of unsupported explanation after the expected word. Both failures come from treating a string operation as if it understood the product requirement.

A string checker is useful precisely because it is simple. It compares text and returns a binary score. The operational work is deciding which textual differences matter, proving the negative cases, and keeping enough evidence to tell a model regression from a broken grader.

Know exactly what the four operations compare

OpenAI's current grader schema defines a string checker with five fields: type, name, operation, input, and reference. The type is string_check. The operation enum in the API reference is eq, ne, like, or ilike. The input and reference may contain templates, so the strings compared at runtime can come from the model sample, the dataset item, or a mixture of literal text and substituted values.

The important boundary is what the checker does not know. It does not know that "NYC" and "New York City" may refer to the same place. It does not know that two JSON objects with different property order can represent the same data. It does not decide whether a disclaimer is harmless, whether an answer is sufficiently complete, or whether a tool argument is safe. Those are separate requirements and need separate oracles.

Use eq when the exact representation is the contract. A routing model that must emit one token from a fixed vocabulary is a good example. If the allowed result is APPROVE, then approve, APPROVE\n, and I choose APPROVE are different outputs. A zero is useful because each difference violates the interface consumed by the next component.

Use like when the reference only needs to appear inside the input and case still matters. That can fit a generated notice that must contain an exact legal phrase while allowing other text around it. It is a poor fit when extra text could reverse the meaning. "The request is APPROVED" contains APPROVED. So does "The request is not APPROVED." A containment score cannot distinguish those statements.

Use ilike when containment is enough and capitalization is not part of the contract. This is often practical for a required heading or product name in prose. Case-insensitive containment widens the pass set in two ways: it accepts extra surrounding text and ignores case. That convenience costs defect-detection power. A team that switches from eq to ilike to make a flaky chart green may silently approve answers that no downstream parser accepts.

Use ne only for whole-string inequality. It is not a "must not contain" operator. If the reference is secret, a model output of the secret is 123 is not equal to the reference and therefore satisfies an inequality check, even though it contains the forbidden word. Absence, redaction, and policy checks need an oracle that inspects the actual forbidden condition.

There is a documentation trap worth recording in your repository. The current API reference and schema list ne. A prose bullet in the grader guide uses neq while the nearby schema uses ne. Treat the API reference as the payload contract and call the validate endpoint during deployment. Do not let a screenshot from an older dashboard or a copied blog post settle an enum question that the service can answer directly.

Templates add a second boundary. {{ sample.output_text }} refers to the model's text output. Values such as {{ item.reference_answer }} come from the dataset row. The model sample and item are not interchangeable. If both sides resolve from the item, the grader can award a perfect score without reading the model output. If both sides resolve from the sample, it can compare the answer with itself. The JSON looks legitimate in code review, which is why fixture mutation matters more than visual inspection.

A useful review question is simple: what model-output change makes this grader fail? Replace the output with an empty string, an unrelated label, the right label in the wrong case, and the right label with extra text. If no mutation changes the reward, the oracle is disconnected. If every mutation changes the reward, the checker may be too strict for the product contract. The desired boundary sits between those extremes and must be written down case by case.

Build fixtures that make the oracle prove itself

Start with the consumer of the answer, not the wording a model happened to produce last Tuesday. Suppose an application routes support requests using exactly one of three uppercase labels. The next service performs a dictionary lookup and rejects anything else. Exact matching is earned here because representation is behavior.

The first fixture set needs more than one positive row. Include every allowed label so a typo in one reference cannot hide behind another. Negative rows should each violate one property: case, surrounding whitespace, an unsupported label, and explanatory prose. Give each row a reason. When the expected result changes later, a reviewer can see whether the interface changed or someone merely relaxed a failing test.

The following local test defines the application's contract. It is not presented as a reimplementation of OpenAI's hosted grader. Its job is to catch a wrong fixture, a swapped expected value, or an accidental decision to normalize text before the remote check runs.

from dataclasses import dataclass

ALLOWED = {"BILLING", "CANCEL", "TECHNICAL"}

@dataclass(frozen=True)
class Case:
    output: str
    expected_reward: float
    reason: str

CASES = [
    Case("BILLING", 1.0, "allowed routing token"),
    Case("CANCEL", 1.0, "allowed routing token"),
    Case("TECHNICAL", 1.0, "allowed routing token"),
    Case("billing", 0.0, "case is part of the parser contract"),
    Case(" BILLING", 0.0, "leading whitespace breaks the parser"),
    Case("BILLING\n", 0.0, "trailing newline breaks the parser"),
    Case("Route to BILLING", 0.0, "prose is not a routing token"),
    Case("SALES", 0.0, "unsupported routing token"),
]

def application_reward(output: str) -> float:
    return 1.0 if output in ALLOWED else 0.0

for case in CASES:
    actual = application_reward(case.output)
    assert actual == case.expected_reward, (
        f"{case.reason}: {case.output!r} produced {actual}, "
        f"expected {case.expected_reward}"
    )

mutated = [Case("UNKNOWN", 1.0, "intentional bad expectation"), *CASES[1:]]
try:
    for case in mutated:
        assert application_reward(case.output) == case.expected_reward
except AssertionError:
    pass
else:
    raise AssertionError("The contract test did not detect a bad expectation")

The last mutation is deliberate. It demonstrates that the test can fail when an expected value is corrupted. That is a stronger proof than a green loop over hard-coded examples whose assertions simply repeat the fixture.

Next, construct one hosted grader and validate its shape. Put the model sample on one side and the dataset value on the other. Keep the name stable enough to appear in logs, but version the file containing the full JSON. Validation checks whether the service accepts the grader definition. It does not prove that the definition represents your business rule.

set -euo pipefail

: "${OPENAI_API_KEY:?OPENAI_API_KEY is required}"

body="$(mktemp)"
trap 'rm -f "$body"' EXIT

if ! status="$(curl --silent --show-error \
  --output "$body" \
  --write-out '%{http_code}' \
  https://api.openai.com/v1/fine_tuning/alpha/graders/validate \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{
    "grader": {
      "type": "string_check",
      "name": "support_route_exact_v1",
      "operation": "eq",
      "input": "{{sample.output_text}}",
      "reference": "{{item.expected_route}}"
    }
  }')"; then
  echo "grader validation request did not complete" >&2
  exit 1
fi

python3 -m json.tool < "$body" || cat "$body"

if [ "$status" -lt 200 ] || [ "$status" -ge 300 ]; then
  echo "grader validation rejected the definition: HTTP $status" >&2
  exit 1
fi

The status capture is the point of that script, not decoration. The obvious shorter version, curl --fail-with-body ... | python -m json.tool under set -eu, is a gate that cannot fail. A shell pipeline reports the exit status of its last command unless pipefail is set, so the script's status is the pretty-printer's, never curl's. The API's rejection body is itself well-formed JSON, so json.tool parses it happily and exits 0. Run that version against an endpoint returning 401 and you can watch curl print curl: (22) The requested URL returned error: 401, watch the error object print underneath it, and watch the script exit 0 anyway. A grader definition the service refused would sail through CI, and the refusal would be visible only to whoever reads the log. Writing the body to a file and testing %{http_code} removes the pipeline from the decision entirely. Adding pipefail to the set line fixes the narrower version of the same bug and costs nothing, so do both. Use python3 rather than python, because many stock CI images and current distributions ship no python on the PATH at all.

Check the corrected script the same way you would check any other oracle, by making it fail on purpose. Point it at a deliberately bad key and confirm it exits non-zero and names the status. Point it at a valid definition and confirm it exits 0. Stop the endpoint entirely and confirm the transport branch fires, since a connection refusal is a different failure from a rejected payload and should not be reported as a valid grader.

A 2xx response tells you the payload is valid at that moment. It does not tell you whether expected_route exists on every dataset row. It does not tell you whether the prompt promises lowercase labels while the grader expects uppercase. It does not tell you whether a later preprocessing step trims the output before production consumes it. Schema validation and behavioral validation belong in the same pipeline, but they answer different questions.

Run the grader against a small truth table after validation. The run endpoint accepts the grader, a model_sample string, and an optional item object. Its response includes a numeric reward and metadata. Assert the reward rather than checking only that the request returned HTTP 200.

import json
import os
import urllib.error
import urllib.request

URL = "https://api.openai.com/v1/fine_tuning/alpha/graders/run"
GRADER = {
    "type": "string_check",
    "name": "support_route_exact_v1",
    "operation": "eq",
    "input": "{{sample.output_text}}",
    "reference": "{{item.expected_route}}",
}
CASES = [
    ("BILLING", "BILLING", 1.0),
    ("billing", "BILLING", 0.0),
    ("BILLING\n", "BILLING", 0.0),
    ("Route to BILLING", "BILLING", 0.0),
    ("CANCEL", "BILLING", 0.0),
]

def run_grader(model_sample: str, expected_route: str) -> dict:
    payload = json.dumps({
        "grader": GRADER,
        "model_sample": model_sample,
        "item": {"expected_route": expected_route},
    }).encode("utf-8")
    request = urllib.request.Request(
        URL,
        data=payload,
        method="POST",
        headers={
            "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
            "Content-Type": "application/json",
        },
    )
    try:
        with urllib.request.urlopen(request, timeout=30) as response:
            return json.load(response)
    except urllib.error.HTTPError as error:
        detail = error.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"grader request failed: {error.code} {detail}") from error

for model_sample, expected_route, expected_reward in CASES:
    result = run_grader(model_sample, expected_route)
    reward = float(result["reward"])
    assert reward == expected_reward, {
        "model_sample": repr(model_sample),
        "expected_route": expected_route,
        "expected_reward": expected_reward,
        "actual_reward": reward,
        "metadata": result.get("metadata"),
    }

This test costs network time and depends on API availability, so it should not replace the fast local contract. It proves a different thing: the payload accepted by the current service produces the boundary your application expects for representative positive and negative strings.

A second worked example deserves a different matcher. Imagine a disclosure paragraph that may vary, but must contain the exact phrase "may produce inaccurate results." like can check that phrase while preserving case. Add adversarial negatives such as "It is false that this may produce inaccurate results" and "The statement that this may produce inaccurate results is incorrect." Both contain the required exact substring while reversing its meaning, so like accepts them even though a meaning-based policy should not. If contradictory surrounding language matters, a lone string checker is insufficient because it detects the substring but not the contradiction.

A third example is a human-facing category name where capitalization does not matter and extra prose is allowed. ilike can be appropriate for the presence of "account recovery." Test "Account Recovery," "ACCOUNT RECOVERY," and a sentence containing the phrase as positives. Test "password reset" as a negative. Then ask whether those phrases are actually interchangeable in the product. If they are, a substring grader still under-rewards a valid synonym. If they are not, widening the reference list may be safer than pretending the checker understands concepts.

Read the failure before changing the matcher

The most common operational mistake is to see a zero reward and immediately loosen eq to ilike. That move changes two dimensions at once. It ignores case and accepts surrounding text. Without the original strings, nobody can tell which difference caused the failure or whether the relaxed rule now passes something dangerous.

Record the model sample using a representation that exposes invisible characters. In Python, repr(output) will show \n, \t, and leading spaces. In JavaScript, JSON.stringify(output) serves the same purpose. Do not log raw customer content merely to diagnose a grader. Use test fixtures in CI, and redact or hash production samples according to the system's data policy.

Capture five facts for every failed fixture: the fixture identifier, grader configuration version, literal model sample, literal resolved reference, and returned reward. For a remote execution failure, also retain the request identifier header when available and the response metadata. Those facts separate four cases that a dashboard bar collapses into "failed."

A reward of zero with a successful request usually means the grader ran and the comparison did not pass. A rejected validation request means the configuration shape or a field value is unacceptable. An HTTP authentication or authorization error means the grader was never evaluated. A missing dataset field can surface as a template or variable problem rather than a model-quality problem. Do not count all four as incorrect model answers.

Inspect the actual resolved values before blaming templates. Suppose the grader input is {{ item.expected_route }} and the reference is also {{ item.expected_route }}. Every valid item can pass regardless of the model sample. The inverse mistake puts {{ sample.output_text }} on both sides. Both configurations look symmetrical and tidy. Mutation exposes them: replace the model sample with WRONG while keeping the item constant. A connected exact-match grader must return zero.

Another subtle configuration error reverses the operands of a containment check. The documented meaning of like is that the input contains the reference. If the long generated paragraph is placed in reference and the short required phrase is placed in input, the relationship is backwards. Exact matching hides this distinction because equality is symmetric. Containment is not. Name fixture fields generated_text and required_phrase instead of generic a and b so review catches the direction.

Whitespace failures need an ownership decision. Trimming in the grader only can make evaluation disagree with production. Trimming in the application can be correct if the public interface declares surrounding whitespace insignificant. Requiring the model to emit a clean token can be correct when downstream code consumes raw output. The right fix lives at the boundary that owns normalization, not wherever it makes the score rise.

Case failures follow the same rule. A UI heading may be case-insensitive for evaluation. An enum used by a case-sensitive parser is not. Before selecting ilike, run the candidate strings through the real consumer or a faithful contract test of that consumer. The grader should measure the interface users and services experience.

The report also needs denominator discipline. If validation failed and no comparison occurred, do not put that row in the model-accuracy denominator as a zero. Mark it as evaluation infrastructure failure and fail the evaluation job itself. Conversely, a real zero reward should not disappear because a retry later produced a one. Preserve attempts separately when the generation is stochastic. String comparison is deterministic for fixed resolved strings, but the model sample may not be.

Separate look-alike failures before you fix them

Several failures produce the same red row while requiring opposite fixes. The first near-miss is canonical JSON. A model emits {"status":"ok","count":2} while the reference is {"count":2,"status":"ok"}. An exact string check returns zero even though a JSON parser sees equivalent objects. Changing to ilike does nothing useful because property order still differs. Changing to like can reward an incomplete fragment. The correct oracle parses JSON and asserts required keys, types, values, and any policy on extra fields.

The opposite JSON problem is a system that signs or hashes the exact serialized representation. There, property order and whitespace may genuinely be part of the contract. A semantic object comparison would hide a production failure. Keep eq and generate the reference with the same canonicalization specification used by the signer. This is why "JSON should be compared structurally" is not a universal rule.

A second near-miss appears when a prompt changed but the grader did not. The old prompt required YES or NO. The new prompt asks for a one-sentence explanation followed by the decision. A sudden collapse in exact-match rewards is expected because the interface changed. If production still needs a bare token, the prompt change is the defect. If the product intentionally adopted explanatory output, the grader and downstream parser need a coordinated migration. Loosening the matcher without updating the consumer makes the eval pass while production continues to reject answers.

A third near-miss is data contamination. A CSV import leaves a carriage return on some references. Model samples are clean, so only rows created on one operating system fail. The evidence is a reference ending in \r across failures, not a distribution of model wording changes. Fix the ingestion path, re-export the dataset, and preserve one contaminated row as a parser regression test. Do not teach the model to reproduce a file-format artifact.

A fourth case is a missing value turned into text by an upstream serializer. The reference becomes None, null, or undefined rather than raising a data-quality error. A model can occasionally emit the same token and receive credit. Add a dataset preflight that rejects absent or non-string references before any remote run. This check belongs outside the grader because a grader should not be asked to decide whether its own test data is valid.

The ne operation creates its own family of false confidence. Teams sometimes use it to assert that an answer is not empty by setting the reference to an empty string. That only proves whole-string inequality. Whitespace, a refusal, or arbitrary garbage can pass. A meaningful completeness check must specify the allowed shape or required content. Likewise, comparing against one forbidden phrase does not exclude variants, containment, casing changes, or encoded forms.

Another look-alike is evaluator drift. The grader JSON in source control says eq, but the dashboard run used an older saved configuration with ilike. The model output has not changed, yet local reproduction and hosted history disagree. Record a digest of the serialized grader configuration with each result. When a run is created elsewhere, export the effective definition and compare the digest before investigating model behavior.

Finally, do not confuse a correct string score with a correct task. A customer-support answer can contain the required refund phrase and still quote the wrong amount. A tool call can have the correct function name and unsafe arguments. A response can emit the expected route after leaking sensitive context in earlier text. Give each independent requirement its own assertion, then define how those assertions affect release. One easy binary check should never stand in for the rest of the product contract.

Roll out changes without laundering the baseline

Changing a grader changes historical meaning. If yesterday's eq score and today's ilike score appear on one trend line, the chart implies comparability that no longer exists. Version the grader as data, store its complete JSON, and attach that version to each result. A human-readable reason for the change belongs beside the version, not only in a pull request that the operations team may never see.

Build a frozen calibration set before modifying the operation. Include ordinary positives, ordinary negatives, prior production incidents, formatting variants, refusals, and adversarial strings that exploit containment. Have a reviewer label the product outcome without seeing the old grader reward. Run both grader versions against the same stored model outputs. The comparison reveals which cases move and why, without spending model calls or introducing sampling variation.

Classify every changed row. "Intended newly accepted" covers capitalization differences the consumer ignores. "Intended newly rejected" covers a fixed loophole. "Unexpected" means the proposed grader is not ready. "Disputed requirement" means the product contract needs an owner. Do not average these categories into a single improvement number. A small set of newly accepted unsafe outputs matters more than a large set of harmless formatting recoveries.

For pull requests, keep local contract tests and dataset preflight mandatory. Run hosted validation when the grader JSON changes. Run the remote truth table on a protected branch, scheduled job, or release candidate so transient network issues do not block every unrelated edit. The exact cadence depends on your release process, but the distinction should be visible in CI.

name: string-grader-contract

on:
  pull_request:
    paths:
      - "graders/**"
      - "tests/grader_cases/**"
  workflow_dispatch:

jobs:
  local-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python tests/test_support_route_contract.py

  hosted-contract:
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python tests/run_openai_string_grader_cases.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

This wiring makes the cost explicit. The pull-request job is fast, deterministic, and credential-free. The manually dispatched job proves compatibility with the hosted endpoint. Its trade-off is delayed feedback unless the release checklist requires it. Making the remote job mandatory on every commit gives earlier signal but adds secret exposure, network latency, rate-limit risk, and an external dependency to routine development.

Do not overwrite the first failed attempt with a retry. For a fixed string and fixed grader, a changed reward indicates changed inputs, changed configuration, or service behavior worth investigating. Store each request as its own record. If the model is regenerated between attempts, say so and retain each model sample. "Passed after retry" describes generation reliability, not grader reliability.

The 2026 platform transition makes export part of normal operations. OpenAI's deprecation page says the Evals platform was deprecated on June 3, 2026, existing evals are scheduled to become read-only on October 31, and the dashboard and API are scheduled to shut down on November 30. Graders used in eval workflows are included in that transition. Fine-tuning has separate availability dates, so confirm which workflow owns each grader instead of assuming one deadline covers both.

Preserve the portable assets now: grader definition, dataset schema, labeled rows, raw outputs, expected per-row rewards, aggregation rule, release threshold, and evidence format. Recreate the deterministic string operation in the destination harness and run both systems on the same inputs while both remain available. A matching aggregate is not enough. Compare row by row so offsetting errors cannot hide a migration defect.

Migration costs engineering time and temporarily duplicates execution. It also pays down a dependency that already has a published end date. Freeze new dashboard-only logic. Put new grader changes in source control first, and make the hosted system one executor of that versioned contract rather than its only home.

Know when a string checker is the wrong oracle

Do not use a string checker for semantic equivalence. "Two" and "2" can be equivalent in one task and different in another. An evaluator needs domain-aware normalization, structured parsing, a carefully tested model grader, or human review. Substring matching cannot acquire that context through a longer reference string.

Do not use it to validate structured tool arguments. JSON text can differ in spacing, property order, numeric representation, and escaping while decoding to the same values. Parse the arguments and assert their schema and business constraints. Exact text remains appropriate only when the serialized form itself is the interface.

Do not use like or ilike as a safety policy. Required phrases can appear inside negation, quotation, user-supplied text, or an explanation of why the phrase is wrong. Forbidden content can be rephrased or encoded. Safety evaluation needs specific adversarial cases and an oracle built for the policy, often with deterministic checks plus expert review.

Do not normalize away evidence before you understand it. Trimming, lowercasing, punctuation removal, and Unicode normalization can reduce harmless false failures, but each transformation also merges distinct outputs. Apply only transformations the real consumer applies. Store the raw value alongside the normalized diagnostic so a later incident remains reconstructable.

Avoid a hosted string grader when a local assertion fully covers the requirement and no OpenAI workflow consumes the score. A five-line unit test is easier to debug, has no network dependency, and survives platform migration. The hosted grader earns its place when the evaluation or fine-tuning system needs that score in its native workflow, or when parity with an existing hosted baseline matters during transition.

Do not keep an exact checker merely because it is deterministic. Determinism is valuable only when the oracle matches the requirement. An always-wrong ruler gives repeatable measurements. If reviewers repeatedly override the same formatting failures, either the product contract is stricter than users need or the grader is measuring the wrong representation. Resolve that disagreement instead of adding an undocumented exception list.

Finally, do not interpret a one as proof of overall quality. It proves one configured textual relationship for one resolved pair of strings. Keep that claim narrow. A release decision can combine it with schema checks, factual checks, policy checks, latency evidence, and human judgment, but the string grader should remain the small, inspectable instrument it actually is.