A Python grader can be wrong while every run stays green

A malformed answer receives full credit, and the dashboard stays green because the grader returns 1.0 from its exception handler. The model did fail, but the larger defect is the oracle that converts every surprise into success. Until the grader is tested as production code, its score is only an assertion about itself.

Custom Python is valuable when the requirement cannot be expressed as a plain string or similarity check. It can parse structured output, enforce cross-field rules, and award partial credit. That power also creates another program with inputs, branches, dependencies, and failure modes. QA owns those branches before trusting them to judge a model.

Follow the execution contract that actually exists

OpenAI documents the Python grader as a JSON object with type set to python, a source string, a name in the current API schema, and an optional image_tag. The source must define a function named grade that accepts exactly two arguments. The function returns a float. An exception, a non-float result, or another invalid result is marked invalid and receives a zero grade.

The first argument, sample, is a dictionary populated from the model output. Documented fields include output_text, output_json, output_tools, choices, and output_audio, although which values exist depends on the sampling setup. The second argument, item, contains evaluation context from the data source or fine-tuning row. The grader should treat sample data as untrusted candidate output and item data as a test fixture whose schema was checked before the run.

That distinction changes error handling. Invalid JSON in sample["output_text"] can be a legitimate model failure, so returning zero is reasonable when JSON is required. A missing expected_total_cents in item is not evidence that the model failed. It means the test row is corrupt. Letting that condition raise an exception gives the hosted response a chance to identify a Python grader runtime error instead of silently mixing fixture defects into the model-failure denominator.

The hosted runtime is constrained. OpenAI's current guide says uploaded code must be under 256 kB, execution has a two-minute limit, and the runtime has limits of 2 GB memory, 1 GB disk, and two CPU cores. It has no network access. Those are service limits, not a performance budget to consume. A grader that needs most of two minutes per row will make a large evaluation slow and expensive to operate even when it technically fits.

The guide lists packages for the dated 2025-05-08 image tag, including NumPy, SciPy, pandas, RapidFuzz, scikit-learn, jsonschema, Pydantic, and several others. That list supports a specific image, not every possible future runtime. If a grader imports a third-party package, pin an image tag that documents the dependency and validate it remotely. Standard-library code is easier to migrate and usually enough for parsing, arithmetic, sets, regular expressions, and basic structural checks.

No network access means the grader cannot call your production database, fetch a reference answer, or ask another service whether an identifier is valid. Put required facts in item before execution. This improves reproducibility, but it increases dataset responsibility. A stale reference embedded in the item will be applied consistently and still be wrong.

A float creates another design decision. A binary contract can return 0.0 or 1.0. Partial credit needs a rule that reviewers can explain per field. Avoid a smooth-looking formula that lets a severe violation disappear inside an average. If an answer has the right total but the wrong currency, deciding that it deserves half credit is a product policy, not a mathematical fact.

Finally, the function must be connected to the model output. A grader that reads only item can return the expected score for every candidate. A grader that uses a constant or compares a value with itself can never detect a regression. Review data flow from a sample field through parsing and decisions to the returned float. Then mutate that field and prove the score changes.

Build an oracle with a real failure surface

Consider an order-summary model that must return one JSON object. The object must contain exactly status, currency, and total_cents. Status must be approved. Currency and total must match the evaluation item. Extra keys are forbidden because the downstream service rejects unknown fields.

This is a good Python-grader case because semantic JSON structure matters while raw serialization does not. Property order and insignificant whitespace should not affect the score. Types do matter. Python treats True as an instance of int, so a careless isinstance(value, int) check can accept a boolean where cents are required. Using type(value) is int makes that boundary explicit.

The grader below catches expected candidate failures and returns zero. It raises for malformed fixtures. Each branch can be triggered by a change in sample or item data.

import json
from typing import Any

REQUIRED_KEYS = {"status", "currency", "total_cents"}

def grade(sample: dict[str, Any], item: dict[str, Any]) -> float:
    expected_currency = item.get("expected_currency")
    expected_total = item.get("expected_total_cents")

    if not isinstance(expected_currency, str) or not expected_currency:
        raise ValueError("item.expected_currency must be a non-empty string")
    if type(expected_total) is not int or expected_total < 0:
        raise ValueError("item.expected_total_cents must be a non-negative integer")

    output_text = sample.get("output_text")
    if not isinstance(output_text, str):
        return 0.0

    try:
        candidate = json.loads(output_text)
    except json.JSONDecodeError:
        return 0.0

    if type(candidate) is not dict:
        return 0.0
    if set(candidate) != REQUIRED_KEYS:
        return 0.0
    if candidate["status"] != "approved":
        return 0.0
    if candidate["currency"] != expected_currency:
        return 0.0
    if type(candidate["total_cents"]) is not int:
        return 0.0
    if candidate["total_cents"] != expected_total:
        return 0.0

    return 1.0

A change to the model can make this fail by emitting invalid JSON, the wrong top-level type, a missing key, an extra key, a different status, the wrong currency, a boolean total, or a different integer total. A change to the fixture can raise by removing or corrupting either expectation. That is a meaningful failure surface.

The strict key set has a cost. Adding a harmless order_id field will receive zero until the contract and grader move together. If production ignores unknown fields, rejecting them in the eval measures a stricter interface than users experience. Decide the extra-field policy from the real consumer. Do not choose strictness merely because set(candidate) == REQUIRED_KEYS is easy to write.

A second worked example may need partial credit. Suppose a product-description response must be JSON with a correct SKU, an allowed language, and three required bullet fields. The SKU may be a hard gate while each correctly populated bullet earns part of the remaining score. One defensible rule is to return zero for the wrong SKU, then allocate explicit weights to valid fields. Test every weight branch and confirm the maximum is exactly one. Also test that an unsafe or forbidden claim forces zero rather than merely subtracting a small amount.

Partial credit is not automatically better. It can help rank incremental improvements during optimization, but a release gate often still needs mandatory conditions. Store the sub-check results outside the single float in your local harness so a score of 0.8 does not hide whether the missing portion was formatting or a prohibited claim. The hosted Python function returns a float, so diagnostic detail must come from fixture-level reproduction and surrounding evaluation records.

A third example concerns ordered operations. A model outputs a JSON list of requested actions, and the policy requires verify_identity before issue_refund. Checking only that both strings appear rewards the reversed, unsafe sequence. Python can parse the list, verify element shape, locate the two actions, and compare their indices. The negative fixtures must include missing verification, reversed order, duplicate refund, and an unrelated action inserted between steps if adjacency matters.

Do not hard-code one passing list and assert that its known elements appear in the expected order. Mutate the output list and run the same function. The reversed case must fail because the indices change. The missing-verification case must fail before index lookup. The duplicate-refund case must fail because cardinality is part of the policy. These mutations make the oracle demonstrate the promised behavior.

Test the grader separately from the model

A model evaluation combines at least three changeable systems: generation, test data, and grading. Debugging all three from one aggregate score is slow. Run the grader locally against hand-labeled fixtures before sending any model output to the hosted service. The local suite should be deterministic and should not need an API key.

Use ordinary unit-test discipline. Name each case for the rule it exercises. Keep one clean positive and many focused negatives. Verify corrupt item data raises instead of returning an ordinary model score. Test exact floats if the function deliberately returns fixed values. For calculated partial credit, test boundaries and use a stated numerical tolerance only when floating-point arithmetic requires it.

import json
import unittest

from order_summary_grader import grade

def sample(value) -> dict:
    if isinstance(value, str):
        return {"output_text": value}
    return {"output_text": json.dumps(value, separators=(",", ":"))}

ITEM = {"expected_currency": "USD", "expected_total_cents": 2599}
ONE_CENT_ITEM = {"expected_currency": "USD", "expected_total_cents": 1}
GOOD = {"status": "approved", "currency": "USD", "total_cents": 2599}

class OrderSummaryGraderTests(unittest.TestCase):
    def test_accepts_equivalent_json_serialization(self):
        text = '{ "total_cents": 2599, "currency": "USD", "status": "approved" }'
        self.assertEqual(grade(sample(text), ITEM), 1.0)

    def test_rejects_malformed_json(self):
        self.assertEqual(grade(sample('{"status":'), ITEM), 0.0)

    def test_rejects_wrong_total(self):
        changed = {**GOOD, "total_cents": 2600}
        self.assertEqual(grade(sample(changed), ITEM), 0.0)

    def test_rejects_boolean_total(self):
        changed = {**GOOD, "total_cents": True}
        self.assertEqual(grade(sample(changed), ITEM), 0.0)

    def test_rejects_boolean_total_equal_to_expected(self):
        changed = {**GOOD, "total_cents": True}
        self.assertEqual(grade(sample(changed), ONE_CENT_ITEM), 0.0)

    def test_rejects_extra_field(self):
        changed = {**GOOD, "note": "looks fine"}
        self.assertEqual(grade(sample(changed), ITEM), 0.0)

    def test_rejects_non_object(self):
        self.assertEqual(grade(sample([GOOD]), ITEM), 0.0)

    def test_exposes_corrupt_fixture(self):
        with self.assertRaises(ValueError):
            grade(sample(GOOD), {"expected_currency": "USD"})

if __name__ == "__main__":
    unittest.main()

The positive test changes property order and whitespace. That proves the function measures decoded structure rather than raw text. Each negative changes a different property. The corrupt-fixture test protects the denominator.

The two boolean tests deserve a note, because only one of them actually covers the guard the section is about. test_rejects_boolean_total sends True against an item expecting 2599 cents. It does return zero, but it would return zero even if the type check were deleted, because True != 2599 and the value comparison two lines later catches it. That test therefore proves the grader rejects a boolean; it does not prove the grader rejects a boolean because it is a boolean. Replace type(candidate["total_cents"]) is not int with the careless not isinstance(candidate["total_cents"], int) and it stays green, along with every other case built on the 2599 fixture. The mutation is only observable where True == expected_total, which is why ONE_CENT_ITEM exists: at an expected total of 1 cent, True == 1 passes the value comparison, and the type check is the sole surviving barrier. Under the strict type(...) is not int form the grader returns 0.0 and the test passes; under isinstance it returns 1.0 and the test fails. That is the difference between a fixture that mentions a boolean and a fixture that tests one.

The same reasoning applies to expected_total itself. The fixture validator uses type(expected_total) is not int, so an item written as {"expected_total_cents": True} is rejected as corrupt rather than silently graded as one cent. Choose a negative fixture that only the intended branch can reject, then confirm by deleting that branch.

Add mutation checks around the code, not only more examples. Temporarily replace candidate["total_cents"] != expected_total with equality and confirm tests fail. Swap type(...) is not int for isinstance(..., int) and confirm the one-cent boolean case goes red. Remove the extra-key condition and confirm its test fails. Change the success return to zero and confirm the positive fails. A mutation tool can automate this, but manual mutations during review are enough to catch an oracle that cannot fail.

Test the source packaging path too. OpenAI's guide shows using inspect.getsource to load grader code rather than maintaining an escaped copy inside JSON. If CI validates one source string while production submits another, the tests prove the wrong artifact. Build the payload using the same module file the local tests import, and record a digest of that source with results.

Fixtures need provenance. A reference total should come from a controlled test case, not be recomputed by the same buggy function the model is supposed to replace. If expected and actual values share an implementation, one defect can make them agree. For historical incidents, preserve the input and the independently reviewed expected outcome. Do not paste production personal data into an eval merely because it is realistic.

Test score polarity. Some pipelines treat one as pass, others apply a threshold, and an internal report can accidentally sort ascending. Include one known positive and one known negative at the aggregation boundary. Assert the positive is accepted and the negative is rejected by the release code, not only by grade. A correct grader attached to an inverted gate is still a broken release control.

Test duplicate rows and weighting separately. If the dataset contains the same easy positive many times, an average can look healthy while hard cases fail. That is an aggregation defect, not a Python-function defect. Keep stable item identifiers, detect unintended duplicates in preprocessing, and report mandatory slices beside any overall score.

Diagnose zeros without hiding evaluator failures

A zero is ambiguous until you inspect metadata. It can be a deliberate result from grade, an exception raised by the function, an invalid return, or a broader request failure represented elsewhere. The run API documents reward plus metadata that includes error flags such as python_grader_runtime_error and python_grader_runtime_error_details. Preserve both.

Broad exception handling erases this distinction. Code such as except Exception: return 0.0 makes missing fixture fields, programmer typos, dependency import failures, and malformed model JSON look identical. Catch the specific parsing errors that are an expected candidate outcome. Let impossible fixture states and programming defects raise. The hosted reward may still be zero, but its error metadata tells operations that the evaluator failed.

The reverse pattern is also dangerous: raising on every invalid model answer. If malformed JSON is a known output failure, throwing creates noisy runtime errors and can make the eval job look unhealthy rather than accurately scoring the candidate. Treat expected model invalidity as data. Reserve exceptions for conditions that mean the grader could not apply its contract.

Validate the grader before running cases, then run a positive and negative canary. Validation proves the service accepts the object shape and source. It does not execute every branch or confirm reward polarity. A canary pair catches a constant return, a packaging mismatch, and several template mistakes.

The following script packages the locally tested module, validates it, runs two canaries, and fails if hosted rewards or metadata disagree with expectations. It uses only the Python standard library on the client. The grader itself also uses only the standard library.

import hashlib
import importlib
import inspect
import json
import os
import urllib.error
import urllib.request

grader_module = importlib.import_module("order_summary_grader")
source = inspect.getsource(grader_module)
grader = {
    "type": "python",
    "name": "order_summary_contract_v1",
    "source": source,
    "image_tag": "2025-05-08",
}
headers = {
    "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
    "Content-Type": "application/json",
}

def post(path: str, body: dict) -> dict:
    request = urllib.request.Request(
        f"https://api.openai.com/v1{path}",
        data=json.dumps(body).encode("utf-8"),
        method="POST",
        headers=headers,
    )
    try:
        with urllib.request.urlopen(request, timeout=45) as response:
            return json.load(response)
    except urllib.error.HTTPError as error:
        detail = error.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"{path} failed: {error.code} {detail}") from error

validation = post(
    "/fine_tuning/alpha/graders/validate",
    {"grader": grader},
)
print("validated source sha256:", hashlib.sha256(source.encode()).hexdigest())
print("validated grader:", validation.get("grader", validation))

item = {"expected_currency": "USD", "expected_total_cents": 2599}
canaries = [
    ('{"status":"approved","currency":"USD","total_cents":2599}', 1.0),
    ('{"status":"approved","currency":"USD","total_cents":2600}', 0.0),
]

for model_sample, expected in canaries:
    result = post(
        "/fine_tuning/alpha/graders/run",
        {"grader": grader, "model_sample": model_sample, "item": item},
    )
    errors = result.get("metadata", {}).get("errors", {})
    assert not errors.get("python_grader_runtime_error"), errors
    assert float(result["reward"]) == expected, result

The script prints returned structures rather than promising one frozen validation response shape. It asserts only fields documented for the run response and relevant to the test. If the API rejects the dated image tag or source, the HTTP body is retained in the raised error. That is an integration failure, not a model-quality zero.

Look at python_grader_runtime_error_details when the runtime flag is true, but do not expose secrets in logs. The runtime has no network access, so API credentials do not belong in grader source or item data. A stack trace can still include candidate text or fixture values if an exception message embeds them. Keep exception messages descriptive about field names and types without copying sensitive contents.

Timeouts and resource failures need their own classification. An algorithm that is quadratic in output length may pass short canaries and fail large samples. Add an upper-bound fixture near the maximum output your application permits. Measure locally for engineering feedback, but do not publish invented performance figures. In the release gate, a hosted resource failure means no trustworthy score was produced.

Dependency failures look similar. A local laptop may have a newer package than the hosted image. Validation or execution can fail even though unit tests pass. Pin the documented image tag, test the packaged source remotely, and avoid imports you do not need. If a third-party function supplies central grading behavior, add fixtures that pin its expected semantics rather than trusting the import alone.

A data-shape mismatch is another near-miss. A local test passes {"output_text": "..."}, but an eval configured for structured output expects the grader to read output_json. Or the grader reads output_json while a run supplies ordinary text. Pick one source deliberately and test the actual sampling configuration. Parsing output_text yourself is portable when JSON text is the contract, but it may differ from a system that populates a parsed structured-output field.

Roll out the grader and its migration together

Treat grader source like release code. Put it in its own module, add unit tests, review changes, and attach a source digest to evaluation evidence. Do not edit the dashboard copy while leaving the repository copy unchanged. The artifact that ran must be reconstructable from the result.

A practical CI split keeps fast checks close to development and hosted checks close to release. Run unit tests and fixture validation on every relevant pull request. Run OpenAI validation and canaries when the source changes on a protected branch or through a release workflow. This keeps ordinary work credential-free while still detecting drift before promotion.

name: python-grader-contract

on:
  pull_request:
    paths:
      - "graders/order_summary_grader.py"
      - "tests/test_order_summary_grader.py"
  workflow_dispatch:

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m unittest tests.test_order_summary_grader

  hosted-canary:
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python scripts/verify_hosted_order_grader.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The remote job adds network latency, secret handling, and dependence on a hosted alpha endpoint. Running it only on demand reduces those costs but creates a process requirement: a release cannot proceed unless someone triggers it and reviews the evidence. Automating it on merges to the protected branch gives earlier assurance at the cost of more calls and more frequent external failures. Choose explicitly and encode the rule.

Roll out scoring changes with shadow comparison. Freeze a labeled corpus and stored model samples. Run the old and new grader source on identical inputs. Review every changed per-item reward before considering aggregates. A new parser may correctly accept reordered JSON while accidentally accepting booleans as integers. Those changes can offset in the average and leave the headline unchanged.

Preserve the first failure. If a remote canary raises and a retry passes, store both attempts. A transient service issue and a deterministic grader defect have different owners, but deleting the first response prevents that diagnosis. Do not regenerate model output during grader canaries. Stored samples isolate the evaluator.

Version the item schema with the source. Adding expected_tax to the grader before all rows contain it creates evaluator errors. Adding it to rows without updating the grader can create unused, misleading data. A migration check should load every item, validate required names and types, and report unknown fields that may indicate a typo.

OpenAI's current deprecation notice changes the rollout horizon. The Evals platform was deprecated on June 3, 2026. Existing evals are scheduled to become read-only on October 31, and the dashboard and API are scheduled to shut down on November 30. Graders used by eval workflows are part of that transition. Fine-tuning availability follows separate dates, so inventory each Python grader by workflow before assigning a deadline.

Export source is not enough. Preserve the image tag, imported-package versions documented for that tag, item schema, calibration cases, raw model samples, expected rewards, aggregation logic, thresholds, and handling of evaluator errors. The destination runner must reproduce both the function and the decision made from its result.

Build a provider-neutral adapter around grade(sample, item) while the hosted workflow still runs. Feed the same dictionaries to the local function and the hosted run endpoint. Compare per-case rewards and error classification. Once parity is established, make the new harness authoritative and keep the old system as a temporary shadow. This sequence costs duplicate execution, but it avoids a deadline-driven cutover with no trustworthy baseline.

Do not carry hosted limitations into the new design by accident. Keeping the pure two-argument function is useful because it is easy to test. Keeping an arbitrary single float may not be. A destination harness can return structured diagnostics alongside the reward, distinguish fixture errors from candidate failures directly, and enforce timeouts at the process boundary. Preserve semantics that matter, not every constraint of the retiring executor.

Know when Python is the expensive wrong tool

Use a string checker for an exact label or required substring. Python adds source packaging, runtime failure modes, image compatibility, and migration work without improving a truly simple oracle. The smaller grader is easier to inspect and harder to reward-hack accidentally.

Use ordinary application tests when the score is not needed inside an OpenAI workflow. A local parser test can return rich diagnostics, run without credentials, and use your normal dependency management. Wrapping it in a hosted grader makes debugging slower. During migration, many custom graders can become plain functions in the evaluation harness.

Do not use a Python grader to execute the model's code as if it were a general-purpose security sandbox. The documented feature runs your grading source in a constrained hosted environment. Evaluating untrusted programs safely requires a purpose-built isolation model, explicit resource controls, filesystem rules, and adversarial testing beyond a grader callback.

Avoid Python when the requirement is primarily semantic and human experts disagree. More code cannot manufacture a stable reference. Define the rubric, collect adjudicated examples, and measure reviewer agreement before automating. A model-based grader or human review may fit, but both need their own calibration.

Do not fetch live truth during grading. The hosted runtime has no network access, and even a different runner with network access would make results depend on changing external state. Snapshot the facts needed for a case, record their provenance, and decide when to refresh them. A release result should be replayable later.

Do not hide policy severity inside partial credit. If one prohibited action must block release, return zero or expose a mandatory failure that the gate checks separately. An average that combines safe formatting wins with one serious violation is easy to optimize and hard to defend.

Avoid elaborate dependency stacks for operations available in the standard library. Every import ties the grader to an image and a package version. A dependency is justified when its tested algorithm materially improves the oracle, such as a validated schema library for a complex schema. Convenience alone is a weak reason in code that decides releases.

Finally, do not trust custom code because it looks rigorous. Branches, type checks, and formulas can create an impression of precision while reading the wrong field or rewarding a constant. Demand one known positive, several independent negatives, a corrupt-fixture case, a packaging check, and a hosted canary. The grader earns authority only after those tests prove it can fail for the right reasons.