OpenAI Evals Guide: Build Reliable LLM Evaluation Suites

An invoice extractor returned valid JSON on 99 test files, then interpreted a credit note as a positive payable amount. Schema validation passed, every field was present, and the response looked perfectly structured. The release question was not “does it return JSON?” It was “does the candidate preserve financial meaning across the documents we actually process?”

OpenAI’s evaluation guidance emphasizes task-specific tests, representative data, and explicit graders. The practical work is converting a product decision into examples and scoring rules that can distinguish a formatting success from a business failure.

Write the evaluation contract before choosing graders

Define the unit under test. It may be a prompt and model, a complete API-backed workflow, or one component such as document classification. For the invoice extractor, compare a candidate prompt against the production baseline with the OCR service and normalization code held constant.

Name the release-critical outcomes:

• Document type is classified correctly.
• Currency and decimal interpretation match the source.
• Credits have the correct sign and payable status.
• Line totals reconcile within the permitted rounding rule.
• Missing evidence produces null or a review flag, not an invented value.
• Output conforms to the application schema within latency and token budgets.

These outcomes imply different graders. Do not compress them into one generic “accuracy” score. Decide which are blockers, which can tolerate small variation, and which require human review.

Build examples with structured references

Use de-identified or synthetic documents that preserve layout and business complexity. Include normal invoices, credit notes, multi-page files, tax-inclusive pricing, locale-specific separators, handwritten corrections, low-quality scans, and unsupported formats.

An illustrative JSONL record can keep the source reference and expected semantics together:

{"id":"credit-note-de-004","input":{"documentRef":"fixture://credit-note-de-004.pdf"},"reference":{"documentType":"credit_note","currency":"EUR","payableAmount":-1840.5,"requiresReview":false},"metadata":{"locale":"de-DE","layout":"table","risk":"critical","origin":"production-defect"}}

Store only the fields needed to grade the case. When multiple outputs are valid, encode invariants and tolerances rather than a single reference string. Add metadata for slicing, including document type, locale, scan quality, page count, supplier format, risk, and case origin.

Split examples by purpose. Development examples help prompt authors iterate. A held-out test set supports release decisions. A small challenge set contains rare high-risk failures and should not become the only set the team optimizes against. Version the dataset and record label changes.

Protect the held-out set from informal prompt tuning. If engineers repeatedly inspect and optimize against every failed test case, the set becomes another development set and loses value as an independent release signal. Rotate a portion using newly reviewed production patterns, and report how much of the release suite was previously visible to the prompt author.

Layer deterministic and model-based graders

Deterministic graders should handle exact properties: JSON parsing, schema conformance, enum membership, sign, date normalization, arithmetic reconciliation, and numeric tolerance. They are fast, explainable, and stable enough for frequent CI.

from decimal import Decimal

def payable_amount_grade(output, reference):
    actual = Decimal(str(output["payableAmount"]))
    expected = Decimal(str(reference["payableAmount"]))
    return {
        "pass": abs(actual - expected) <= Decimal("0.01"),
        "actual": str(actual),
        "expected": str(expected),
    }

Use a model grader where semantic judgment is unavoidable, such as whether the review reason accurately describes ambiguous source evidence. Supply the relevant document text or extracted evidence, candidate output, reference criteria, and a narrow rubric. Ask the grader for a categorical result and rationale.

Do not ask a model grader to verify arithmetic that code can check. Do not let it infer ground truth from the same ambiguous image without calibration. Compare model grades with expert labels on a separate sample, track disagreement by slice, and keep human adjudication for critical cases.

Evaluate the production-shaped task

The eval task should call the same orchestration path the product uses, with controlled fixtures for external services. Return the user-visible output plus diagnostic fields such as parsed structure, retries, token usage, latency, and trace ID.

Keep generation settings and model identifiers explicit. Pin them during a baseline comparison. If you deliberately test model variability, repeat selected cases and report pass frequency rather than hiding multiple attempts behind the best output.

The following Python is architecture-level pseudocode and not a guaranteed OpenAI evaluation API signature:

def run_case(example, candidate):
    result = candidate.extract(example["input"]["documentRef"])
    return {
        "output": result.data,
        "latency_ms": result.latency_ms,
        "usage": result.usage,
        "trace_id": result.trace_id,
    }

def grade_case(run, example):
    return {
        "schema": schema_grade(run["output"]),
        "amount": payable_amount_grade(run["output"], example["reference"]),
        "reconciliation": reconcile_grade(run["output"]),
    }

Separate infrastructure failures from quality failures. A timeout, rate limit, fixture error, or invalid test file should not silently become a semantic zero. Report both because reliability affects release, but diagnose them differently.

Analyze paired changes and slices

Run baseline and candidate on the same versioned examples. For each grader, count fixed cases, regressions, unchanged passes, and unchanged failures. Paired outcomes reveal whether a higher mean comes from fixing many cases or trading one severe defect for several easy gains.

Slice results by document type, locale, scan quality, page count, supplier template, and risk. The invoice candidate may improve ordinary US invoices while breaking every credit note in a comma-decimal locale. That candidate should not ship even if global field accuracy rises.

Keep denominators visible and define minimum sample sizes. For small critical slices, review every example rather than relying on an unstable percentage. Use bootstrap intervals or repeated runs when model variation is material, but do not use statistics to soften a deterministic financial error.

Inspect correlations between graders. Schema pass with reconciliation fail signals semantically wrong structured output. A high semantic score with rising review flags may indicate the candidate is avoiding errors by abstaining too often.

Turn failures into evidence and new coverage

For each regression, capture the example ID, source fixture version, prompt and model revision, raw response, parsed result, grader details, retry history, and trace ID. Identify the first incorrect transformation: OCR, document classification, prompt interpretation, number normalization, schema parsing, or post-processing.

Minimize the case while preserving the defect. If a credit sign error depends only on a “Total credit” label, create a focused unit fixture for the parser and retain one end-to-end document example. This reduces feedback time without discarding realistic coverage.

Review label defects openly. If the source is genuinely ambiguous, mark the expected output as review-required rather than forcing a number. Grader false positives and false negatives belong in a calibration backlog. A failing eval can reveal a bad test just as easily as bad model behavior.

Sample production outputs using risk signals such as low OCR confidence, reconciliation failure, user corrections, repeated extraction, or high-value totals. After privacy review, promote confirmed new patterns into the dataset. This keeps the suite aligned with changing document traffic.

Design CI for signal and cost

Use a compact suite of deterministic, critical, and recently regressed cases on pull requests. Run a broader hosted or offline evaluation when prompts, models, schemas, OCR, or normalization change. Schedule periodic full-suite runs to detect provider or traffic drift.

Cache immutable fixture preparation, not model outputs that should be measured. Set concurrency and retry behavior deliberately. Report the cost of evaluation separately from product cost so teams can budget coverage. A model grader on every field of every file is usually less useful than deterministic field checks plus targeted semantic review.

Fail CI on clear blockers: new critical sign errors, invalid schema, unsupported currency invention, or a known regression returning. Use thresholds for probabilistic or low-risk measures, and require enough repeated evidence before treating a tiny change as real. Quarantine infrastructure-flaky cases with an owner and expiry rather than deleting them.

Make a release decision that can be audited

Define the gate in advance. For example: zero new critical document-type or sign errors, no regression on held-out reconciliation rate, review-flag rate within an approved range, and no more than a set increase in P95 latency or average tokens per processed page. Your actual thresholds must reflect financial exposure and service objectives.

Require a human reviewer to inspect all newly failed high-value cases and a sample of model-grader changes. Document waivers with case IDs, business impact, compensating controls, owner, and expiry.

The release report should include dataset version, coverage distribution, baseline and candidate configuration, grader definitions, paired outcomes, slice results, grader calibration status, operational failures, latency, usage, and unresolved defects. The invoice extractor is ready only when structured correctness, financial meaning, and review behavior meet the agreed contract. A polished aggregate score is not a substitute for that decision.