Property-Based Evals for Structured LLM JSON Outputs

Valid JSON is only the outer shell of a structured LLM response. A payload can parse cleanly while assigning an impossible status, inventing an unsupported source, exposing a forbidden field, or producing totals that disagree with its line items. Property-based evaluation tests the contract across a generated input space and checks invariants that must hold for every accepted response.

The technique is strongest when generation is constrained by domain rules, deterministic checks do most of the judging, and every failure is stored with its seed and execution trace. It complements curated examples by exploring combinations humans may not think to write, while preserving a clear distinction between generated coverage and observed production behavior.

Define Properties at the Contract Boundary

Start with the consumer of the JSON. List what must be true before downstream code, a user, or another agent can safely act. Separate syntactic validity, schema conformance, domain invariants, evidence constraints, authorization rules, and semantic quality. Each property needs an owner and a failure severity.

The OpenAI Graders API reference is a useful implementation reference for evaluation grader resources. For structured output, design the oracle independently of any platform: exact parsers and business checks should remain executable locally, while subjective criteria can be delegated to a calibrated semantic grader.

Model the Input Domain Before Generating

An unconstrained random JSON generator mostly creates nonsense. Define entities, valid relationships, boundary values, mutually exclusive fields, optional context, and adversarial but plausible strings. Generate both valid requests and explicitly labeled invalid requests if the product has a rejection contract.

Capture provenance for every generated case: generator name and version, random seed, parent template if any, generation parameters, intended validity, and covered features. Mark synthetic cases distinctly from production-derived and expert-authored examples. A release report should never imply that generator frequency equals user frequency.

Bias generation toward meaningful boundaries such as empty collections, maximum allowed counts, duplicated identifiers, conflicting instructions, unusual Unicode at ingestion, long-but-valid text, and permission edges. Keep the committed article and diagram ASCII, but the test domain may legitimately include non-ASCII values when the product supports them.

Turn JSON Schema Into the First Oracle

Use a strict parser that rejects trailing commentary and multiple payloads. Validate required fields, types, enums, numeric ranges, formats, collection sizes, and whether additional properties are allowed. Apply the same schema at the producer-consumer boundary, not a weaker test-only approximation.

Schema checks should return structured failure codes. PARSE_ERROR, MISSING_FIELD, and UNEXPECTED_PROPERTY lead to different fixes and slice analysis. Preserve the raw response separately from the parsed value so lossy repair logic does not hide model behavior.

Do not silently coerce strings to numbers, invent defaults, or drop unknown fields inside the eval harness unless production performs the identical transformation. If production repairs responses, evaluate both raw contract adherence and post-repair usability.

Encode Cross-Field Business Invariants

Business properties relate several fields or compare output with input context. An order refund must not exceed captured payment. A cited document ID must come from the supplied retrieval set. A denied decision must not include an executable action. Percent allocations may need to sum to a defined total within an explicit rounding tolerance.

type Context = {
  allowedDocumentIds: Set<string>;
  capturedCents: number;
  canRefund: boolean;
};

type Decision = {
  action: "refund" | "deny";
  amountCents: number;
  evidenceIds: string[];
};

export function checkDecision(ctx: Context, value: Decision): string[] {
  const errors: string[] = [];
  if (value.action === "refund" && !ctx.canRefund) errors.push("UNAUTHORIZED_REFUND");
  if (value.amountCents < 0 || value.amountCents > ctx.capturedCents) {
    errors.push("INVALID_AMOUNT");
  }
  if (value.evidenceIds.some((id) => !ctx.allowedDocumentIds.has(id))) {
    errors.push("UNSUPPORTED_EVIDENCE");
  }
  if (value.action === "deny" && value.amountCents !== 0) {
    errors.push("DENIAL_WITH_AMOUNT");
  }
  return errors;
}

These checks are deterministic and explainable. The selected rules are illustrative; the real contract must come from product and domain requirements.

Generate Cases and Preserve Reproducibility

A property-testing library can generate input structures and shrink failing values. Wrap model invocation so every request and response is persisted before an assertion runs. Store generator seed, example path, prompt version, model configuration, retrieval snapshot, tool versions, and grader bundle.

Model calls may be nondeterministic even with a recorded seed. Define failure stability, such as observing the same invariant violation across a planned number of replays. The replay count and decision rule are local risk choices, not universal thresholds. For irreversible or severe actions, one confirmed violation may be enough to stop release; lower-risk formatting failures may require evidence of a regression rate.

Separate generator shrinking from response mutation. Shrink the input, invoke the system again, and cache that new output. Never edit the response to manufacture a smaller case. The final artifact should retain lineage to the original generated failure and all intermediate trials.

Shrink Without Losing the Failure's Meaning

Generic shrinking may remove the premise that makes a case valid. A minimized request with no account permission context might still fail an authorization assertion, but it no longer tests the intended scenario. Add domain-aware shrinkers that preserve required relationships and validity labels.

After automatic shrinking, run three checks: the minimized input still belongs to the declared domain, the failure reproduces under the recorded environment, and a reviewer can explain the violated property. If any check fails, keep the smallest stable meaningful ancestor instead.

A shrunk case should usually become a curated regression example. Mark its synthetic lineage and avoid placing its parent or siblings across a sealed holdout boundary.

Use a Layered Grader Stack

Run deterministic checks in an order that produces useful diagnostics: transport success, strict JSON parse, schema validation, security and authorization properties, domain invariants, then semantic criteria. A parse failure should not be converted into a misleading semantic score.

Some properties remain subjective. A category may be valid only if its rationale is supported by the supplied evidence, even when all identifiers are legal. Write a narrow model-grader rubric for that criterion, require evidence references, and allow abstention. Human annotators should calibrate on clear, boundary, and adversarial cases before the grader is trusted.

Measure model-grader agreement against adjudicated labels by failure type and slice. Freeze grader configuration for candidate comparison. If a new grader is introduced, bridge it against the old grader and human evidence rather than mixing scores silently.

Prevent Generator and Evaluation Leakage

Generated cases can leak when the same seeds, schemas, or minimized failures become prompt examples, fine-tuning records, or grader demonstrations. Maintain family identifiers across original, mutated, and shrunk examples. Split by family, not row, and record when engineers inspect a sealed failure.

There is also generator overfitting. A system can improve on one known generator while remaining fragile elsewhere. Rotate generation strategies, retain a sealed generator configuration or seed region where justified, and confirm failures against production and expert-designed cases. Do not hide the property definitions themselves; systems should satisfy real contracts, not rely on obscurity.

Analyze Coverage, Slices, and Uncertainty

Measure generator coverage in domain terms: enum values, boundary classes, optional-field combinations, permission states, evidence counts, and known incident patterns. Raw case count is weak evidence if thousands of inputs exercise the same easy path.

Report failure rates by property and slices such as locale, payload size, product route, action severity, and retrieval condition. Track parse failures separately from business-invariant failures and semantic-grader abstentions. Use family-aware or cluster-aware intervals when generated cases share templates.

Generated distributions are designed, not natural. Report unweighted property discovery and any policy-weighted release score separately. If production prevalence is used for weighting, document its source window and uncertainty.

Set Release Gates by Consequence

Map each property to severity and decision behavior. Structural failures might block an integration release because downstream code cannot consume the response. Authorization or unsupported-evidence violations may be critical vetoes. Low-risk optional formatting differences might create a warning and backlog item.

An illustrative gate can require zero confirmed critical violations in the executed set and no statistically supported regression on high-priority noncritical properties. These are examples of policy shape, not generally valid thresholds. Include ship, hold, and escalate outcomes for uncertain replays or grader disputes.

Compare baseline and candidate on identical generated inputs when possible. Keep generator, schema, and grader versions fixed during the comparison so a score change has one plausible source.

Failure Modes and Tradeoffs

Overly broad generators waste calls on irrelevant combinations. Narrow generators miss interactions outside their model. Strict schemas improve safety but can reject harmless additions and slow evolution. Repair layers improve availability while masking raw contract failures. Shrinking reduces debugging effort but can produce unrealistic cases if domain constraints are weak.

Property-based evals are also resource intensive because generation, model calls, replay, and shrinking multiply executions. Prioritize high-consequence invariants, cache responses, and run smaller deterministic subsets on every change with broader campaigns at chosen checkpoints.

Operational Checklist

• Define the consumer contract and severity for every property.
• Version the input domain, generator, schema, and invariant bundle.
• Label generated, production, incident, and expert cases separately.
• Persist every raw response before parsing or repair.
• Return specific failure codes from deterministic checks.
• Preserve domain validity during shrinking.
• Record seeds, paths, prompts, dependencies, and replay outcomes.
• Calibrate semantic graders against adjudicated human labels.
• Group generated families during splitting and uncertainty analysis.
• Report feature coverage, property failures, slices, and abstentions.
• Freeze the measurement stack for release comparison.

Action Plan: Add Properties to One JSON Contract

Choose one structured output consumed by real code. Freeze its production schema, list five cross-field invariants that protect money, access, evidence, or state, and implement those checks as pure functions. Build a constrained generator around the input domain and persist every call with a reproducible case identity.

Run the current baseline first, review and shrink each meaningful failure, then calibrate any remaining semantic oracle. Only after the harness can explain its generated coverage and reproduce its severe failures should it judge a candidate release. The decisive outcome is not a large test count; it is a small set of enforceable properties with trustworthy evidence.