Combining Deterministic and Model Graders with Veto Rules

A composite evaluation should not let excellent tone cancel an exposed secret. It should not let valid JSON cancel a fabricated policy answer either. Deterministic and model graders observe different properties, so the ensemble needs an explicit policy that preserves each result, applies critical vetoes, handles missing evidence, and produces a disposition people can reconstruct.

The central design choice is precedence, not arithmetic. The OpenAI graders API reference includes grader forms for structured labels, scores, and multi-grader definitions. A production policy still must decide which findings block, which combine, which require review, and which are merely diagnostic.

Decompose quality into decision criteria

Begin with the release contract. List properties that have one inspectable answer: output parses, required fields exist, amounts reconcile, cited document IDs are allowed, no forbidden identifier appears, and tool arguments respect policy. These belong to deterministic graders whenever reliable code can express them.

List properties that require interpretation: whether the answer addresses the user's intent, whether a caveat is materially misleading, whether evidence supports a claim, or whether an escalation explanation is useful. Give each semantic criterion its own rubric and label set. A single "quality" judge cannot show which dimension failed.

For every criterion, record severity, evidence input, supported slices, owner, and operational consequence. Some checks are informational during development. Others block a release or route an individual response to review. Make that difference visible before implementation.

Make deterministic graders authoritative in their domain

Code graders should return structured evidence, not only booleans. A schema failure should include the failed path and validator message. A calculation grader should include expected, actual, and tolerance. A permission grader should name the disallowed resource without leaking sensitive content into broad logs.

Deterministic does not mean infallible. Parsers, allowlists, reference data, and tolerances can be wrong. Unit-test the grader with positive, negative, boundary, and malformed fixtures. Version its dependencies and separate evaluator errors from candidate failures.

Do not ask a model to overrule a deterministic fact it cannot observe better. If code proves a required JSON object is missing, a model's statement that the response "mostly follows the format" is irrelevant. Conversely, do not use brittle string matching to decide a nuanced semantic condition just to avoid calibration work.

Constrain model graders to narrow rubrics

Provide only the evidence needed for one criterion and require a structured label with an abstain option. Blind model identity and candidate status. For pairwise semantic graders, randomize visible response order and retain the mapping. For reference-based checks, ensure the reference is trusted and not selected by the candidate.

Calibrate each model grader against human-adjudicated cases by label and slice. Measure invalid output, abstention, false pass, and false fail separately. Repeated calls can quantify instability, but the aggregation rule must be fixed in advance and validated as part of the grader.

The LangSmith evaluation concepts distinguish human, code, model-based, pairwise, reference-free, and reference-based evaluation techniques. That separation is a useful architecture rule: choose the technique from the evidence available instead of forcing every property through one grader type.

Write the veto policy as data

Keep policy outside grader implementations. A grader reports evidence; a versioned decision engine interprets it. This makes precedence reviewable and allows a policy change without pretending the underlying measurements changed.

{
  "policyId": "support-answer-release-v4",
  "requiredGraders": ["schema", "secret_scan", "policy_grounding", "helpfulness"],
  "vetoes": [
    {"grader": "secret_scan", "when": "detected", "reason": "critical_disclosure"},
    {"grader": "schema", "when": "invalid", "reason": "contract_unusable"}
  ],
  "reviewRules": [
    {"grader": "policy_grounding", "when": "abstain", "reason": "insufficient_evidence"}
  ],
  "aggregate": {"criteria": ["policy_grounding", "helpfulness"], "method": "precalibrated-table"}
}

The policy deliberately avoids arbitrary cross-scale averaging. If aggregation is needed, define compatible labels or transform scores using calibration evidence, then test the transformation. Preserve the original component outputs in every report.

Implement precedence as a total decision table

The resolver must define every state: veto, required-grader error, review, ordinary failure, and pass. Evaluate vetoes first, but do not hide missing required graders behind a veto because operations still need to know the run was incomplete.

type Grade = {
  id: string
  status: 'pass' | 'fail' | 'abstain' | 'error'
  severity: 'info' | 'minor' | 'major' | 'critical'
  evidence: Record<string, unknown>
}

type Disposition = 'pass' | 'fail' | 'review' | 'incomplete'

export function resolve(grades: Grade[], required: string[]): Disposition {
  const byId = new Map(grades.map((grade) => [grade.id, grade]))
  if (required.some((id) => !byId.has(id) || byId.get(id)?.status === 'error')) {
    return 'incomplete'
  }
  if (grades.some((grade) => grade.severity === 'critical' && grade.status === 'fail')) {
    return 'fail'
  }
  if (grades.some((grade) => grade.status === 'abstain')) return 'review'
  if (grades.some((grade) => grade.status === 'fail')) return 'fail'
  return 'pass'
}

This simplified TypeScript illustrates precedence, not a complete production policy engine. Real code should emit reason codes and matched policy clauses, validate unknown graders, distinguish optional from required abstentions, and reject duplicate result IDs.

Reserve vetoes for defensible boundaries

A veto should have a direct relationship to unacceptable impact. Examples include unauthorized data disclosure, execution of a prohibited tool action, invalid required transaction structure, a critical financial sign error, or use of a superseded safety policy. Each needs an evidence contract that a reviewer can inspect.

Avoid vetoes for vague low-confidence signals such as "style score below 0.7" unless calibration and impact analysis justify that exact boundary. A model grader can route uncertainty to review without declaring a failure. This is especially important when the judge has not been validated on a new language or domain.

Numeric thresholds are local policy. Label examples accordingly. An illustrative release rule might require no observed critical deterministic failures, send any semantic abstain to review, and require the lower confidence bound for a calibrated pass rate to exceed a chosen target. None of those numbers should be copied without consequence analysis and adequate cases.

Handle disagreement and dependence openly

Two graders can disagree because one is wrong, because they assess different criteria, or because the policy maps their evidence poorly. Do not force agreement. Store a result vector and mine recurring combinations such as schema pass plus grounding fail, or secret-scan alert plus semantic pass.

Account for dependence. Two model graders using similar prompts, references, or underlying models may share failure modes. Multiple votes do not automatically provide independent confidence. Add diversity through different evidence or methods: exact validation, human review, reference checks, and bounded semantic judgment.

When a deterministic and semantic result conflict within the same claimed criterion, stop and investigate the contract. The schema may be too weak, the semantic rubric may be looking at presentation rather than structure, or preprocessing may feed different artifacts to each grader.

Calibrate the complete decision engine

Unit tests prove each rule executes; a policy-level gold set proves the combined disposition matches expert decisions. Include each veto path, required-grader outage, abstention, multiple simultaneous failures, optional grader absence, and ordinary pass. Test precedence explicitly.

Compare final dispositions with human-adjudicated outcomes and inspect confusion by severity and slice. Report confidence over independent cases. Also measure review volume, because an ensemble that sends nearly everything to humans may be safe but operationally unusable. Do not lower a veto simply to meet review capacity; improve evidence or product controls.

Run shadow decisions before enforcing a new policy. Record what the old and new engines would decide on the same artifacts, adjudicate meaningful differences, then activate with rollback criteria. Policy versions must remain attached to historical results.

Monitor rules, graders, and overrides

Track trigger rate, false-positive appeals, confirmed misses, abstentions, errors, latency, and cost for each grader. Track final disposition and matched policy clauses separately. A stable aggregate pass rate can hide a broken required grader if the resolver defaults missing data incorrectly.

Every override needs case ID, original disposition, reviewer evidence, owner, reason, and expiry where appropriate. Repeated overrides of one rule indicate a bad threshold, stale reference data, or a product behavior the policy no longer represents. Review the rule rather than normalizing manual exceptions.

Alert when a critical grader stops reporting, a veto rate changes abruptly, an unsupported slice grows, or sentinels change verdict. Freeze automatic releases until the measurement path is understood.

Put the ensemble into action

1. Decompose the release contract into exact, semantic, operational, and critical criteria with named owners.
2. Implement evidence-rich deterministic graders and narrow structured model graders; calibrate each on the slices where it will operate.
3. Encode required graders, vetoes, review routes, compatible aggregation, and unknown-state behavior in a versioned policy.
4. Test every decision-table branch, including missing results and simultaneous failures, against human-adjudicated policy cases.
5. Shadow the new resolver beside the current process and adjudicate disposition changes before enforcement.
6. Release only when critical vetoes are clear, uncertainty routes to review, and all required evidence is present.
7. Monitor component results, final clauses, sentinels, and overrides so one green summary cannot conceal a failed measurement path.

The ensemble is trustworthy when a reviewer can answer two questions quickly: what did each grader observe, and which approved rule turned those observations into the decision? If either answer is hidden in an average, the policy is not ready to govern a release.