Architecture for Layered LLM Evaluation with Code, Model, and Human Review

A layered LLM evaluation architecture should allocate each decision to the cheapest reliable authority. Code checks exact invariants, model graders assess semantic criteria that cannot be reduced cleanly to rules, and humans resolve ambiguity, severity, novelty, and policy. The layers are not three scores averaged together; they are a governed decision system with vetoes, escalation, and traceable evidence.

Design the decision contract before selecting evaluation tools. Specify what can block a release, what may warn, what becomes inconclusive, and who can approve an exception. Without that contract, teams tend to add evaluators until a dashboard looks comprehensive while no one can explain why a build passed.

Start with a decision and evidence contract

Define an evaluation unit: one response, one conversation, one agent trajectory, or one business outcome. Then list criteria, severity, required evidence, evaluator authority, and disposition. "Helpful" may require a model rubric and periodic human audit; "valid JSON" belongs to code; "performed an unauthorized transfer" belongs to deterministic policy and domain records.

Every decision must reference immutable evidence. Store case ID, dataset revision, input and reference hashes, application artifact, model and prompt configuration, retrieval or tool trace, candidate output, evaluator versions, raw evaluator outputs, human labels, and final policy version. Corrections should create a new evaluation record rather than mutating the original.

Separate measurement from enforcement. Evaluators produce observations with status such as valid, invalid, unavailable, or not applicable. A policy engine turns those observations into pass, fail, warn, review, or inconclusive. This lets the same evidence support a pull-request gate, a shadow experiment, and an audit without changing evaluator semantics.

Make code checks the exact first layer

Use deterministic code for output schemas, required fields, value ranges, exact citations, known forbidden strings, permission boundaries, tool argument constraints, and domain invariants. Code checks are fast, reproducible, and easy to unit test. They also provide direct remediation: a missing field is clearer than a low semantic quality score.

Do not force subjective policy into brittle regular expressions. A code check should have a stable specification and fixtures for valid, invalid, and edge inputs. Emit structured findings with criterion, severity, evidence path, and rule version. Preserve all findings rather than stopping at the first failure when doing so is safe, because the full defect set helps attribution.

Hard vetoes should short-circuit expensive grading when no later layer is authorized to override them. Advisory checks can continue through the pipeline for diagnostic coverage. The distinction belongs in policy, not in scattered evaluator code.

Use model graders for bounded semantic questions

Give each model grader one narrow rubric, the minimum evidence needed, explicit treatment of missing context, and a constrained response schema. Examples include whether a claim is supported by supplied evidence, whether an answer addresses the user's request, or whether a tool trajectory follows a declared plan. Avoid a single "overall quality" grader that blends unrelated risks.

Pin grader configuration and store raw responses. The OpenAI graders API reference documents multiple grader forms, including model-based and deterministic checks. Whatever implementation is selected, expose its type in the evidence record so downstream policy does not treat a string comparison and a semantic judgment as equally uncertain.

Calibrate graders against blinded human labels by criterion and slice. Test candidate-output prompt injection, position effects for pairwise comparisons, irrelevant formatting changes, and repeated calls on boundary cases. A model grader should never receive credentials or unrestricted tools; grading is interpretation, not execution.

Reserve humans for high-value decisions

Route cases to people when deterministic and model evidence disagree, model repetitions conflict on a severe criterion, a new failure cluster appears, policy requires domain expertise, or the case is selected for routine audit. Include an "insufficient evidence" option so reviewers are not forced to invent certainty.

Present the exact candidate evidence, rubric, criterion, and prior automated findings, but blind information that could bias judgment when possible. For pairwise review, balance positions. Retain individual labels before adjudication, record rationale for severe decisions, and distinguish correction of the application from correction of the evaluator.

The LangSmith annotation queue documentation describes focused single-run and pairwise workflows with rubrics and reviewer progress. The architectural lesson is broader than one product: human review needs queue ownership, reservations, completion rules, and feedback that can flow into datasets rather than an informal spreadsheet.

Route work with a versioned policy engine

Routing must be deterministic even when some observations are probabilistic. The policy evaluates severity, disagreement, confidence evidence, novelty, slice criticality, review capacity, and release mode. It should never silently convert evaluator failure into a passing score.

This TypeScript example shows an explicit result envelope:

type Finding = {
  criterion: string;
  evaluatorType: "code" | "model" | "human";
  status: "pass" | "fail" | "unknown";
  severity: "info" | "major" | "critical";
  evidenceRef: string;
};

type EvaluationDecision = {
  policyVersion: string;
  disposition: "pass" | "fail" | "review" | "inconclusive";
  findings: Finding[];
  exceptionId?: string;
};

Keep queue capacity visible. If human backlog exceeds its service objective, do not auto-pass pending cases. Options include holding the release, narrowing exposure, using the previous baseline, or having an authorized owner accept a time-limited exception. Each choice has a product cost and should be observable.

Build failure attribution into the architecture

Use a root-cause tree for every surprising decision. Input evidence covers wrong references, missing retrieval spans, and dataset corruption. Code layer covers rule defects, parser changes, and stale schemas. Model layer covers rubric ambiguity, prompt assembly, service errors, and grader disagreement. Human layer covers unclear guidance, domain mismatch, and adjudication delay. Policy layer covers weighting, veto precedence, slice filters, missing-value handling, and exception logic.

Emit trace spans for evaluator start, evidence fetch, result validation, policy routing, queue transition, and final decision. Join all spans with an evaluation ID, but maintain separate evaluator attempt IDs. That distinction makes retries visible without creating duplicate logical evaluations.

Dashboards should expose counts moving between layers, unknown results, disagreement, severe findings, review age, and exception expiry. A single quality score conceals which authority produced the decision and whether the pipeline was complete.

Run isolation and fault experiments

Replay stored outputs through one evaluator version at a time. Swap the parser while keeping raw grader responses fixed; swap the grader prompt while keeping evidence fixed; recalculate policy without rerunning evaluators; and send the same review item through a blinded second adjudication. These experiments distinguish application change from evaluation-system change.

Inject evaluator timeout, malformed structured output, unavailable reference data, duplicate queue delivery, reviewer abandonment, and policy-service failure. Verify declared outcomes. Code-veto tests must fail even when the model grader is unavailable. A model-only quality criterion may become inconclusive, not zero and not pass.

Shadow new evaluators and routing policies against the current system. Review changed decisions by severity and slice before promotion. This costs compute and review time, but it is cheaper than changing both measurement and release behavior in one opaque rollout.

Protect evaluation data and reviewer independence

Minimize sensitive inputs, redact secrets, encrypt retained artifacts, and scope access by role. Grader prompts and human interfaces can expose user content, retrieved documents, and tool arguments. Retention must match the most sensitive included field, not the harmless aggregate score.

Treat model output as untrusted data. Delimit it, validate grader schemas, and prohibit it from altering rubric instructions. Keep release approvers separate from evaluator developers for high-risk exceptions where feasible. Log who changed a rubric, policy, dataset, or adjudicated label.

Human reviewers need a safe escalation path for harmful content and a way to report rubric defects. Evaluation quality degrades when reviewers must choose between an inaccurate label and blocking their queue.

Set ownership and operational tradeoffs

Product and risk owners define criteria, severity, and override authority. Evaluation engineering owns datasets, grader calibration, and evidence schemas. Application teams own trace completeness and remediation. Platform teams own orchestration, queues, reliability, cost, and latency. Domain reviewers own adjudication; release engineering owns enforcement and exception expiry.

More code checks improve speed and repeatability but increase maintenance when product rules evolve. More model grading expands semantic coverage but adds cost, latency, and uncertainty. More human review increases contextual judgment but creates capacity and consistency constraints. The architecture should route based on consequence and ambiguity, not aim to maximize any one layer.

Gate releases with non-compensating rules

Do not average a critical deterministic failure with favorable model scores. Require layer completeness, severe-slice conditions, calibrated semantic evidence, and disposition of mandatory reviews. Example values below are illustrative:

{
  "illustrativeThresholds": true,
  "releasePolicy": {
    "maximumCriticalCodeFailures": 0,
    "maximumUnresolvedCriticalReviews": 0,
    "minimumModelGraderCalibrationAgreement": 0.8,
    "maximumEvaluatorOperationalErrorRate": 0.01,
    "inconclusiveDisposition": "hold-release"
  },
  "exceptions": {
    "requireNamedOwner": true,
    "requireExpiry": true,
    "cannotOverride": ["authorization-violation", "secret-exposure"]
  }
}

Replace these values with thresholds calibrated to domain risk and measured evaluator behavior. Version the whole policy and prove that the same evidence produces the same decision before enabling enforcement.

Put the architecture into operation

Define the decision contract, build the immutable evidence envelope, implement exact checks, add narrow calibrated model graders, establish human queues, and place one policy engine after the observations. Test each layer alone, inject missing and malformed evidence, and shadow the final decision path.

The architecture is ready when every pass, failure, review, and exception can be traced to an authority, criterion, evidence record, and policy version. Layering succeeds not because it adds evaluators, but because it makes uncertainty and responsibility explicit before a release reaches users.