Architecture for Continuous Evals from Production Traces to Release Gates

Continuous LLM evaluation is a controlled evidence loop, not a cron job that scores random conversations. Production traces reveal real intents and failure modes; labels turn selected traces into reusable examples; offline experiments compare a candidate with the deployed baseline; release gates enforce risk policy; later outcomes test whether the chosen change helped in use.

The architecture must prevent feedback from becoming self-confirming. If the same traces tune prompts, select thresholds, and prove the release, apparent improvement is expected even when generalization is absent. Preserve provenance, version every transformation, and keep independent confirmation data.

Define the loop's decision contract

Choose the decisions the system supports: alert on a production safety pattern, promote a prompt, change a retrieval policy, replace a model, or expand traffic. For each decision, declare evaluation unit, metrics, severe failure taxonomy, critical slices, baseline, candidate, minimum evidence, uncertainty treatment, and owner. A monitoring signal is not automatically suitable as a release criterion.

Keep three data products separate. The trace lake stores governed operational evidence. The evaluation registry stores examples, labels, splits, rubrics, and provenance. The experiment ledger stores immutable runs and decisions. Joining them is necessary; letting one mutable table serve all three makes leakage and accidental relabeling hard to detect.

Every release decision should be reproducible from artifact IDs. Include application, prompt, model configuration, tool and retriever versions, dataset revision, evaluator versions, and policy revision. Aliases such as "production" are useful for routing but inadequate as historical evidence.

Capture traces with evaluation-grade identity

Create a root trace for the user interaction and child spans for model calls, retrieval, tool proposals and results, guardrails, retries, and post-processing. Attach tenant-safe trace ID, application version, route, language, query class, latency, token or cost accounting where available, error categories, and feedback references. Use stable evidence and tool operation IDs so failures can be attributed across retries.

The OpenAI Agents SDK tracing documentation describes traces containing model generations, tool calls, handoffs, guardrails, and custom events. Regardless of tracing vendor, define a portable internal envelope. Raw provider spans can change shape; the evaluation loop needs stable semantic fields.

Do not assume full payload retention is acceptable. Store consent and retention classifications on the trace, redact before evaluation export, and allow a tombstone to invalidate downstream copies when policy requires deletion. When raw text is prohibited, hashes, categories, and separately authorized evidence references can still support operational attribution.

Sample for coverage, severity, and novelty

Uniform random sampling estimates common behavior but misses rare costly failures. Combine a probability sample for trend estimation, stratified samples for important cohorts, always-capture rules for declared severe events, disagreement sampling for evaluators, and novelty sampling for unseen intents or trajectories. Tag every selected trace with its sampling reason and probability when meaningful.

Prevent loud users and long conversations from dominating. Sample at an independent unit such as account, conversation, or task, then select turns within that group. Deduplicate retries, near-identical prompts, and copied content before splitting. Preserve temporal slices so a current incident is not washed out by historical volume.

An illustrative selector can keep sampling logic auditable:

type TraceSummary = {
  traceId: string;
  accountId: string;
  severeSignal: boolean;
  slice: string;
};

function samplingReason(trace: TraceSummary, stableBucket: number) {
  if (trace.severeSignal) return "declared-severe";
  if (trace.slice === "new-intent") return "novelty-review";
  if (stableBucket < 5) return "probability-sample"; // Illustrative threshold.
  return null;
}

Compute the bucket from a stable keyed hash in real systems, not runtime randomness, and protect the key. The value 5 is illustrative; sampling rates must reflect volume, cost, and estimation needs.

Turn traces into governed labels

Redaction and authorization precede annotation. Then attach a rubric version, label source, reviewer identity or automated evaluator version, timestamp, confidence or adjudication status, and evidence available at labeling time. Keep user feedback, model-judge output, deterministic findings, domain outcomes, and human labels as separate observations.

Promote a trace into a dataset only after validating that the input is complete and legally retainable. Add the original trace reference and transformation log. Corrected references should not overwrite observed production output. For conversations, preserve the required prior turns while excluding irrelevant sensitive history.

The LangSmith dataset management documentation describes creating datasets from tracing projects and annotation workflows. The key architectural property is repeatability: an example is a versioned evaluation object, not merely a link to a mutable production record.

Maintain train, development, and sealed test roles

Use a working set to design prompts and repairs, a development set to choose configurations and thresholds, and a sealed confirmation set for release evidence. Group related traces before splitting. A paraphrase, retry, translated copy, or adjacent chunk from one incident should not cross partitions if it can leak the expected answer.

Retain stable anchor cases across dataset revisions for longitudinal comparison, but add current traffic and novel failures. Record why examples were added, changed, or retired. Dataset growth without curation can overweight old incidents and increase evaluation cost without improving decision coverage.

Production-derived data creates a temporal trap: outcomes may arrive after a release and be mistakenly attached to the wrong application version. Join outcome events to the trace and version active at decision time, and represent unresolved outcomes explicitly.

Run paired offline experiments

Execute baseline and candidate against the same frozen examples and environment contract. Store per-example outputs before grading so application generation and evaluator changes can be isolated. Use deterministic checks for exact behavior, calibrated model graders for bounded semantic criteria, and targeted human review for disagreement or severity.

The LangSmith evaluation overview separates offline evaluation on curated datasets from online evaluation on production runs and describes feeding failed traces back into datasets. Preserve that distinction: online signals discover and monitor; offline experiments provide controlled paired release evidence.

Report per-example deltas, critical slices, severe failure counts, operational errors, cost, and latency. Macro, traffic-weighted, and severity-weighted summaries answer different questions. Name the aggregation rather than publishing one unlabeled quality number.

Diagnose regressions with a root-cause tree

When a gate moves, branch first by traffic evidence, sampling, labeling, application, evaluator, and policy. Traffic evidence may be incomplete or delayed. Sampling may have changed cohort mix. Labels may use a revised rubric. Application differences may belong to retrieval, prompting, tool execution, or post-processing. Evaluator configuration or parsing may drift. Policy may change weighting, thresholds, or missing-value behavior.

Trace every transformation from production span to dataset example to experiment row to gate contribution. Store parent IDs and content hashes. A gate report should let an investigator select one failed case and reconstruct its raw authorized evidence, redaction, label, candidate outputs, evaluator observations, and policy result.

This lineage also prevents double counting. One production incident can generate several traces, reviews, and regression examples; those artifacts may be useful, but they are not independent observations unless the analysis declares them so.

Use isolation experiments before changing the gate

Replay stored baseline and candidate outputs through the current evaluator to isolate evaluator behavior. Recalculate the gate from stored observations to isolate policy. Replay both applications against frozen tool and retrieval fixtures to isolate application logic. Rebuild one dataset revision from source lineage to verify transformation determinism.

Run counterfactual sampling reports: would the decision change under the previous slice mix or after removing duplicated incidents? Blind-adjudicate a sample of model-grader disagreements. Inject missing outcomes, delayed labels, trace export failure, evaluator timeout, and duplicate event delivery. The system must become visibly incomplete rather than manufacturing a pass.

Enforce safety and ownership boundaries

Application teams own complete trace emission and remediation. Evaluation engineering owns sampling, datasets, rubrics, graders, and experiments. Data governance owns consent, retention, redaction, and deletion propagation. Domain reviewers own adjudication. Release engineering owns gate execution, while product and risk owners define severity and exception authority.

Production credentials never belong in offline replay. Replace tools with controlled fixtures or scoped test environments. Sanitize untrusted trace content before model grading, and deny graders arbitrary tool use. Limit who can read raw traces separately from who can view derived evaluation reports.

Exceptions require a named owner, affected slices, rationale, compensating control, and expiry. The same person should not be able to alter a sealed dataset, change the gate, and approve the resulting release without review in high-risk systems.

Combine quality, operations, and uncertainty in gates

Use non-compensating vetoes for severe safety and authorization failures. Add slice-level quality conditions, operational reliability, evaluator completeness, and paired uncertainty. Example values below are illustrative only:

{
  "illustrativeThresholds": true,
  "releaseGate": {
    "maximumNewCriticalFailures": 0,
    "minimumOverallDeltaLowerBound": 0.0,
    "maximumCriticalSliceRegression": 0.01,
    "maximumEvaluationPipelineErrorRate": 0.005,
    "minimumTraceLineageCompleteness": 0.99
  },
  "inconclusive": "retain-baseline",
  "canaryRollback": ["critical-safety-signal", "trace-loss", "outcome-regression"]
}

Choose actual thresholds from risk, historical variation, reviewer capacity, and service objectives. A narrow gate catches less and runs cheaply; a broad gate improves coverage but adds latency, compute, and maintenance. Make that tradeoff per release class rather than silently skipping expensive checks under time pressure.

Close the loop with measured outcomes

After a passing candidate enters canary traffic, compare matched or appropriately designed outcome cohorts and continue safety monitoring. Link feedback to the exact trace and version. Do not declare causal success from a raw before-after dashboard when traffic, seasonality, or routing changed.

Send novel failures and adjudicated disagreements into the working dataset, not automatically into the sealed test set. Review whether a new evaluator or slice is needed, validate the repair offline, and repeat the release process. Retire temporary incident gates only through a recorded policy change.

The operating plan is direct: instrument trace lineage, govern sampling, label with provenance, maintain independent dataset roles, run paired experiments, enforce versioned gates, canary safely, and feed back only adjudicated evidence. Continuous evaluation is trustworthy when each cycle adds knowledge without erasing the conditions under which earlier decisions were made.