Shadow Evaluations for LLM Model and Prompt Rollouts

A shadow rollout answers a narrow question: what would the candidate have done on eligible live requests if users had not received it? The production response remains authoritative. The candidate runs in an isolated path, receives equivalent read evidence, produces no real side effects, and is evaluated as a matched pair with complete lineage.

Shadowing is not a shortcut around offline tests or human review. It exposes current traffic and operational behavior that curated sets may miss. The LangSmith evaluation overview distinguishes offline evaluation on curated datasets from online evaluation on production interactions and describes sampling or filtering online traces. A shadow comparison deliberately connects those modes while preserving a hard non-serving boundary.

Write the shadow safety contract

Specify which requests may be duplicated, what data may reach the candidate, where outputs are stored, and how long artifacts are retained. Exclude flows without appropriate privacy treatment, regulated data the evaluation environment cannot handle, and requests whose evidence cannot be replayed safely.

Define "no user impact" concretely. The candidate response is never returned, cached into the serving path, written to user-visible memory, or used to trigger follow-up actions. Candidate failures must not delay production. Shadow compute must have quotas so it cannot exhaust resources needed by the live service.

Name stop conditions before launch: any real side effect, access outside the shadow scope, critical data disclosure, identity mismatch, corrupted paired records, or production resource pressure. Assign an operator who can disable the fork without changing the production model path.

Fork requests without changing production

Generate a stable evaluation ID at the eligibility boundary. The production path proceeds normally. An asynchronous copy of approved inputs, trusted retrieval evidence, recorded tool results, and configuration references enters the shadow queue. Avoid copying mutable credentials or ambient request context that the candidate does not need.

type ShadowJob = {
  evaluationId: string
  requestHash: string
  approvedInputRef: string
  productionConfigId: string
  candidateConfigId: string
  evidenceSnapshotRef: string
  sampleStratum: string
}

async function enqueueShadow(job: ShadowJob): Promise<void> {
  await shadowQueue.publish(job, {
    deduplicationKey: job.evaluationId,
    priority: 'background',
  })
}

This is architecture-level TypeScript. The queue and deduplication APIs are placeholders. The important contract is asynchronous isolation, stable identity, immutable evidence references, and a recorded sampling stratum.

If production retrieval changes between baseline and candidate execution, store the exact retrieved passages or tool results when policy allows. Otherwise the comparison mixes model behavior with changing evidence. When the candidate intentionally includes a new retriever, label that as a full-system comparison and retain both evidence paths.

Isolate tools and side effects

For read-only tools, prefer recorded responses or a shadow replica pinned to the production observation time. For mutable tools, simulate calls and grade intent plus arguments without execution. Use separate credentials, network policies, queues, databases, and memory namespaces so a missed flag cannot write to production.

Never shadow a purchase, email, ticket creation, account update, or deletion by calling the real endpoint twice. An idempotency key designed for production retries may still mutate state or interfere with the baseline request. A tool simulator should validate schema, authorization intent, and expected effect while returning controlled fixtures.

Capture attempted tool actions as candidate output artifacts. Deterministic graders can check allowlists, argument constraints, ordering, and prohibited side effects. A semantic grader can assess whether the chosen action addresses the user request, but it must not receive tool authority.

Trace complete matched pairs

Store production and candidate configuration IDs, prompt or workflow revisions, request hash, evidence versions, tool transcripts, output hashes, errors, latency, and usage under one evaluation ID. Pairing by timestamp or user text later is fragile and can join the wrong requests.

The OpenAI Agents SDK tracing guide documents grouping multiple runs inside a higher-level trace. Whether using that SDK or another tracing system, make the production run and shadow run separately identifiable children of a shared evaluation context without exposing candidate details to the serving response.

Represent missingness. A candidate timeout, queue expiry, policy exclusion, and parser error are different states. Do not silently evaluate only pairs where both systems succeeded, because that removes candidate reliability failures and biases the comparison toward completed runs.

Sample traffic with known probabilities

Apply eligibility filters before random sampling and log both. Use a stable hash or recorded random draw so assignment can be audited. Stratify rare high-risk intents, languages, long threads, retrieval-heavy requests, and tool workflows when those slices matter to release safety.

Oversampling changes the sample composition. Store inclusion probabilities or stratum weights, then report an unweighted diagnostic view and an appropriately weighted traffic view. Do not claim the oversampled critical set represents normal incidence.

Watch for time bias. A run confined to one weekday, campaign, incident, or geographic window may not cover expected traffic. Extend across meaningful operating periods or explicitly limit the inference. Repeated messages from one thread are correlated; use thread-aware analysis when the product outcome is conversational.

Evaluate with blinded paired evidence

Run deterministic checks first for schema, tool policy, safety rules, citations, and known critical invariants. For semantic comparison, hide configuration identity, randomize output order, permit ties and abstentions, and retain the slot mapping. Use a swapped diagnostic panel to audit position sensitivity.

Calibrate model judges against human-adjudicated shadow pairs. The review sample should include wins, losses, ties, critical slices, evaluator disagreements, and a random component. Reviewers must see equivalent evidence for both systems and should not know which one is the rollout candidate.

Keep reference-free online signals distinct from reference-based offline scores. User feedback, escalation, or correction can arrive later and may be missing for many requests. Treat it as another outcome with its own observation window, not as an automatic truth label for each pair.

Estimate uncertainty and inspect slices

The unit of analysis should match assignment and user experience: request, thread, account, or another stable cluster. Calculate paired differences and confidence intervals at that level. Repeated evaluator calls on one request measure judge stability, not additional production coverage.

Report candidate wins, production wins, ties, abstentions, missing candidate runs, and invalid evaluations before deriving a summary. Inspect high-risk intents, language, customer segment where permitted, response length, retrieval mode, tool class, and latency band. Keep denominators visible.

Do not treat a wide interval as a pass. It indicates insufficient evidence for the chosen boundary. Collect more eligible cases, narrow the supported rollout slice, or retain production. Critical deterministic failures can veto regardless of aggregate uncertainty.

Predeclare the rollout decision

Combine quality, safety, reliability, latency, cost, and operational readiness without averaging incompatible units. The following values are illustrative thresholds only:

{
  "policyLabel": "illustrative-only",
  "candidatePreferenceLowerBound": 0.51,
  "maximumRelativeP95LatencyIncrease": 0.10,
  "maximumShadowErrorRate": 0.01,
  "criticalRegressionRule": "zero-observed-and-mandatory-review",
  "unsupportedSliceDisposition": "exclude-from-rollout",
  "judgeCalibrationState": "must-be-current"
}

An observed zero is not proof of zero future failures; it is a local gate coupled with monitoring. Require enough cases for each reported automatic boundary. If the candidate wins globally but loses a material slice, start with a constrained rollout only when routing that slice is reliable and policy-approved.

Define sequential stopping if teams will inspect results daily. Without a predeclared rule, stopping as soon as the preferred answer appears can bias the conclusion. Security stop conditions remain immediate and separate from statistical early stopping.

Continue evaluation after promotion

Shadow evidence supports a controlled rollout, not an instant full switch. Begin with an approved cohort, preserve rollback, and compare online outcomes with the shadow forecast. Monitor errors, safety signals, user corrections, task completion proxies, latency, cost, and judge health.

Keep a small holdback or continued shadow path when policy and product design permit it. Revalidate offline gold cases and production sentinels after configuration changes. A model, prompt, retriever, tool, or policy update starts a new evaluation version.

Investigate discrepancies between shadow predictions and served outcomes. Exposure can change user follow-up behavior, tool state, and feedback availability. Update the evaluation design rather than rewriting the prior decision report.

Run a decisive shadow plan

1. Approve eligibility, privacy, retention, isolation, quotas, stop conditions, and a named operator before duplicating traffic.
2. Fork immutable evidence asynchronously under a stable evaluation ID while production remains the only serving path.
3. Block real side effects, use simulations or replicas, and record every attempted tool action and missing run state.
4. Sample with known probabilities across meaningful slices and periods; preserve weights and thread or account clustering.
5. Apply deterministic vetoes plus calibrated, identity-blinded, order-randomized pairwise judgment and human review.
6. Report paired uncertainty, raw outcomes, slices, reliability, latency, and cost against predeclared boundaries.
7. Promote through a reversible cohort and verify served outcomes against the shadow forecast while sentinels continue running.

A good shadow run leaves production untouched and the decision better informed. If the candidate can affect users, tools, or measurement selection before approval, it is no longer a shadow.