Sampling Production Traces for Risk-Weighted LLM Eval Datasets

Production traces are evidence of real use, not a ready-made evaluation dataset. Traffic frequency, feedback visibility, privacy rules, and logging gaps all shape what appears in a trace store. A defensible sampling process makes those forces explicit, then selects examples according to product risk and the release decision the evaluation must support.

The goal is not to recreate production at miniature scale. It is to preserve enough common behavior to detect broad regressions while deliberately increasing coverage of failures that are rare, costly, or hard to observe. That requires provenance, a risk taxonomy, calibrated labels, controlled sampling, and a coverage report that travels with every eval run.

Begin With the Release Question

Write the decision before writing the query. A launch comparison, a weekly regression check, and an investigation of one safety incident need different populations. Name the candidate change, affected surfaces, unacceptable regressions, and which results can block release. Without that contract, trace curation becomes an attractive collection of examples with no defensible denominator.

The LangSmith dataset management guide describes ways to add traces, annotation-queue examples, imported records, and synthetic examples to a dataset. Regardless of storage tool, preserve the distinction between how an example was found and why it was admitted. Selection reason is part of the data, not a curator's private memory.

Establish an Eligible Trace Population

Define inclusion before risk weighting. Specify collection windows, environments, product versions, supported locales, account classes, and whether retried or abandoned interactions count. Remove internal tests, monitoring probes, corrupted records, and interactions that cannot legally or ethically be retained. If redaction changes task meaning, exclude the example or use a controlled review environment rather than pretending the transformed trace is equivalent.

Create an immutable provenance record before normalization. Useful fields include a protected source identifier, capture timestamp, product route, model configuration family, input and output schema versions, feedback channel, retention class, transformation log, and ingestion job identifier. Keep raw content access narrower than access to the eval representation. A reviewer should be able to reconstruct lineage without seeing unrelated personal data.

Logging coverage is itself a slice. If tool errors are recorded only after a final response, for example, the trace population cannot estimate tool-call failure frequency. Document blind spots in the dataset card and avoid generalizing beyond the observable population.

Build a Risk Taxonomy That Supports Sampling

Use dimensions that lead to action. A practical taxonomy might include product surface, user intent, consequence severity, language, input length band, tool-use path, retrieval status, policy category, user feedback, and known incident family. Avoid one overloaded label such as hard; it cannot explain what the sampler protected or what engineering team owns a regression.

Severity and frequency should remain separate. A low-frequency action with irreversible financial or account impact may deserve a guaranteed sample allocation, while a frequent greeting can be represented with a smaller fraction. Add an unknown value for dimensions that cannot be labeled reliably. Silent guessing creates false coverage.

Start with deterministic rules for facts in metadata, such as route, locale, HTTP status, or tool name. Use trained annotators for judgment such as harmfulness or instruction satisfaction. A model grader can prioritize a review queue, but sampled cases near its decision boundary should receive human review until its errors are understood.

Allocate Samples Across Strata

Partition eligible traces into strata, then assign each stratum a target count. One simple planning score is:

allocation_weight(h) = traffic_share(h) * risk_multiplier(h) * uncertainty_multiplier(h)
target_count(h) = budget * allocation_weight(h) / sum(allocation_weights)

This is a planning device, not a universal formula. Multipliers encode policy choices and should be reviewed by product, safety, and domain owners. Set minimum floors for critical strata so rounding or low traffic cannot erase them. Cap dominant strata so ordinary traffic does not consume the full annotation budget.

A deterministic sampler should record its seed and reason for every selection:

from random import Random

def sample_strata(rows_by_slice, targets, seed):
    rng = Random(seed)
    selected = []
    for slice_id in sorted(targets):
        rows = sorted(rows_by_slice[slice_id], key=lambda row: row["trace_hash"])
        count = min(targets[slice_id], len(rows))
        chosen = rng.sample(rows, count)
        selected.extend({**row, "sample_reason": slice_id} for row in chosen)
    return selected

Store unsatisfied targets when a stratum has too few eligible traces. That gap may trigger targeted collection or carefully labeled synthetic cases, but synthetic data should not be reported as observed production coverage.

Calibrate Labels and Graders Before Scaling

Write rubrics around observable evidence. Include positive examples, boundary cases, exclusion rules, and an abstain path. Have multiple annotators label a shared calibration batch independently, compare disagreements by slice, adjudicate the reasons, and revise the rubric before distributing the main queue. Agreement is a diagnostic for ambiguous instructions and difficult cases, not proof that the labels are true.

Use deterministic graders for parseability, required fields, exact tool names, known policy flags, and other reproducible properties. Use model graders only for criteria that require semantic judgment, and pin their prompt, rubric, input projection, and configuration in the eval run. Calibrate their decisions against adjudicated human labels and inspect false-positive and false-negative patterns separately.

Do not let one grader both define the risk label and score the candidate output without independent checks. Correlated errors can make a slice look healthy because the same interpretation shaped admission and judgment.

Prevent Leakage and Duplicate Influence

Production traces can overlap with prompt-development examples, fine-tuning data, support transcripts, incident reproductions, and earlier eval sets. Maintain a registry of example fingerprints and lineage across those stores. Check exact normalized hashes, shared conversation identifiers, near duplicates, and repeated templates with different names or numbers.

Split related interactions as a group. Turns from one conversation, repeated requests from one document, and variants derived from one incident should not land on both sides of a development and sealed holdout boundary. Time-based separation helps when prompts were tuned on earlier traffic, but it does not replace entity grouping.

Deduplicate for statistical influence, not only storage. A burst of the same outage response can overwhelm a sample and make one event appear to be many independent observations. Retain one representative, assign an incident cluster, or use cluster-aware analysis.

Audit Coverage and Uncertainty by Slice

Publish the eligible population count, sampled count, labeled count, and scoreable count for every required slice. Compare planned and achieved allocations. A top-line pass rate can improve while a critical locale or tool path regresses, so release reports need both aggregate and protected-slice results.

Sampling changes the meaning of the aggregate. If the eval intentionally oversamples high-risk traces, either report stratified results without pretending they represent traffic, or reweight estimates using documented population shares. Show confidence intervals or bootstrap intervals where the sampling design supports them. For tiny critical slices, state that evidence is insufficient rather than converting one success into certainty.

Track grader abstentions and annotation disagreements as outcomes. High uncertainty in a high-severity slice is itself a reason for investigation, even when the point estimate clears an illustrative threshold.

Convert Evidence Into a Release Decision

Define three outcomes: ship, hold, and escalate for review. A policy might require no confirmed critical-policy failures, no material paired regression on protected slices, and adequate coverage for changed surfaces. Those are examples of policy structure, not universal standards. Thresholds should follow consequence, reversibility, and the organization's tolerance for uncertainty.

When the sample lacks a required slice, do not silently average around it. Mark the gate incomplete and choose whether to collect more traces, run a targeted test set, narrow the release, or accept risk through an accountable exception. Record the decision beside dataset version, sampler seed, grader versions, and candidate identifier.

Recognize Failure Modes and Tradeoffs

Feedback-only sampling overrepresents users who can and will report problems. Error-only sampling loses normal behavior and inflates apparent defect rates. Pure traffic sampling hides rare severe cases. Aggressive deduplication can remove meaningful variation, while weak deduplication gives repeated incidents excessive influence. Rich metadata improves slicing but raises privacy and governance cost.

Risk weighting also creates maintenance work. Taxonomies drift as features change, and minimum slice floors can make datasets expensive. That cost buys diagnostic power only when owners act on slice results. Retire labels that no longer influence collection or decisions, but retain mappings so historical reports remain interpretable.

Operational Checklist

• State the release question and target population.
• Define eligibility, exclusions, privacy treatment, and known logging gaps.
• Capture source lineage before redaction or normalization.
• Separate traffic frequency from consequence severity.
• Set stratum floors, caps, sample targets, and a reproducible seed.
• Calibrate annotators on boundary cases and preserve adjudication notes.
• Assign exact checks to deterministic graders and semantic criteria to calibrated model graders.
• Group conversations, incidents, and derived variants during leakage checks.
• Report planned, sampled, labeled, and scoreable counts by slice.
• Show uncertainty and abstentions instead of forcing every case into pass or fail.
• Bind release rules to protected slices and document accountable exceptions.

Action Plan: Build the First Governed Sample

Choose one upcoming release and freeze a collection window. Inventory eligible traces, write a compact risk taxonomy, and review the privacy boundary with the data owner. Draw a sample that combines routine traffic with guaranteed critical-slice floors. Run a blind calibration batch, repair the rubric, and complete leakage and duplicate checks before scoring any candidate.

Finally, publish a dataset card with lineage, exclusions, sampling targets, achieved coverage, grader definitions, uncertainty method, and release rules. If a reader cannot tell what population the results describe or which missing slice could change the decision, the sample is not ready to gate a release.