Preventing Train-Test Leakage in Private LLM Evaluation Sets

A private eval set is useful only while its outcomes remain independent of the work it judges. If examples leak into prompt iterations, fine-tuning records, retrieval indexes, grader calibration, or debugging notes, the resulting score measures familiarity with the test as much as generalization to the product risk.

Leakage prevention is therefore an access and lineage problem, not merely a deduplication job. Teams need to know where each example came from, which related artifacts exist, who saw it, what transformations were derived from it, and when exposure changed its eligibility for a sealed release gate.

Define the Integrity Boundary

List every activity that can adapt to evaluation evidence: model training, prompt edits, few-shot selection, tool descriptions, retrieval corpus changes, output post-processing, model-grader prompts, threshold tuning, and release policy changes. The holdout must be independent of all of them for the comparison being claimed.

The OpenAI Evals API reference provides an implementation surface for defining and running evaluations. A platform can execute a cleanly specified eval, but it cannot infer whether an example appeared in a private tuning spreadsheet last month. Maintain integrity metadata outside the run configuration and attach the resulting dataset version to every execution.

Build an Example Lineage Ledger

Give every root source and derived example stable identities. Record the source system, creation time, collection purpose, owner, consent or retention class, normalization steps, parent identifiers, and all dataset memberships. A derived paraphrase should point to its original case; a translated variant should share a family identifier; turns from one conversation should share a group key.

Track exposure events as append-only records. Useful events include export, reviewer assignment, prompt-debug session, fine-tuning inclusion, retrieval indexing, incident reproduction, and release-result disclosure. Access logs do not prove that a person remembered content, but they establish which independence claims remain plausible.

Keep sensitive raw data behind stricter controls than normalized eval inputs. The ledger can store protected hashes and references instead of personal content. Deleting a raw trace for retention reasons should not erase the fact that its derived example once influenced a dataset.

Threat-Model the Leakage Paths

Direct contamination is easy to describe: the exact holdout prompt appears in training or few-shot examples. Indirect contamination is more common. An engineer may read failures and add matching rules to the system prompt. A support case may become both a retrieval document and an eval example. An annotator may copy a reference answer into a rubric. A model grader may be tuned against the same cases it later scores.

Organizational leakage matters too. A public dashboard showing per-example failures lets teams optimize toward the sealed set even if the inputs are hidden. Repeatedly running candidates against one holdout and selecting the best is adaptive overfitting. Limit run frequency, delay granular disclosure, or require a fresh confirmation set for major decisions.

Write each threat as source, transfer path, exposed artifact, and affected claim. That format turns a vague concern into controls that can be tested.

Detect More Than Exact Duplicates

Canonicalize superficial differences before exact matching: normalize line endings, Unicode representation at ingestion, stable JSON key order, and approved whitespace differences. Keep the original representation for audit. Do not lowercase or strip punctuation when those features alter the task.

Use family-level checks for shared documents, accounts, incidents, and conversations. Add near-duplicate detection for templates with changed names, IDs, or dates. Embedding similarity can help create a review queue, but it is not a proof of leakage; semantically related examples can be independently valid, and lexically different examples can share a hidden source.

import hashlib
import json

def canonical_hash(example: dict) -> str:
    projected = {
        "input": example["input"],
        "reference": example.get("reference"),
        "source_family": example["source_family"],
    }
    payload = json.dumps(projected, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

def has_group_overlap(dev_rows, holdout_rows) -> bool:
    dev_groups = {row["source_family"] for row in dev_rows}
    return any(row["source_family"] in dev_groups for row in holdout_rows)

The hash projection and grouping fields must match the product's threat model. This example is illustrative, not a complete contamination detector.

Split by Family, Entity, and Time

Random row splitting is unsafe when rows share a source. Assign the group first, then place the complete group into development, validation, or holdout. Relevant groups may be conversation, customer issue, source document, generated seed, task template, or incident. When several keys apply, use the broadest relationship that could transfer an answer.

Time-based splits answer whether a system tuned on earlier evidence generalizes to later cases. They are useful for evolving products, but future cases can still duplicate old templates or documents. Combine time boundaries with family checks.

Stratify after grouping so important risk slices remain represented. If grouping leaves a rare slice too small, report that limitation. Moving siblings across the boundary to balance counts would trade visible coverage for invalid independence.

Separate Development, Shadow, and Sealed Sets

The development set is expected to be inspected and optimized against. The shadow set supports periodic internal checks with restricted example visibility. The sealed set answers infrequent high-stakes release questions and should have the strongest controls. Naming these roles prevents a heavily used regression suite from being described as an untouched holdout.

When a sealed case is disclosed for debugging, mark it exposed immediately. It can remain useful as a permanent regression test but should leave future claims of unseen performance. Replenishment needs the same provenance scan and calibration as the original set; silent replacement changes the measurement target.

Budget secrecy. Specify who can view examples, who can run candidates, what aggregate results are visible, how often runs are permitted, and when examples expire. Operational friction is a real tradeoff, so reserve strict controls for decisions that genuinely require an independent estimate.

Keep Labels and Graders Independent

Annotators need calibrated rubrics, but calibration examples should not all come from the sealed holdout. Build a separate calibration pack with similar decision boundaries. Measure agreement, adjudicate ambiguity, and freeze the rubric version before final labeling. If the rubric changes after seeing candidate results, create a new evaluation version and explain why.

Assign deterministic graders to exact constraints such as JSON validity, forbidden field presence, tool-call sequence, or reference-set membership. Model graders can assess semantic criteria, but their prompt and examples can also leak holdout content. Calibrate them on an independent labeled sample and keep grader-development data separate from the cases used for final model comparison.

Blind human review reduces expectation bias. Hide candidate identity and treatment order where practical. If the annotator knows which output is the proposed release, leakage has moved from training data into judgment.

Report Uncertainty and Slice Integrity

An uncontaminated sample can still be too small or unrepresentative. Report uncertainty intervals, abstentions, missing labels, and sample counts for each protected slice. Separate score uncertainty from integrity uncertainty. A narrow statistical interval does not repair a known overlap with prompt-tuning data.

Create an integrity status per slice: clean, partially exposed, unverified, or invalid. One source-family collision in a rare critical slice can matter more than several collisions in a large routine slice. Release reports should show both performance and integrity status so leaders do not average away the caveat.

For adaptive experimentation, treat repeated holdout access as another uncertainty source. A candidate chosen after many attempts against the same gate has more opportunity to overfit than the final comparison alone reveals.

Make Release Rules Respond to Exposure

Specify decisions before results arrive. A clean holdout may support ship, hold, or escalate outcomes based on paired aggregate and slice criteria. A partially exposed holdout should trigger confirmation on fresh cases, a narrower claim, or an explicit risk exception. An invalid holdout cannot support the original release claim, even if its score is high.

Illustrative policy might say that any confirmed overlap in a critical slice blocks automatic approval, while low-risk duplicate clusters require sensitivity analysis with the cluster removed. These are examples, not universal thresholds. Severity, sample replaceability, and decision reversibility determine the response.

Preserve the original result and the corrected rerun. Auditability requires showing why the decision changed, not rewriting history after contamination is discovered.

Failure Modes and Tradeoffs

Overly strict similarity filters can reject legitimate cases and narrow the task distribution. Weak filters can miss paraphrased or source-level overlap. Sealed sets improve independence but slow debugging and create access bottlenecks. Rotating examples reduces repeated exposure but can break baseline comparability. Public rubrics improve transparency while detailed exploit patterns can invite gate-specific optimization.

Resolve these tensions by matching controls to claim strength. Daily engineering tests can be transparent and fast. A final generalization claim needs stronger separation, controlled disclosure, and a bridge design when examples rotate.

Operational Checklist

• Define which adaptation activities the holdout must be independent from.
• Assign stable root, family, example, and transformation identifiers.
• Log exports, inspections, tuning use, indexing, and disclosure events.
• Run exact, normalized, group, temporal, and reviewed semantic checks.
• Split source families before balancing risk slices.
• Label datasets clearly as development, shadow, or sealed.
• Use an independent pack to calibrate annotators and model graders.
• Blind candidate identity during subjective final review.
• Report integrity status beside uncertainty and performance for every critical slice.
• Invalidate and rerun affected comparisons when exposure changes the claim.

Action Plan: Seal a Defensible Holdout

Inventory every private eval source and adaptation channel this week. Build a lineage ledger, group related examples, and scan the proposed holdout against prompt repositories, tuning exports, retrieval documents, support cases, and grader-calibration data. Quarantine anything exposed or unresolved rather than arguing it clean by default.

Then assign access roles, freeze rubric and grader versions, record a dataset snapshot, and define what disclosure follows each run. Run one dry release review using aggregate results first and reveal examples only through the documented process. The holdout is defensible when a later auditor can reconstruct both its data lineage and every opportunity the product team had to adapt to it.