Versioning LLM Eval Datasets Without Losing Baseline Comparability

An evaluation score has meaning only relative to a measurement system. Change the examples, references, rubric, grader, slice definitions, or weights, and the number may no longer be comparable with yesterday's baseline. Dataset versioning must therefore capture more than a file revision; it must identify the complete scoring contract.

The practical solution is immutable snapshots plus explicit change classes. Small corrections, population refreshes, new risk slices, and rubric redesigns should follow different comparison rules. A bridge experiment then shows whether an apparent score movement came from the candidate system or from the ruler used to measure it.

Treat the Baseline as a Measurement Contract

Write down what a reported metric estimates: population, inclusion rules, example weights, task schema, reference policy, grader stack, aggregation, slice taxonomy, and uncertainty method. A version number that covers only input rows leaves silent changes free to rewrite the baseline.

The LangSmith evaluation concepts guide describes datasets, examples, evaluators, experiments, and related evaluation concepts. Those are useful building blocks for version control, but comparability still depends on a team-level policy that freezes the complete eval specification and records why each component changed.

Give Every Component an Immutable Identity

Assign the dataset snapshot a stable version and content hash. Give separate identities to the input schema, reference outputs, rubric, deterministic graders, model-grader prompts and configurations, slice rules, weighting policy, and aggregation code. The run manifest should resolve every identity without relying on mutable latest names.

Preserve data provenance at example level: source system, collection interval, root-family identifier, transformation history, annotator and adjudication lineage, privacy class, and exposure status. A corrected label should become a new labeled artifact linked to the original, not an unexplained overwrite.

An example manifest can keep these dependencies explicit:

{
  "datasetVersion": "support-eval/2.1.0",
  "datasetSha256": "<content-hash>",
  "schemaVersion": "support-task/3",
  "rubricVersion": "resolution-quality/5",
  "graderBundle": "grader-bundle/8",
  "slicePolicy": "support-risk-slices/4",
  "weightPolicy": "production-mix/2",
  "parentVersion": "support-eval/2.0.0",
  "changeClass": "label-correction"
}

The version strings are illustrative. The important property is that a historical run can recover the exact contract it used.

Classify Changes by Measurement Impact

Use a small change taxonomy. A metadata-only change does not affect inputs, labels, or scoring. A correction fixes an error while preserving the intended construct. A refresh replaces examples to represent a newer population. An extension adds a new slice or task while retaining existing anchors. A redesign changes the construct, rubric scale, grader, weighting, or aggregation.

Do not borrow software semantic-versioning rules mechanically. One reference-label correction can materially change a tiny critical slice, while hundreds of added descriptive tags may change no score. Classification should follow measurement impact.

Every proposal should state the reason, affected example IDs, expected score direction if known, changed slices, leakage status, re-annotation needs, and compatibility claim. A reviewer should approve the class before results from the new version are used.

Protect Provenance and Holdout Integrity

Refreshing a dataset can introduce leakage even when the old version was clean. New production traces may have been used in prompt debugging, incident response, fine-tuning, or grader calibration. Run exact, family-level, temporal, and reviewed similarity checks against all development sources before admitting them to a sealed snapshot.

Track exposure separately from content version. If an engineer inspects a sealed example after a failure, the bytes did not change but its role did. Mark it exposed, move it to a development regression set for future tuning, and decide whether the sealed version needs replenishment.

Grouping rules must survive version changes. Conversation siblings, source-document variants, and synthetic transformations should carry the same family identifier into new snapshots so a refresh does not accidentally split related evidence across development and holdout.

Version Labels, Rubrics, and Graders Together

Reference corrections and rubric edits can move scores without any model change. Preserve original independent annotations, adjudicated outcomes, rubric version, reviewer qualifications, and reasons for correction. If a rubric is materially revised, re-label a bridge subset under both definitions rather than assuming old labels can be translated.

Deterministic graders need versioning when parsers, tolerances, schemas, or business rules change. Model graders require the same treatment for prompt, rubric projection, few-shot examples, output parsing, and configuration. Calibrate a new grader against a stable adjudicated set and report disagreement by slice.

Do not update dataset and grader simultaneously unless the bridge design can separate their effects. A staged migration, first grader then data or vice versa, often produces clearer evidence, even though it takes an extra run.

Design a Bridge Experiment

Select stable anchor systems, ideally including the current production configuration and one or more historically understood variants. Run each anchor on both old and new contracts. If possible, also score shared examples under both grader versions. This creates a connected comparison instead of two unrelated leaderboards.

For paired shared cases, inspect the distribution of score differences rather than only means. For added or removed cases, report the old shared-core metric, new shared-core metric, and new full-set metric separately. Slice changes deserve the same treatment.

def bridge_summary(old_scores, new_scores, shared_ids):
    paired = [new_scores[i] - old_scores[i] for i in shared_ids]
    return {
        "shared_count": len(paired),
        "mean_paired_shift": sum(paired) / len(paired),
        "improved": sum(delta > 0 for delta in paired),
        "unchanged": sum(delta == 0 for delta in paired),
        "regressed": sum(delta < 0 for delta in paired),
    }

This summary does not establish uncertainty by itself. Add an interval suited to the paired design and cluster related examples when necessary.

Decide the Level of Comparability

Use explicit labels. Directly comparable means the scoring contract is unchanged for the claimed metric. Linked by bridge means a documented overlap supports interpretation with uncertainty. Restated means historical systems were rerun under the new contract. Not comparable means the construct or evidence changed too much for a defensible numeric trend.

Do not force continuity for dashboard aesthetics. A broken trend line with a clear annotation is more honest than a smooth line that silently combines different populations or rubrics. Keep old reports available and add the new version boundary to every chart and export.

Comparability can differ by slice. A new language slice may have no historical baseline, while unchanged English support cases remain directly comparable. Publish that matrix instead of assigning one status to the entire dataset.

Carry Uncertainty Into Slice Analysis

Bridge samples are finite, and many changes target rare failures. Report paired confidence intervals or cluster-aware bootstrap intervals for aggregate and critical-slice shifts. A small observed bridge shift with a wide interval does not prove equivalence; it says the bridge cannot resolve the difference precisely.

Adding difficult cases will often lower the full-set score by design. That is not necessarily a model regression. Show performance on shared anchors and new slices separately, and explain any weighting change. If the new score is reweighted to production prevalence, preserve the weights used for both versions.

Annotator disagreement and model-grader calibration uncertainty also belong in the version report. A measurement change that reduces label ambiguity may be worthwhile even if it breaks direct continuity, but the decision should be explicit.

Bind Version Changes to Release Decisions

Freeze the dataset version before evaluating release candidates. If a critical label error is found during the run, stop, create a corrected snapshot, determine which candidates need rerunning, and update the comparison plan. Quietly patching the active set allows candidates to be judged by different contracts.

An illustrative policy might allow metadata-only changes without a bridge, require shared-anchor analysis for corrections and extensions, and require full restatement for a construct redesign. These categories and thresholds are local governance choices, not universal rules.

The release record should identify candidate, baseline, dataset contract, bridge status, per-slice uncertainty, exceptions, and approver. When evidence is not comparable, choose a fresh baseline rather than presenting an unsupported improvement claim.

Failure Modes and Tradeoffs

Never changing a golden set preserves trend continuity but lets it become stale and overexposed. Frequent refreshes improve relevance but consume annotation budget and weaken historical links. Large shared cores support comparability yet can crowd out emerging risks. Rerunning all historical systems gives the cleanest restatement but may be expensive or impossible when dependencies are gone.

Versioning every component adds operational overhead. Minimize that burden with automated manifests and hashes, but keep human review for measurement-impact classification. Automation can detect changed files; it cannot decide whether the construct still means the same thing.

Operational Checklist

• Define the population, construct, scoring, slices, and weights in one contract.
• Hash immutable datasets and version every dependent measurement component.
• Preserve source, transformation, annotation, and exposure lineage per example.
• Classify proposed changes by measurement impact.
• Recheck leakage when admitting refreshed or derived cases.
• Calibrate changed rubrics and model graders on stable evidence.
• Run anchor systems across old and new contracts.
• Report shared-core, added-slice, and full-set results separately.
• Quantify paired uncertainty and account for clustered examples.
• Label comparison as direct, bridged, restated, or not comparable.
• Freeze one contract for every release comparison.

Action Plan: Introduce Versioning Without Losing History

Freeze today's complete eval contract and call it the baseline snapshot. Add component hashes and provenance fields, then classify the next proposed change before applying it. Choose anchor systems and shared examples for a bridge, and predefine what evidence will support direct, linked, or broken comparability.

Run the bridge, publish aggregate and slice-level shifts with uncertainty, and make the compatibility label part of the release report. Only then promote the new snapshot as the default. The process succeeds when future readers can reproduce an old score and understand exactly why a newer score can, or cannot, be compared with it.