Change the LLM judge without erasing your baseline

Yesterday’s evaluator passed a fixed answer. Its replacement fails the identical bytes today, and the release dashboard now shows a regression in code that never changed. Before blaming the product model, prove what moved inside the judging system.

Decide which meaning must survive the migration

An LLM judge is part of the test system. Its model version, prompt, rubric, evidence order, output schema, parser, and thresholds all influence the final decision. Replacing only the model name can still change label tendencies, explanation style, token usage, and how often the parser receives an unexpected response.

Start by naming the invariant. Perhaps historical “grounded” labels must remain comparable. Perhaps only the binary release decision must remain stable, while scalar scores may move. Perhaps the old judge is known to be too permissive and intentional reversals are the purpose of the migration. A team cannot diagnose drift until it distinguishes expected movement from accidental movement.

Freeze the old pipeline as a versioned specification. Record the judge prompt template, rubric text, label vocabulary, output presentation order, parser version, retry behavior, and decision mapping. Do the same for the candidate. A note saying “upgraded judge” is not enough to reproduce a disagreement six weeks later.

Keep generation out of the bridge run. The bridge dataset should contain fixed product inputs and fixed candidate outputs. Both judges review the same artifacts. If the product model generates again during the comparison, a different completion can be mistaken for judge movement. Store hashes for the evidence bundle and evaluated output so the equality claim can be checked rather than assumed.

Separate the raw judge response from its parsed label. A migration can preserve semantic judgment while breaking the parser because the new judge emits a different key, capitalization, or explanation shape. Conversely, two raw explanations can differ while both map correctly to the same approved label. Those are different outcomes with different owners.

Define a common decision vocabulary. The old judge may return pass, fail, and uncertain while the new one returns scores from another scale. Do not compare raw values as though they share units. Map each judge into a documented canonical result such as pass, fail, review, or invalid. Preserve the raw value beside the mapping.

Thresholds are part of the old meaning. If a score of four out of five previously passed, converting a new zero-to-one score by dividing by five is an assumption, not calibration. Use reviewed bridge cases to determine how the new outputs relate to the product decision. If no stable mapping exists, keep the new rubric’s categories rather than forcing cosmetic continuity.

A bridge dataset does not prove either judge is correct. It shows where their decisions agree, where they differ, and whether the migration mechanics are trustworthy. Human adjudication or deterministic contracts must resolve release-critical disagreements. Agreement between two automated judges can also mean they share the same blind spot.

Assemble cases that expose boundaries, not just averages

Random historical sampling tends to reproduce the largest, easiest class. A useful bridge deliberately covers the cases that can change a release: near-threshold groundedness, acceptable refusals, partially completed tool tasks, ambiguous citations, locale-specific tone, and known parser stress cases.

Start from several sources. Include stable calibration anchors with reviewed decisions. Add recent production-like cases after required privacy controls. Add prior false positives and false negatives from the old judge. Add cases around each rubric boundary. Add a modest number of adversarial variants that challenge output order, long evidence, conflicting documents, and short answers.

Keep the original population fields. Slice, locale, risk class, task type, output length band, and source date help explain clustered movement. A transition that appears only on long evidence may be context handling or truncation. A transition limited to one locale may expose a rubric translation issue. Without these fields, both look like general judge drift.

The Python program below creates a deterministic stratified sample from a CSV history. It reserves required boundary cases first, then fills each requested slice with a seeded sample. The seed makes selection repeatable. The sample sizes are command-line inputs chosen by the team, not claims about sufficient coverage.

from __future__ import annotations

import argparse
import csv
import random
from collections import defaultdict
from pathlib import Path


def load(path: Path) -> list[dict[str, str]]:
    with path.open(newline='', encoding='utf-8') as handle:
        rows = list(csv.DictReader(handle))

    if not rows:
        raise ValueError('history is empty')

    seen: set[str] = set()
    for row in rows:
        case_id = row['case_id'].strip()
        if not case_id:
            raise ValueError('blank case_id')
        if case_id in seen:
            raise ValueError('duplicate case_id: ' + case_id)
        seen.add(case_id)
    return rows


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('history_csv', type=Path)
    parser.add_argument('output_csv', type=Path)
    parser.add_argument('--per-slice', type=int, required=True)
    parser.add_argument('--seed', type=int, default=20260804)
    args = parser.parse_args()

    if args.per_slice < 1:
        raise ValueError('--per-slice must be positive')

    rows = load(args.history_csv)
    randomizer = random.Random(args.seed)
    required = [row for row in rows if row['bridge_required'].lower() == 'true']
    required_ids = {row['case_id'] for row in required}

    by_slice: dict[str, list[dict[str, str]]] = defaultdict(list)
    for row in rows:
        if row['case_id'] not in required_ids:
            by_slice[row['slice']].append(row)

    selected = list(required)
    slice_names = sorted({row['slice'] for row in rows})
    for slice_name in slice_names:
        candidates = by_slice[slice_name]
        randomizer.shuffle(candidates)
        already = sum(1 for row in required if row['slice'] == slice_name)
        needed = max(0, args.per_slice - already)
        if len(candidates) < needed:
            raise ValueError(
                slice_name + ': needs ' + str(needed) +
                ' additional cases, found ' + str(len(candidates))
            )
        selected.extend(candidates[:needed])

    fieldnames = list(rows[0].keys())
    with args.output_csv.open('w', newline='', encoding='utf-8') as handle:
        writer = csv.DictWriter(handle, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(sorted(selected, key=lambda row: row['case_id']))

    print('wrote', len(selected), 'bridge cases')

Sampling code cannot decide what a boundary is. Engineers and reviewers must mark those cases from incident history, rubric calibration, and product risk. Make the selection manifest reviewable so a future maintainer knows why each forced case exists.

Store one record per case and judge version, or store one wide paired record if the schema remains manageable. The essential join key is the versioned case ID. A mutable title or prompt text is not a stable key. Keep exact artifact hashes to detect accidental edits.

A bridge row should distinguish at least these states:

• Both judges returned valid results and agree after canonical mapping.
• Both returned valid results and disagree.
• The old judge returned invalid output.
• The new judge returned invalid output.
• One result is missing because execution failed.
• The mapping is impossible because the rubric vocabularies do not overlap.
• Human review confirms the old decision.
• Human review confirms the new decision.
• Human review chooses a third result or marks the case underspecified.

Invalid and missing are not failures of the evaluated answer. They are failures of the evaluation run. Turning them into fail labels makes a flaky judge appear strict and can block good product changes for the wrong reason.

Treat bridge membership as versioned data. When a case is corrected, create a new case version and keep the prior result tied to the old hash. Quiet edits can manufacture agreement by changing the evidence after a dispute. Record who approved forced boundary cases and why they remain in the suite.

Protect the holdout from calibration leakage. Engineers revising the candidate prompt should not read its labels case by case and then call the next replay an independent check. Keep a development partition for iteration, a holdout partition for the cutover decision, and a small public smoke partition for plumbing. If the holdout is opened to diagnose a failure, retire that version and prepare another reviewed partition before claiming fresh validation.

Trace three migration failures to different owners

The first worked case changes score scales. The old judge produces integers from one to five and the release mapper treats four or five as passing. The new judge produces labels supported, partial, and unsupported. An engineer maps supported to five, partial to three, and unsupported to one because the values look tidy.

A reviewed case contains a correct answer that omits one nonessential detail. The old rubric calls it four, while the new rubric calls it partial. The automatic mapping flips pass to fail. This may be a legitimate policy change, but no mathematical fact says partial equals three. The owner is the canonical mapping and rubric decision, not the product model.

Preserve both raw outputs, ask reviewers whether omission of that detail should block the product task, and encode the answer in a versioned mapping. If “partial” contains both acceptable omissions and material omissions, the new label vocabulary is too coarse for the old release contract. Add a criterion or keep those cases in review rather than inventing precision.

The second case involves cautious refusals. A support assistant receives an account request without authentication. Its answer refuses to reveal details and explains how the customer can verify identity. The old judge rewards policy compliance. The new judge sees that the user’s request was not completed and marks it unhelpful.

Both judgments reflect criteria that may exist. The bridge record should show which criterion dominated. If authentication is a blocking requirement, the new judge is wrong for the product contract even if its general notion of helpfulness is reasonable. Repair the rubric prompt, add refusal anchors, and rerun fixed outputs. Do not tune the product assistant to satisfy an evaluator that forgot the policy.

The third case is a parser failure. The new judge returns a valid label plus a nested rationale, while the parser expects a top-level field. The dashboard records fail because the parsed label is absent. Raw judge output shows the intended pass. No amount of rubric calibration fixes this.

Classify parsing separately and fail the evaluation job as invalid. Add contract tests around accepted response shapes and explicit handling for unknown shapes. Avoid a permissive fallback that searches arbitrary prose for the word “pass”; an explanation such as “does not pass” can be misclassified. The migration cannot proceed until the parser failure rate and affected rows are visible.

A fourth near-miss looks like semantic drift but is data drift. One judge receives evidence documents in chronological order and the other receives them in relevance order. A disputed answer cites an older policy that appears first in one bundle. Compare evidence hashes and ordered document IDs. If they differ, rerun after fixing the packaging pipeline before adjudicating the judges.

The TypeScript comparator below validates paired exports after both judge clients have already run. It names invalid results, checks artifact equality, maps raw labels through version-specific tables, and emits changed decisions. It does not call any model API or assume a provider response format.

type Canonical = 'pass' | 'fail' | 'review';
type JudgeVersion = 'old-v7' | 'new-v1';

type JudgeResult = {
  caseId: string;
  judgeVersion: JudgeVersion;
  outputHash: string;
  evidenceHash: string;
  rawLabel: string | null;
  parseError: string | null;
};

const mappings: Record<JudgeVersion, Record<string, Canonical>> = {
  'old-v7': {
    '5': 'pass',
    '4': 'pass',
    '3': 'review',
    '2': 'fail',
    '1': 'fail',
  },
  'new-v1': {
    supported: 'pass',
    partial: 'review',
    unsupported: 'fail',
  },
};

function canonicalize(result: JudgeResult): Canonical {
  if (result.parseError || result.rawLabel === null) {
    throw new Error(
      result.caseId + ' ' + result.judgeVersion + ': invalid judge result'
    );
  }
  const decision = mappings[result.judgeVersion][result.rawLabel];
  if (!decision) {
    throw new Error(
      result.caseId + ' ' + result.judgeVersion +
      ': unmapped label ' + result.rawLabel
    );
  }
  return decision;
}

function compare(results: JudgeResult[]): object[] {
  const byCase = new Map<string, JudgeResult[]>();

  for (const result of results) {
    const group = byCase.get(result.caseId) ?? [];
    if (group.some(item => item.judgeVersion === result.judgeVersion)) {
      throw new Error(result.caseId + ': duplicate judge version');
    }
    group.push(result);
    byCase.set(result.caseId, group);
  }

  const transitions: object[] = [];
  for (const [caseId, group] of byCase) {
    if (group.length !== 2) throw new Error(caseId + ': incomplete bridge pair');
    const oldResult = group.find(item => item.judgeVersion === 'old-v7');
    const newResult = group.find(item => item.judgeVersion === 'new-v1');
    if (!oldResult || !newResult) throw new Error(caseId + ': wrong judge pair');

    if (oldResult.outputHash !== newResult.outputHash) {
      throw new Error(caseId + ': evaluated outputs differ');
    }
    if (oldResult.evidenceHash !== newResult.evidenceHash) {
      throw new Error(caseId + ': evidence bundles differ');
    }

    const oldDecision = canonicalize(oldResult);
    const newDecision = canonicalize(newResult);
    transitions.push({
      caseId,
      oldDecision,
      newDecision,
      changed: oldDecision !== newDecision,
    });
  }
  return transitions;
}

const input = JSON.parse(process.argv[2] ?? '[]') as JudgeResult[];
console.log(JSON.stringify(compare(input), null, 2));

Run validation before calculating an agreement rate. Otherwise invalid results enter the denominator or are silently coerced into decisions. A good bridge report begins with execution completeness, pairing integrity, and mapping coverage.

A stale cached judgment can impersonate semantic drift

One more failure produces the familiar pass to fail transition. The new judge may truly read the frozen answer and decide it is unsupported. Alternatively, a cache built for the old single-judge pipeline may use only case ID as its key. During migration it returns a prior fail label created for another output version, rubric version, or judge version. The bridge row still says that the new canonical decision is fail, so the transition matrix alone cannot separate these roots.

The evidence is judgment provenance. A healthy candidate result ties the raw judge response to the exact evaluated-output hash, ordered evidence hash, judge version, prompt or rubric version, parser version, and time of creation. A true semantic reversal has valid matching provenance and a raw response whose stated criterion supports the changed canonical decision. A stale-cache failure has no candidate call for that artifact or points to provenance created under a different content or pipeline identity.

The most misleading implementation recomputes current hashes after a cache lookup and wraps the cached label in a fresh result record. Its bridge row appears internally consistent because the wrapper contains today’s output and evidence hashes. Those hashes prove what the comparison process had in hand, not what the judge saw when the cached label was created. Preserve a content identity with the cached judgment at creation time and compare that identity before accepting the hit.

Diagnostic output should show, for each side, execution state, cache state, judge-input content identity, raw-response reference, raw label, parse state, mapping version, and canonical decision. A healthy cache hit has a stored input identity equal to the requested input identity and a pipeline identity permitted by the migration plan. A healthy fresh result has a completed execution record and the same identity chain. A broken cache hit differs on at least one required component or lacks creation provenance. A cache miss followed by no completed execution is missing, not a semantic fail.

This distinction changes both owner and remedy. A valid raw unsupported judgment on the correct bytes belongs to rubric calibration or product-policy review. A stale result belongs to cache identity and evaluator infrastructure. Rewording the judge prompt in response to the latter can make a sound rubric worse while leaving old labels available for reuse.

Content-addressed caching has a concrete cost. Adding output, evidence, judge, rubric, parser, and mapping identities to the key reduces hit rate whenever any part changes. It also increases metadata storage and requires deliberate invalidation rules. That cost is the price of knowing that a saved inference applies to the current decision. A broad key with a high hit rate is not an optimization if it silently joins judgments to the wrong artifacts.

The handoff should include the bridge case and versions, requested and stored content identities, cache lookup outcome, cached-result creation provenance, raw-response reference, parse and mapping identities, and a reproduction that shows whether the candidate client was called. The evaluator-platform owner repairs cache semantics. The judge-prompt or policy owner reviews a genuine semantic reversal only after that provenance passes.

Read transition patterns before setting a cutover gate

A single agreement percentage collapses direction. Pass-to-fail movement can block releases that previously passed. Fail-to-pass movement can admit regressions the old judge caught. Review-to-pass reduces human workload but may remove useful uncertainty. Report a transition matrix with the old decision on rows and the new decision on columns.

Slice the matrix by criterion and risk. If nearly all movement comes from verbosity preferences, the team may accept it. If movement clusters on prompt injection or approval bypass, read every case. A migration decision is not a vote across unrelated criteria.

The Python script below consumes a paired CSV with canonical old_decision and new_decision fields. It prints transition counts and writes changed cases in deterministic order. The optional blocking slice list makes release-critical movement explicit. It does not declare a universal tolerance.

from __future__ import annotations

import argparse
import csv
from collections import Counter
from pathlib import Path


VALID = {'pass', 'fail', 'review'}


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('bridge_csv', type=Path)
    parser.add_argument('--blocking-slice', action='append', default=[])
    args = parser.parse_args()

    with args.bridge_csv.open(newline='', encoding='utf-8') as handle:
        rows = list(csv.DictReader(handle))

    transitions: Counter[tuple[str, str]] = Counter()
    changed: list[dict[str, str]] = []
    blocking_changes: list[str] = []

    for row in rows:
        case_id = row['case_id'].strip()
        old = row['old_decision'].strip()
        new = row['new_decision'].strip()
        if old not in VALID or new not in VALID:
            raise ValueError(case_id + ': invalid canonical decision')

        transitions[(old, new)] += 1
        if old != new:
            changed.append(row)
            if row['slice'] in set(args.blocking_slice):
                blocking_changes.append(case_id)

    print('transition matrix entries')
    for transition, count in sorted(transitions.items()):
        print(transition[0] + ' -> ' + transition[1] + ':', count)

    print('changed cases')
    for row in sorted(changed, key=lambda item: (item['slice'], item['case_id'])):
        print(
            row['slice'],
            row['case_id'],
            row['old_decision'] + ' -> ' + row['new_decision'],
        )

    print('blocking_slice_changes:', blocking_changes)
    raise SystemExit(1 if blocking_changes else 0)

Use human adjudication strategically. Review every transition that changes a blocking decision. Review a stratified sample of nonblocking transitions and agreements. Reviewing agreements matters because both judges can share an error. Include known gold anchors to estimate that blind spot, and allow adjudicators to mark the rubric itself as insufficient.

Do not fit the new judge to the bridge until every disagreement disappears. That creates an evaluator specialized to one finite set. Keep a holdout bridge partition that is not used to revise prompts or mappings. After calibration, run once on the holdout and report the result without another round of tuning.

Illustrative transition counts might show 420 pass-to-pass cases, 28 fail-to-fail cases, 17 pass-to-review cases, six review-to-fail cases, and two pass-to-fail cases. These are illustrative figures, not measurements. The two pass-to-fail rows could be more important than the hundreds of agreements if they concern a required refusal.

Reasons should be coded without discarding free text. Useful reason codes include rubric boundary, evidence interpretation, output-order sensitivity, parser invalid, mapping gap, judge nondeterminism, and gold-label dispute. The free-text rationale points to the actual span or criterion. Codes support trend analysis; rationales support engineering action.

Repeat a small subset to inspect stability. If the new judge changes its own decision on fixed inputs, distinguish a stable difference from stochastic variance. Repetition consumes judge calls and can complicate denominators, so predeclare how repeated labels are summarized. Never keep retrying only the rows that fail until they agree with the old judge.

Cut over without rewriting historical results

The safest migration runs old and new judges in parallel for a bounded period. Both consume the same immutable outputs, and neither silently edits past records. The old judge remains the official release signal while the bridge is calibrated. The candidate produces a shadow decision and disagreement artifact.

For an existing suite, land identity and storage changes before enabling the second judge. A legacy result table often treats case ID as unique, a cache may omit evaluator version, and a dashboard may assume one label per case. The first dual run can therefore overwrite the official result, reuse the wrong label, or count two judgments as two product cases. Backfill version identity onto the immutable baseline where provenance supports it, and label unverifiable history as legacy rather than manufacturing details.

Next, make the result store accept a case-and-judge pair while the old writer remains authoritative. Add parser contract tests and explicit invalid states before the candidate client writes production bridge data. Then run a small public smoke partition that contains valid agreements, intentional disagreements, an unmapped label, a missing pair, and mismatched evidence. The report must classify each condition correctly before the full bridge consumes inference or adjudication time.

The rollout is working when baseline rows never change, every bridge case has exactly the intended judge versions, raw and parsed states remain separately queryable, cache provenance matches requested content, invalid counts are visible, and saved reports reproduce without new model calls. Seeded mismatches should fail integrity checks, while intentional rubric transitions should reach adjudication rather than appearing as infrastructure errors.

What breaks first operationally is often downstream reporting. Saved queries may mix evaluator versions, alert thresholds may interpret a categorical mapping as an old scalar, and data consumers may silently select the latest row without defining “latest.” The data or analytics owner must inventory those consumers before cutover. The evaluator-platform owner owns execution, storage identity, caching, and parser status. The rubric and policy owner approves meaning changes. The suite owner owns bridge coverage. The release owner authorizes cutover and rollback.

Their handoff packet should contain the frozen case manifest, artifact and pipeline identities, execution-completeness report, cache-hit audit, transition matrices by risk slice, adjudication decisions, invalid and missing counts, threshold changes, downstream consumer inventory, operating-cost change, holdout status, and rollback path. Each intentional transition should name the policy decision it represents. Each unresolved infrastructure issue should name the component owner rather than entering a general “judge disagreement” queue.

Put migration identity into configuration. The manifest below is project-owned and illustrative. It records what must be frozen and which outcomes require review.

migration_id: groundedness-judge-2026-08
case_set_version: bridge-v4
canonical_labels:
  - pass
  - fail
  - review
judges:
  baseline:
    version: old-v7
    prompt_version: grounded-rubric-6
    parser_version: score-parser-3
  candidate:
    version: new-v1
    prompt_version: grounded-rubric-7
    parser_version: label-parser-1
integrity:
  require_equal_output_hash: true
  require_equal_evidence_hash: true
  fail_on_missing_pair: true
  fail_on_unmapped_label: true
review:
  adjudicate_all_blocking_transitions: true
  preserve_original_results: true
  holdout_partition: bridge-holdout-v1
cutover:
  keep_baseline_results_immutable: true
  publish_transition_matrix: true
  rollback_judge_version: old-v7

After pairing integrity passes, calibrate the canonical mapping and candidate rubric on the development partition. Ask a policy owner to approve intentional boundary changes. Run the frozen candidate on the holdout. Publish invalid counts, transition matrices, slices, adjudication outcomes, stability replays, and operating cost.

Change downstream thresholds at the same explicit cutover. A dashboard that swaps judges but keeps a threshold whose units came from the old score is internally inconsistent. Version saved queries and alerts if they interpret judge output. Mark the time series with the migration boundary rather than drawing one continuous line.

Keep dual-write capability through the first product release evaluated by the new judge. If an incident appears, replay stored outputs through both versions. A fast rollback requires retained prompts, parser code, and routing, not only the old model name.

Migration costs are unavoidable:

• Dual judging increases inference expense and wall-clock time.
• Immutable raw results consume storage and may contain sensitive output that needs retention controls.
• Human adjudication creates a specialist queue.
• A broad bridge improves coverage but becomes expensive to rerun.
• Strict parser validation causes visible invalid runs that a permissive legacy system may have hidden.
• Versioned dashboards complicate trend reporting while preserving its honesty.

Reduce cost by stratifying intelligently, caching fixed judge results under content hashes where policy permits, and separating a small smoke bridge from the full cutover set. Do not reduce cost by dropping the exact cases most likely to change a release decision.

Do not build a bridge for the wrong kind of change

Avoid using a bridge to declare the new judge objectively better merely because it agrees with the old one. High agreement establishes continuity. Accuracy requires independent contracts and reviewed labels. A new evaluator that copies a biased baseline perfectly has migrated the bias successfully.

Do not change the product outputs, judge, rubric, evidence packaging, and release threshold in one experiment. The team will know that results moved and little else. Freeze product outputs, migrate the judge layer, then evaluate new product behavior under the accepted layer.

Skip scalar score conversion when categories carry different meanings. No linear formula can repair a conceptual mismatch between “helpfulness” and “policy-compliant usefulness.” Preserve both results and redesign the canonical decision with product owners.

Do not overwrite historical judge labels after replay. Recomputed results belong to a new evaluator version. Keeping both allows audits, incident reconstruction, and honest charts. Storage pressure is a retention-design problem, not permission to erase provenance.

Avoid a large random bridge made entirely from easy passes. It will generate impressive agreement and weak release protection. Spend coverage on known failures, rare critical slices, rubric edges, parser edges, long context, and behaviors introduced since the old calibration set was built.

Finally, do not cut over while invalid outputs are being counted as product failures. Until missing pairs, parse errors, mapping gaps, and evidence mismatches are separated from semantic decisions, the migration report cannot tell whether the new judge is stricter, broken, or simply reviewing a different artifact.

The bridge technique does not catch an error that both judges share and the reviewed cases omit. If both evaluators reward a polished but unsupported answer, their perfect agreement strengthens continuity while saying nothing about correctness. A frozen bridge also cannot reveal a new behavior outside its case population. Deterministic checks, independently reviewed anchors, incident-derived cases, and post-cutover monitoring remain necessary controls.