Do not promote the challenger on one flattering score

The candidate model wins the dashboard, yet it exposes an account identifier on one support case. Averages call that an improvement. A release engineer should call it a blocked rollout until the team knows which cases moved, why they moved, and whether the comparison was fair.

Make the comparison answer a release question

A champion is simply the version currently trusted in production. The challenger might change the base model, system prompt, retrieval settings, tool policy, decoding configuration, or several of those at once. That last option is tempting and usually hard to diagnose. If the challenger loses, a six-part change gives the team six suspects and no quick rollback lesson.

Write the release question before collecting scores. “Is version B better?” is too vague. A useful question names the population, protected behavior, acceptable cost, and action. For example: “Can the new support assistant replace the current one for English billing conversations without increasing policy violations, while keeping latency inside the existing service objective?” That sentence tells QA which cases belong in the dataset and which failures have veto power.

Both variants must see the same case version. Pairing matters because prompts vary enormously in difficulty. If the champion receives routine password resets while the challenger receives disputed refunds, the difference is traffic composition, not evidence of model quality. Assign a stable case ID, freeze the user input and permitted context, then record one result from each variant against that ID. Any row missing one side is incomplete rather than a loss.

The evaluation unit must match the product promise. A single assistant message is enough for a rewrite feature. An agent that gathers details, calls a tool, asks for approval, and confirms completion needs a trajectory-level case. Scoring only its final prose can reward a response that sounds correct after the wrong tool was called. Store the observable events required for the decision: tool name, normalized arguments, approval state, terminal status, citations, or whatever the contract actually exposes.

Separate gates from preferences. A gate represents behavior the team has decided not to trade away, such as leaking a secret, inventing an order status, bypassing approval, or emitting an invalid response schema. Preferences compare acceptable outputs on usefulness, clarity, tone, or completeness. A challenger that wins nine preference rows and fails one gate has not won ten comparable contests. It has produced one release blocker and nine pieces of secondary evidence.

Do not turn every qualitative judgment into an apparently precise decimal. If reviewers can reliably choose “champion,” “challenger,” “tie,” or “cannot judge,” preserve those labels. The reason is part of the result. A scalar is useful only when its rubric gives adjacent values distinct, repeatable meanings. Illustrative scores such as 0.81 versus 0.78 advertise more certainty than the review process owns when the rubric cannot explain the difference.

Stratification prevents a large easy slice from burying a small risky one. Keep results by capability, locale, customer tier, input length, tool path, and risk class when those dimensions affect behavior. The overall rate can still be reported, but promotion should be evaluated against slice rules. A huge collection of harmless FAQ cases cannot compensate for a regression in the smaller refund-authorization slice.

Build paired evidence before counting wins

The smallest durable record contains the case version, variant version, raw output reference, deterministic check results, reviewer label, and review rationale. Keep generation separate from judging. When outputs are regenerated during scoring, a retry or model update can change the artifact under review. Immutable result files let another engineer rerun the comparator without spending tokens or contacting a provider.

The TypeScript below performs the first useful diagnostic. It rejects duplicate results, missing pairs, fixture-version drift, and gate failures before it counts preferences. The illustrative rows are deliberately small so the behavior is easy to inspect. Replace them with exported results from your own harness.

type Side = 'champion' | 'challenger';
type Preference = Side | 'tie' | 'review';

type Result = {
  caseId: string;
  fixtureVersion: string;
  side: Side;
  blockingChecks: Record<string, boolean>;
};

const results: Result[] = [
  {
    caseId: 'billing-001',
    fixtureVersion: 'billing-v4',
    side: 'champion',
    blockingChecks: { groundedOrderStatus: true, piiSafe: true },
  },
  {
    caseId: 'billing-001',
    fixtureVersion: 'billing-v4',
    side: 'challenger',
    blockingChecks: { groundedOrderStatus: true, piiSafe: true },
  },
  {
    caseId: 'refund-007',
    fixtureVersion: 'refund-v2',
    side: 'champion',
    blockingChecks: { approvalRequired: true, piiSafe: true },
  },
  {
    caseId: 'refund-007',
    fixtureVersion: 'refund-v2',
    side: 'challenger',
    blockingChecks: { approvalRequired: false, piiSafe: true },
  },
];

const preferences: Record<string, Preference> = {
  'billing-001': 'challenger',
  'refund-007': 'challenger',
};

const requiredChecks: Record<string, string[]> = {
  'billing-001': ['groundedOrderStatus', 'piiSafe'],
  'refund-007': ['approvalRequired', 'piiSafe'],
};

const byCase = new Map<string, Map<Side, Result>>();

for (const result of results) {
  const pair = byCase.get(result.caseId) ?? new Map<Side, Result>();
  if (pair.has(result.side)) {
    throw new Error('Duplicate ' + result.side + ' result for ' + result.caseId);
  }
  pair.set(result.side, result);
  byCase.set(result.caseId, pair);
}

let challengerWins = 0;
let championWins = 0;
let ties = 0;
let reviews = 0;
const blockers: string[] = [];

for (const [caseId, pair] of byCase) {
  const champion = pair.get('champion');
  const challenger = pair.get('challenger');

  if (!champion || !challenger) {
    throw new Error('Incomplete pair for ' + caseId);
  }
  if (champion.fixtureVersion !== challenger.fixtureVersion) {
    throw new Error('Fixture mismatch for ' + caseId);
  }

  const checks = requiredChecks[caseId];
  if (!checks) throw new Error('Missing check policy for ' + caseId);
  for (const check of checks) {
    if (challenger.blockingChecks[check] !== true) {
      blockers.push(caseId + ': ' + check);
    }
  }

  const preference = preferences[caseId];
  if (!preference) throw new Error('Missing preference for ' + caseId);
  if (preference === 'challenger') challengerWins += 1;
  if (preference === 'champion') championWins += 1;
  if (preference === 'tie') ties += 1;
  if (preference === 'review') reviews += 1;
}

console.log({ challengerWins, championWins, ties, reviews, blockers });
if (blockers.length > 0) process.exitCode = 1;

Notice what the code refuses to do. It does not award the challenger a preference win merely because its own score is higher than a score generated under another rubric. It does not treat an absent champion output as a challenger victory. It also reports the failed check by case instead of reducing it to zero inside an average.

A real dataset needs more context than the comparator consumes. Keep scenario ownership, data provenance, collection consent where production traces are involved, expected tool constraints, and the rubric version beside each case. Those fields make a future dispute answerable. They also stop a quiet fixture edit from rewriting history.

Run a pairing audit before reviewing model quality. Many apparent regressions are data-join failures: a truncated case ID, two retries recorded as separate cases, or an old champion output joined to a revised fixture. The following script reads a CSV export and reports those conditions without calling either model.

from __future__ import annotations

import argparse
import csv
from collections import Counter, defaultdict
from pathlib import Path


def audit(path: Path) -> list[str]:
    errors: list[str] = []
    rows_by_case: dict[str, list[dict[str, str]]] = defaultdict(list)

    with path.open(newline='', encoding='utf-8') as handle:
        for line_number, row in enumerate(csv.DictReader(handle), start=2):
            case_id = (row.get('case_id') or '').strip()
            side = (row.get('side') or '').strip()
            version = (row.get('fixture_version') or '').strip()

            if not case_id:
                errors.append('line ' + str(line_number) + ': blank case_id')
                continue
            if side not in {'champion', 'challenger'}:
                errors.append(case_id + ': invalid side ' + repr(side))
                continue
            if not version:
                errors.append(case_id + ': blank fixture_version')
            rows_by_case[case_id].append(row)

    for case_id, rows in sorted(rows_by_case.items()):
        sides = Counter(row['side'].strip() for row in rows)
        if sides != Counter({'champion': 1, 'challenger': 1}):
            errors.append(case_id + ': expected one result per side, got ' + str(dict(sides)))

        versions = {row['fixture_version'].strip() for row in rows}
        if len(versions) != 1:
            errors.append(case_id + ': mixed fixture versions ' + str(sorted(versions)))

    return errors


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('results_csv', type=Path)
    args = parser.parse_args()

    problems = audit(args.results_csv)
    for problem in problems:
        print(problem)
    raise SystemExit(1 if problems else 0)

A clean pairing audit does not prove the challenger is better. It proves only that the comparison has two valid sides. That distinction is valuable in CI because it prevents teams from debating judge output when the input table itself is malformed.

Read the losses instead of trusting the aggregate

Consider a support migration with three slices. Billing questions improve because the new model follows long context better. Refund cases regress because a new prompt moved the approval instruction below retrieved text. Spanish cases are mostly ties, but reviewers mark several “cannot judge” because the translated rubric uses a different definition of completeness. One overall win rate blends a real gain, a product blocker, and invalid review work.

The first signal to inspect is the case-level transition. Build a table with champion outcome, challenger outcome, gate changes, preference, rationale, latency, and cost. Sort by changed gates first, then by judge disagreement, then by large metric movement. Reading twenty transition rows often teaches more than staring at a percentage produced from all two thousand.

For a failed gate, inspect the earliest artifact that could establish it. If the refund approval check fails, look at the normalized tool-call record and approval event, not only the final response. If citation grounding fails, compare the cited document IDs with the retrieval snapshot stored for that case. If schema validity fails, preserve the parser error and raw output reference. The goal is to identify whether generation, instrumentation, or evaluation broke first.

Two approval failures that require different fixes

A challenger row can report a failed approval check even when the challenger respected approval. One root cause is the product regression the check was designed to catch: the agent calls the refund tool before any valid approval event. A second root cause is an observation defect: a trace normalizer drops the approval event after its schema changes, and the deterministic check converts missing evidence to false. The dashboard looks the same in both cases because it retains only the final Boolean.

Read the check result beside evidence completeness and the ordered trajectory. A healthy case has a complete trajectory, an approval event tied to the relevant action, and a tool call ordered after that event. A true challenger regression has a complete trajectory too, but the tool call appears without the required prior approval. The observation defect has an incomplete or unresolved trajectory. The immutable raw artifact contains the approval, while the normalized artifact supplied to the check does not.

The most misleading value is false when the checker uses it for both “requirement violated” and “required event unavailable.” Those states need different result classes. A violation is valid release evidence. Missing evidence makes the evaluation invalid and should stop the comparison from declaring either side a winner. If the checker cannot represent invalidity yet, repair that contract before using its Boolean as a blocking signal.

The diagnostic output should therefore show the case and fixture versions, side, check name, check result, artifact-completeness status, raw artifact reference, normalized artifact reference, and the first causal event. On a healthy row, completeness is valid and the check passes. On a product regression, completeness is valid and the first relevant tool action lacks its prerequisite. On an evaluator failure, completeness is invalid or the raw and normalized event counts disagree. A count of zero approval events is not proof of model behavior until the artifact is known to be complete.

Failing closed on incomplete evidence has a concrete operational cost. A telemetry deployment can block every model promotion even when product behavior is safe. Allowing missing evidence to pass avoids that availability cost but creates a direct path for unsafe actions to escape a gate. The practical design is to keep “invalid evaluation” distinct, route it urgently to the telemetry owner, and preserve the champion in production until valid paired evidence exists.

Ownership follows the earliest divergence. The agent or prompt owner fixes a complete trajectory containing an unapproved call. The instrumentation owner fixes an approval present in the raw run but absent from the exported trajectory. The evaluation owner fixes a checker that collapses missing into false or joins the wrong artifact version. A handoff needs the case and fixture versions, both variant identities, raw and normalized artifact hashes, the ordered prerequisite and action events, completeness status, checker version, and a minimal replay. A screenshot of a red gate omits the evidence needed to choose among those owners.

A preference reversal needs different evidence. Show reviewers both outputs without side labels when possible, randomize their display order, and capture the rubric criterion behind the choice. Blinding reduces brand and recency bias. Order randomization catches a reviewer who tends to prefer the first answer. Neither technique guarantees objectivity, but both make obvious procedural bias measurable.

Ties deserve their own count. They can mean both answers are acceptable, both are equally poor, or the rubric cannot distinguish them. Add a reason code if those interpretations affect the decision. “Both acceptable” supports a cost-based choice. “Both fail” indicates a shared product problem. “Rubric unclear” belongs in calibration, not in either model’s loss column.

Review cases are not ties. A missing language skill, corrupted trace, ambiguous source document, or policy question outside the rubric makes the row unresolved. Exclude it from a preference denominator only with an explicit reason, then report how many rows were excluded in every slice. Otherwise a team can improve a rate by quietly moving difficult challenger losses into an unreported bucket.

Latency and cost should remain separate from response quality. A release may choose a slightly weaker but much cheaper model for a low-risk path, or pay more for a clear safety improvement. That is a product decision, not a reason to combine milliseconds, dollars, and rubric labels into one mysterious score. Present a small decision table so the trade-off stays visible.

Illustrative decision data might look like this:

These figures are illustrative, not measurements from an experiment. They show why the apparent 44-to-25 preference advantage cannot authorize promotion: two blocking regressions remain, and the Spanish review backlog is too large to describe that slice confidently.

Distinguish model drift from evaluation drift

A challenger can look worse even when its product behavior is unchanged. The judge prompt may have changed, label order may be reversed, a parser may map “B” to champion after output order was randomized, or a retrieval fixture may have been refreshed for only one side. These failures often print the same symptom: a sudden change in the win rate.

Start with deterministic integrity checks. Confirm the same case IDs and fixture hashes reached both variants. Confirm the judge received the intended output order and that its returned label was mapped back to a side correctly. Confirm rubric version, judge version, and parsing code are identical across the compared batch. Only then investigate generation quality.

One useful diagnostic is a swap test. Take a sample of fixed output pairs, reverse their display order, and rerun the judging step. After mapping labels back to the underlying variants, the decision should usually remain stable. A high reversal rate points to positional sensitivity or a mapping bug. It does not, by itself, prove which earlier judgment was correct.

Another is a replay test. Feed previously stored champion and challenger outputs into the current comparator. If past decisions move without any generation change, the evaluation layer changed. That may be intentional after rubric calibration, but it means historical and current win rates are not directly comparable. Version the judge and regenerate a bridge report rather than overwriting old labels.

The Python program below calculates paired preference movement across two evaluation exports. It reports changed decisions by case and slice. It deliberately avoids a statistical verdict because the acceptable amount and type of movement are release-policy choices.

from __future__ import annotations

import argparse
import csv
from collections import Counter
from pathlib import Path


VALID = {'champion', 'challenger', 'tie', 'review'}


def load(path: Path) -> dict[str, dict[str, str]]:
    with path.open(newline='', encoding='utf-8') as handle:
        rows = list(csv.DictReader(handle))

    result: dict[str, dict[str, str]] = {}
    for row in rows:
        case_id = row['case_id'].strip()
        decision = row['decision'].strip()
        if decision not in VALID:
            raise ValueError(path.name + ': invalid decision for ' + case_id)
        if case_id in result:
            raise ValueError(path.name + ': duplicate case ' + case_id)
        result[case_id] = row
    return result


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('baseline', type=Path)
    parser.add_argument('candidate', type=Path)
    args = parser.parse_args()

    baseline = load(args.baseline)
    candidate = load(args.candidate)
    common = sorted(baseline.keys() & candidate.keys())
    missing = sorted(baseline.keys() ^ candidate.keys())

    transitions: Counter[tuple[str, str]] = Counter()
    changed_by_slice: Counter[str] = Counter()

    for case_id in common:
        before = baseline[case_id]['decision'].strip()
        after = candidate[case_id]['decision'].strip()
        transitions[(before, after)] += 1
        if before != after:
            slice_name = candidate[case_id]['slice'].strip()
            changed_by_slice[slice_name] += 1
            print(case_id + ': ' + before + ' -> ' + after)

    print('transitions:', dict(sorted(transitions.items())))
    print('changed_by_slice:', dict(sorted(changed_by_slice.items())))
    print('unpaired_case_ids:', missing)
    raise SystemExit(1 if missing else 0)

Suppose the judge replay changes only cases containing terse but correct answers. Read those cases together. The new rubric may reward explanation length even though the product requirement values brevity. Suppose changes cluster by output position instead. That implicates judge presentation or label mapping. Suppose stored outputs are stable, judge labels are stable, but blocking checks flip. The deterministic evaluator or its inputs changed. The same top-line movement has three different owners.

Near-misses also occur when traffic shifts. Production shadow data collected on a promotion weekend may contain more refund questions than the earlier champion sample. Pairing within the shadow run helps, but comparing that aggregate with last month’s offline benchmark still confounds model movement with population movement. Report both on the current paired population and keep the historical series clearly labeled.

Gate promotion with uncertainty and operational cost visible

A paired result is stronger than two independent averages because each case acts as its own control. It still has uncertainty. A narrow dataset can miss behaviors, reviewers can disagree, and stochastic generation can produce another valid outcome on the next run. Treat the comparison as evidence for a scoped release, not proof that one model is universally better.

For preference labels, report the win, loss, tie, and review counts before using a derived rate. If the team applies a confidence interval or a paired test, predeclare the method and decision boundary. Do not try several tests and publish only the one that promotes the candidate. Small risky slices may need a hard “no observed blockers” rule plus manual review rather than a noisy significance calculation.

Repeated generations answer a separate question: stability. Running each case several times can reveal a challenger that alternates between excellent and unsafe behavior. It also multiplies inference cost and review volume. Use repetition where randomness changes the contract, such as tool selection or refusal boundaries. A deterministic formatter with temperature controlled by the application may not justify the same spend.

The release policy should be data, not tribal memory. The YAML below is an application-owned manifest, not a provider setting. A small gate script can read it and the result exports. Naming the fields makes review easier and keeps a policy change visible in code review.

policy_version: champion-challenger-v3
comparison:
  require_paired_case_ids: true
  require_same_fixture_version: true
  preserve_ties: true
  preserve_review_cases: true
blocking_checks:
  - grounded_order_status
  - pii_safe
  - approval_required
slices:
  billing:
    max_blocking_regressions: 0
    minimum_reviewed_pairs: 60
  refund_approval:
    max_blocking_regressions: 0
    minimum_reviewed_pairs: 20
  spanish_support:
    max_blocking_regressions: 0
    minimum_reviewed_pairs: 30
operations:
  maximum_p95_latency_increase_ms: 250
  maximum_cost_ratio: 1.20
decision:
  on_missing_pair: fail
  on_unresolved_review: hold

The numbers in this sample policy are illustrative. Copying them would be cargo culting. Derive minimum coverage from the product’s failure history and derive operational limits from its service and budget constraints.

CI should fail loudly on corrupt evidence and write a review artifact for legitimate model differences. A nonzero exit for a gate regression is appropriate. A preference loss might only require approval if the slice remains within its declared tolerance. Keep those outcomes distinct so engineers do not learn to ignore every red pipeline.

An existing single-model suite needs pairing infrastructure before it needs a promotion threshold. First freeze the current case manifest, fixture versions, output artifacts, and gate policy so the champion baseline cannot move during installation. Next make artifact storage side-aware. File names, cache keys, and database uniqueness rules that previously used only case ID often overwrite one side or return the champion response as the challenger response. Prove that two distinct variant records survive for every case and that the pairing audit rejects an intentionally missing side.

Then run the challenger in observation mode and publish case transitions without changing the official release signal. Land deterministic integrity checks before adding human or model preferences, because reviewers should not spend time on mismatched fixtures or duplicated outputs. Add blocking gate comparison next. Preference review and uncertainty rules come after the team can trace every blocking transition to an artifact. Only then let the candidate report participate in CI promotion.

What breaks first is often capacity rather than quality. Paired generation can double calls for cases whose champion outputs were not retained, and blind review can double the number of rendered artifacts. Measure queue time, rate limiting, storage growth, and reviewer backlog during observation mode. Reusing immutable champion outputs reduces spend, but it means the comparison no longer measures fresh champion variance. Label that choice in the report rather than presenting cached and fresh generations as equivalent.

The rollout is working when missing and duplicate pairs are zero, fixture hashes match across sides, invalid evaluations are separate from losses, blocker transitions reproduce from stored artifacts, and a deliberately seeded gate failure stops promotion without corrupting preference totals. The suite owner approves slice coverage, the model-platform owner owns variant execution and attribution, the evaluation owner owns pairing and check integrity, and the product risk owner approves intentional boundary changes. Their cutover packet should include coverage by slice, unresolved review rows, operating-cost movement, known limitations, rollback identity, and the exact cohort authorized for release.

A staged rollout reduces the blast radius after the offline gate. Start with internal or synthetic traffic, then shadow real requests, then expose a small eligible cohort behind a kill switch. Watch the same guardrails used offline, plus operational signals unavailable in the fixture set. Record which model served each request and which configuration version was active. Without attribution, an incident cannot be tied back to the candidate.

Rollback readiness has a cost. Maintaining two routable versions, compatible response schemas, and comparable telemetry takes engineering work. That cost is justified for a material model or prompt migration. It may not be justified for a spelling change in a noncritical instruction. Scale the rollout machinery to the plausible harm, not to the novelty of using an LLM.

The main trade-offs are concrete:

• Pairing improves causal clarity but doubles generation work when champion outputs are not already stored.
• Human blind review captures qualities that deterministic checks miss, but it adds queue time and calibration work.
• More slices expose local regressions, but each slice needs enough cases and an owner.
• Repeated runs reveal variance, while increasing inference cost and potentially multiplying sensitive trace storage.
• Hard gates protect nonnegotiable behavior, but a brittle gate can block a safe release because its detector changed.
• Shadow traffic improves realism, while introducing privacy review, retention limits, and production-load concerns.

The right response is not to remove the costs from the report. Show them beside the quality evidence so the approver knows what has and has not been tested.

An offline paired comparison does not catch a failure created only by live routing, load, or mutable dependencies. Both variants can pass fixed retrieval snapshots while the challenger times out under production concurrency or receives a different document from a live index. It can also miss a rare prompt shape absent from the paired population. The gate supports the scoped release decision represented by its cases. Shadow and staged production signals still own those runtime risks.

Know when a champion/challenger gate is the wrong tool

Do not run a formal promotion contest when the two systems serve different contracts. A concise extraction model and a conversational support model cannot be reduced to one winner unless a shared product task genuinely exists. Give each its own acceptance criteria.

Avoid a direct comparison when the challenger changes the surrounding workflow so much that paired inputs no longer mean the same thing. If one version retrieves documents and the other asks a human, compare end-to-end outcomes under a redesigned evaluation. Pretending their intermediate responses are equivalent will reward the easier path rather than the better product.

Do not use the production champion as a gold answer. Existing behavior may be tolerated, outdated, or wrong. Champion output is a control for movement, not ground truth. Deterministic contracts, reviewed references, user outcomes, and policy rules remain independent evidence.

Skip an automated preference judge when the rubric is unsettled or the domain requires expertise the judge has not demonstrated. A judge can accelerate a calibrated process. It cannot decide whether the process values legal completeness, clinical caution, cultural tone, or terse operator instructions correctly. Route those boundaries to qualified reviewers and use their disagreements to improve the rubric.

Do not collect live prompts merely because shadow data sounds realistic. If synthetic or consented fixtures cover the release question, they may be safer and easier to reproduce. Production traces can contain account data, secrets, and contractual material. Their diagnostic value must justify access, redaction, retention, and deletion controls.

Finally, do not block a minor release on an opaque composite score that nobody can trace to cases. A gate earns authority by making a failure inspectable. If an engineer cannot move from the red status to the exact pair, check, artifact, and owner, the comparison system itself needs repair before it decides which model reaches users.