Your AI release average is green, but the risky slice is not

Candidate B matches Candidate A on the headline score, so the release report turns green. Then someone opens the language breakdown and finds that account-recovery answers regressed for Spanish users. The average was accurate, but the release decision was wrong.

A scorecard is useful only when it preserves the risks that an aggregate can hide. It must connect each gate to cases, versions, and failed evidence. Otherwise it is a dashboard decoration with the authority of a test.

Make the scorecard answer a release question

An evaluation run produces observations. A release scorecard applies policy to those observations. Keeping those responsibilities separate makes the result easier to audit and prevents a plotting choice from quietly becoming a release rule.

The evaluation layer should record what happened for each case: whether execution completed, what each deterministic check observed, what each semantic grader decided, and where the supporting trace lives. The scorecard layer groups those rows into criteria and slices. The policy layer decides which changes are acceptable. A release decision is the final output of all three, not a raw model-generated grade.

Every scorecard should name the decision it supports. “May Candidate B replace Candidate A for customer-support traffic?” is specific. “How good is the model?” is not. The first question implies a baseline, a traffic context, a set of relevant risks, and a candidate. The second invites an impressive number with no operational meaning.

Freeze comparison inputs before looking at the result. At minimum, record:

• candidate build, model, and prompt identifiers;
• baseline identifiers;
• dataset version and exact case IDs;
• grader and rubric versions;
• tool or retrieval configuration relevant to the run;
• run time and execution policy, including retries;
• invalid, skipped, and timed-out row counts;
• scorecard-policy version.

This provenance is not administrative clutter. If the candidate and baseline used different case sets, an apparent improvement may be a composition change. If a semantic grader changed between runs, a score delta may be judge drift. If retries were enabled for only one side, operational reliability is no longer comparable.

Criteria should map to product risks, not whichever metrics a framework exposes by default. A support assistant might have criteria for resolution correctness, unsupported action claims, policy compliance, required disclosures, language quality, and latency. A code assistant needs different categories, such as compilation, API correctness, repository fit, and unsafe change behavior. The scorecard vocabulary should make sense to the people who approve the release.

Keep critical criteria independent. If an unsafe-action rate must not regress, state that rule directly. Do not give safety a weight in a composite average and hope the weight is large enough. Weighted averages are especially dangerous when the units differ. Combining a binary policy check, a five-point style grade, and latency in milliseconds produces arithmetic, not meaning.

An overall indicator can still help readers find the report. Treat it as a summary status derived from named rules: green when all blockers pass, amber when only review rules fail, red when any blocker fails, and invalid when evidence is incomplete. That status remains interpretable because an engineer can trace it to the rule that set it.

Keep criteria, slices, and evidence in the data model

A flat object such as { score: 0.84 } cannot explain a release. Preserve case-level rows and calculate the scorecard from them. That allows the same observations to be regrouped when the team discovers a high-risk slice without rerunning the model.

The following TypeScript defines a compact result contract and validates comparison provenance. It intentionally treats missing evidence as an error rather than as a failed model answer.

type Outcome = "pass" | "fail" | "invalid";

type EvalRow = {
  caseId: string;
  candidateId: string;
  datasetVersion: string;
  graderVersion: string;
  criterion: string;
  slice: Record<string, string>;
  outcome: Outcome;
  evidencePath?: string;
  reason?: string;
};

type ComparisonIdentity = {
  datasetVersion: string;
  graderVersion: string;
  rowIds: string[];
};

function stableSlice(slice: Record<string, string>): string {
  return JSON.stringify(Object.entries(slice).sort(([left], [right]) => left.localeCompare(right)));
}

function identity(rows: EvalRow[]): ComparisonIdentity {
  if (rows.length === 0) throw new Error("Evaluation contains no rows");

  const datasets = new Set(rows.map((row) => row.datasetVersion));
  const graders = new Set(rows.map((row) => row.graderVersion));
  if (datasets.size !== 1 || graders.size !== 1) {
    throw new Error("A candidate contains mixed dataset or grader versions");
  }

  const rowIds = rows
    .map((row) => `${row.caseId}\u0000${row.criterion}\u0000${stableSlice(row.slice)}`)
    .sort();
  if (new Set(rowIds).size !== rowIds.length) {
    throw new Error("A candidate contains duplicate evaluation rows");
  }

  return {
    datasetVersion: rows[0].datasetVersion,
    graderVersion: rows[0].graderVersion,
    rowIds,
  };
}

export function assertComparable(
  baseline: EvalRow[],
  candidate: EvalRow[],
): void {
  const left = identity(baseline);
  const right = identity(candidate);
  if (left.datasetVersion !== right.datasetVersion) {
    throw new Error(
      `Dataset mismatch: ${left.datasetVersion} != ${right.datasetVersion}`,
    );
  }
  if (left.graderVersion !== right.graderVersion) {
    throw new Error(
      `Grader mismatch: ${left.graderVersion} != ${right.graderVersion}`,
    );
  }
  if (JSON.stringify(left.rowIds) !== JSON.stringify(right.rowIds)) {
    throw new Error("Candidate and baseline evaluation rows do not match");
  }
}

The row has no universal score field because not every criterion should be reduced to the same scale. A pass/fail policy check can remain categorical. A preference grader might store a label elsewhere, then translate it into the release criterion through a reviewed rule. Latency can retain its native unit. Conversions happen in a named policy function, not through an accidental spreadsheet formula.

Slices belong on every row. Useful slice dimensions include locale, customer tier, workflow, input length band, tool path, risk class, content source, and case origin. Do not add dimensions merely because telemetry makes them available. Each dimension increases the number of comparisons and the chance of reacting to a tiny group. Include a slice when the product behaves differently there or when harm would be obscured without it.

Evidence paths should resolve to durable artifacts accessible to reviewers. A path can point to a redacted transcript, trace, assertion details, or a stored diff. It should not depend on an engineer’s local temporary directory. Preserve the raw observation behind a semantic grade, including the grader’s reason, because a numeric result alone cannot show whether the rubric was applied sensibly.

Mark invalid rows explicitly. A request timeout, corrupted fixture, missing trace, and grader parse failure did not demonstrate that the product failed the criterion. Counting them as product failures creates false regressions. Dropping them silently makes the denominator look healthier than it was. A release policy can separately limit invalid execution, which is often an important reliability signal.

Compute rules without manufacturing precision

Release rules should be readable in a code review. “No critical case may change from pass to fail” is readable. “Composite quality must remain above 0.817” invites questions about why the third decimal place matters. Use precision supported by the labels and sample size.

One practical policy compares paired case outcomes, checks absolute floors for established criteria, and refuses to decide when too many rows are invalid. The next implementation accepts explicit rules and returns every violation. The figures in its sample data are illustrative inputs, not measured product performance.

type Outcome = "pass" | "fail" | "invalid";

type Result = {
  caseId: string;
  criterion: string;
  sliceKey: string;
  risk: "critical" | "standard";
  outcome: Outcome;
};

type GateRule = {
  id: string;
  criterion: string;
  sliceKey?: string;
  minimumValidCases: number;
  minimumPassRate: number;
  blockCriticalRegressions: boolean;
};

type Violation = {
  ruleId: string;
  message: string;
  caseIds: string[];
};

function key(row: Result): string {
  return `${row.caseId}\u0000${row.criterion}\u0000${row.sliceKey}`;
}

export function evaluateRules(
  baseline: Result[],
  candidate: Result[],
  rules: GateRule[],
): Violation[] {
  const baselineByKey = new Map(baseline.map((row) => [key(row), row]));
  const violations: Violation[] = [];

  for (const rule of rules) {
    const rows = candidate.filter(
      (row) =>
        row.criterion === rule.criterion &&
        (rule.sliceKey === undefined || row.sliceKey === rule.sliceKey),
    );
    const valid = rows.filter((row) => row.outcome !== "invalid");
    const passed = valid.filter((row) => row.outcome === "pass");

    if (valid.length < rule.minimumValidCases) {
      violations.push({
        ruleId: rule.id,
        message: `Only ${valid.length} valid cases; need ${rule.minimumValidCases}`,
        caseIds: rows.filter((row) => row.outcome === "invalid").map((row) => row.caseId),
      });
      continue;
    }

    const passRate = passed.length / valid.length;
    if (passRate < rule.minimumPassRate) {
      violations.push({
        ruleId: rule.id,
        message: `Pass rate ${passRate.toFixed(3)} is below ${rule.minimumPassRate}`,
        caseIds: valid.filter((row) => row.outcome === "fail").map((row) => row.caseId),
      });
    }

    if (rule.blockCriticalRegressions) {
      const regressions = valid.filter((row) => {
        const previous = baselineByKey.get(key(row));
        return row.risk === "critical" && previous?.outcome === "pass" && row.outcome === "fail";
      });
      if (regressions.length > 0) {
        violations.push({
          ruleId: rule.id,
          message: "Critical paired cases regressed",
          caseIds: regressions.map((row) => row.caseId),
        });
      }
    }
  }

  return violations;
}

This policy does not claim statistical certainty. It enforces product rules against the available reviewed cases. If the organization uses confidence intervals or hypothesis tests, define the method, assumptions, minimum sample size, and decision boundary in the scorecard-policy version. Do not paste a statistical formula into the report without explaining what population the cases represent.

Small slices deserve caution. If three Spanish account-recovery cases include one failure, the slice pass rate is 0.667, but the decimal does not make the estimate stable. The correct response might be to block because the failed case is critical, to request review, or to expand the dataset. It should not be to hide the slice inside a larger language average.

Worked example: the average stays flat

Imagine an illustrative dataset with 100 paired cases. Candidate B fixes five low-risk formatting failures and introduces five account-recovery failures. The overall pass count does not change. A headline score sees a tie. A policy that blocks critical paired regressions rejects the release and lists the five case IDs.

That decision does not require asserting that five cases predict a production percentage. It rests on a narrower claim: reviewed critical behaviors that passed in the baseline now fail in the candidate under the same evaluator. The team can inspect each trace and decide whether the cases still represent supported behavior.

Worked example: a quality gain comes from missing evidence

Candidate B appears to improve because eight difficult tool-use rows are absent. If the report calculates only over returned rows, its denominator shrinks and the score rises. The comparable-case assertion should fail before scoring. If both candidates contain the case IDs but Candidate B marks those rows invalid, the invalid-execution rule should prevent a green decision.

This is a test-harness problem until evidence says otherwise. Look for timeouts, worker crashes, rate limits, malformed output, or artifact upload failures. Do not tune the prompt based on a report that did not execute the same work.

Worked example: two judges reverse a preference

A pairwise grader prefers Candidate B when B is shown first, then prefers Candidate A when the order is swapped. That is evaluator sensitivity, not a clean product win. Preserve presentation order in the row and include order-swapped calibration cases when pairwise judgment matters. The scorecard can classify inconsistent pairs as invalid or review, according to policy, instead of awarding a convenient winner.

Worked example: a threshold punishes an honest abstention

An assistant is expected to abstain when retrieval does not contain enough evidence. Candidate A guesses on those cases and sometimes receives a semantic “helpful” label. Candidate B refuses more often after a grounding fix, so its general answer score falls. A scorecard that treats every abstention as a failed answer would reject the safer behavior.

Split the contract into answer quality when an answer is supported, appropriate abstention when it is not, and retrieval sufficiency as an upstream observation. Then compare the candidates on each criterion. The release owner can decide whether the new abstention rate is acceptable, but the report no longer smuggles that decision into a generic helpfulness score.

The same pattern appears when a policy change deliberately makes a flow stricter. Baseline regressions are not automatically bad. They are signals that require an approved exception or a new expected behavior. Keep the old case and its changed rationale in the dataset history so the scorecard does not erase why the decision moved.

Investigate a red or suspiciously green result

Start with provenance and row counts. Confirm candidate IDs, case-set equality, evaluator versions, and invalid totals before reading individual transcripts. This five-minute check catches reports that should never have reached interpretation.

Next, group regressions by criterion and slice. A cluster in one locale suggests different causes from failures spread across every workflow. A cluster in long-context cases points toward truncation, retrieval volume, or context assembly. A cluster after tool calls may involve tool arguments, tool outputs, or response synthesis. Grouping is a way to choose an investigation, not proof of a cause.

The following command assumes newline-delimited rows matching the earlier contract. It prints failed candidate rows with the fields a reviewer needs. The command is real; the displayed values are illustrative.

jq -c '
  select(.candidateId == "candidate-b")
  | select(.outcome == "fail")
  | {
      caseId,
      criterion,
      locale: .slice.locale,
      workflow: .slice.workflow,
      reason,
      evidencePath,
      datasetVersion,
      graderVersion
    }
' artifacts/eval-results.ndjson

A useful line looks like this:

{"caseId":"recovery-es-017","criterion":"required_disclosure","locale":"es-ES","workflow":"account_recovery","reason":"Recovery wait time omitted","evidencePath":"artifacts/traces/recovery-es-017.json","datasetVersion":"support-42","graderVersion":"disclosure-rubric-6"}

Open the evidence for paired baseline and candidate cases side by side. Check the product inputs before comparing prose. Were the same retrieved documents supplied? Did both tools return the same status? Was the conversation history assembled identically? A scorecard result can reveal a regression, but it cannot by itself locate which component changed.

For semantic failures, read the rubric and grader reason. If the reason cites text that is not present, there may be transcript assembly or grader behavior to investigate. If the reason applies a requirement not in the rubric, relabel the row and add it to calibration. If humans consistently agree with the grader, the evidence is stronger, but still inspect the underlying product path before choosing a fix.

Investigate unexpected improvements with the same discipline. A broad jump can come from duplicated easy cases, fewer valid rows, leaked reference answers, changed grader strictness, or a real product improvement. Green reports deserve provenance checks because teams are naturally less skeptical of results they want.

Do not rerun only failed model calls until they pass. Selective reruns change the meaning of the candidate result and hide variance. If execution policy allows retries, apply the same policy to every candidate and record attempt-level outcomes. Keep “passed after retry” distinct if reliability matters to the release.

Separate missing executions from a lossy scorecard join

A minimum-case violation can hide two failures with almost identical release output. In the first, evaluation workers never produced terminal rows because requests timed out or processes stopped. In the second, every evaluation completed, but the scorecard builder keyed rows only by case ID and overwrote the separate criterion or slice rows for that case. Both reports can say that a rule has too few valid cases. Only the second is entirely manufactured after evaluation.

Trace the count through four boundaries: scheduled comparison identities, terminal execution receipts, raw evaluation rows, and rows admitted to each scorecard rule. A healthy run has one expected raw row for every case, criterion, and declared slice identity, with any invalid outcome retained rather than dropped. An execution failure has a scheduled identity without a terminal receipt or has a terminal invalid row with an infrastructure reason. A lossy join has complete terminal receipts and complete raw rows, followed by fewer scorecard rows. Duplicate case IDs across criteria are normal, so the identity must include the criterion and stable slice values as well as the case.

Read a reported pass rate only after those counts agree. The candidate identifier says which build produced the observation. Dataset and grader versions establish comparability. Valid-case count is the denominator actually used. Invalid count says how much execution evidence was unusable, but it does not explain why. The violation case IDs show where to inspect, while the evidence path should lead to the raw observation. A healthy adjacent slice might show all expected identities and a pass rate derived from them. A broken product slice shows the same valid count with one or more explicit fail rows. A misleading green slice has a smaller denominator, no corresponding invalid rows, and raw evidence that was lost during grouping. A three-decimal pass rate cannot compensate for a missing identity.

Slice membership can drift even when the total row count stays constant. A Spanish recovery case may move into an unknown locale bucket because the candidate exporter omitted locale metadata, making the Spanish slice look better without improving an answer. Compare the case’s versioned dataset labels with the slice values on both candidate rows before interpreting movement. A healthy comparison keeps membership fixed and changes only observed outcomes. A product regression keeps the same membership and turns a pass into a fail. A misleading improvement moves cases between buckets while the underlying outcomes remain unchanged. Runtime-derived slices are appropriate for observations such as the tool path actually taken, but case attributes such as intended locale should come from the reviewed case definition. Mixing those sources under one unnamed field makes the report impossible to audit.

Routing follows the boundary where the counts diverge. Evaluation infrastructure owns scheduled work without terminal evidence. The result-schema or scorecard owner owns rows present in raw output but missing after aggregation. Dataset owners resolve duplicate or absent expected identities. The product or model owner should receive a regression only when comparable valid rows actually changed outcome. A handoff needs the candidate and baseline IDs, policy version, ordered counts at all four boundaries, the full composite identities that disappeared or changed, raw and scorecard artifact locations, retry history, and the exact rule whose denominator changed. Without that bundle, a worker rerun may conceal an aggregation defect.

Preserving this audit path costs storage and computation. Keeping case-level rows for every criterion and slice consumes more space than saving aggregates, and rebuilding a scorecard under a revised policy must scan those rows again. Paired evaluation requires one baseline execution and one candidate execution before any repetitions or judge calls. Those costs are concrete reasons to keep the pull-request set compact, not reasons to discard row identities before the release decision is final.

This check does not catch a risk absent from both the dataset and policy. If a new workflow silently falls outside every declared slice, perfect row accounting can still produce a green scorecard. Production discovery, incident review, and dataset coverage audits must find that omission. Aggregation integrity proves that the scorecard faithfully applied its rules, not that the rules describe every important behavior.

Roll out scorecards without freezing bad policy

Begin with a historical replay. Apply the proposed scorecard to several known releases, including one with a defect the team cares about. The scorecard should surface that defect through a named rule. If it does not, either the cases, criteria, or policy do not represent the risk yet.

Before replaying outcomes, make the existing exporter preserve composite row identity and invalid states. This is the dependency most likely to break when an aggregate-only dashboard becomes a release gate. Land schema validation and immutable raw artifacts first, then reproduce the current report from those rows. Add paired baseline comparison next, followed by deterministic critical rules. Slice thresholds and semantic graders come later because they require calibration and minimum-case decisions. The migration is working when the row-based implementation reproduces the approved legacy decisions, explains every deliberate difference, and can rebuild the same scorecard from frozen artifacts without rerunning a model.

Run the scorecard in shadow mode next. Publish its decision beside the existing release process, but do not block. Review false alarms, missed issues, invalid rows, and evidence usability. A rule that catches real defects but takes hours to understand is not ready for frequent CI.

Promote deterministic critical rules first. Semantic aggregate thresholds can remain advisory while calibration matures. Record who owns each rule and what happens on failure. “QA reviews it” is not enough. A policy-compliance regression may need a product safety owner; missing traces may belong to platform engineering.

The workflow below separates evaluation from policy and always uploads the evidence. The repository scripts are local implementation points, not vendor API names.

name: ai-release-scorecard

on:
  workflow_dispatch:
    inputs:
      candidate_id:
        description: Candidate identifier
        required: true
      baseline_id:
        description: Baseline identifier
        required: true

jobs:
  scorecard:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Run paired evaluations
        run: >-
          npm run eval:paired --
          --candidate "${{ inputs.candidate_id }}"
          --baseline "${{ inputs.baseline_id }}"
          --output artifacts/eval-results.ndjson
      - name: Build and enforce scorecard
        run: >-
          npm run scorecard --
          --results artifacts/eval-results.ndjson
          --policy evals/release-policy.json
          --report artifacts/scorecard.json
      - name: Preserve scorecard evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ai-release-scorecard
          path: artifacts/

Version policy changes through review. Tightening a threshold, adding a blocker, or changing a slice definition can stop a release even when product behavior is unchanged. The pull request should show which historical candidates change status under the new policy.

Keep scorecards immutable after approval. If a case label is corrected, issue a new scorecard revision and link the reason. Editing the old report in place breaks the audit trail and makes later incident review unreliable.

Plan the rollback before making the scorecard authoritative. A blocking job can fail because the candidate regressed, because the evaluation service is unavailable, or because evidence upload broke. Those states need different actions. A confirmed critical regression should stop promotion. A short-lived evaluator outage may permit a controlled rerun. Missing artifacts should invalidate the decision even when all visible checks passed. Encode those distinctions in job output and in the release runbook, then test the runbook with a deliberately broken artifact path before depending on it during an urgent release.

After launch, compare scorecard expectations with production observations. Tag incidents and sampled failures with the closest criterion and slice. If an important incident has no matching case, add a minimal reproduction and note its origin. If a rule repeatedly blocks candidates but never corresponds to user harm or reviewer concern, inspect whether the policy is too strict, the case is stale, or the evidence is misleading. This feedback loop changes the dataset based on observed gaps rather than adding random examples to make the suite look complete.

Know the trade-offs and when not to use a scorecard

The first cost is maintenance. Criteria evolve with product promises. Slices change as traffic and workflows change. Graders need calibration. Evidence storage needs retention and access controls. A scorecard that no one owns slowly becomes a confident report about an old product.

The second cost is execution time. Paired candidates roughly double inference work before repeats, judge calls, tool calls, and retrieval. Broad slice coverage expands the dataset further. Use a small representative gate on pull requests and a larger run for release candidates. Do not pretend the small gate covers risks reserved for the larger run.

The third cost is organizational. Explicit rules expose disagreements about acceptable quality. That friction is useful, but it needs a decision owner. QA should make evidence legible, not invent product risk tolerance alone.

Avoid an automatic scorecard gate during early exploration when the task, rubric, and expected behavior change daily. Run evaluations and preserve observations, but keep decisions in review. Automating unstable policy creates churn without improving confidence.

Do not use a scorecard to merge unrelated products into a league table. A customer-support agent and a code-generation assistant do not share one meaningful quality unit. Give each a decision-specific policy, even if the evaluation platform stores both.

Do not release solely because every offline rule passes. Production traffic can contain distributions, tool failures, and adversarial behavior absent from the dataset. Pair the scorecard with staged rollout, monitoring, and rollback criteria. Offline evidence reduces uncertainty; it does not remove it.

Finally, do not block on a score whose evidence cannot be reviewed. If failed rows lack transcripts, traces, reasons, or version metadata, classify the scorecard as invalid and repair the pipeline. A red number without a reproducible case is an alert. It is not yet a release argument.