When two reviewers disagree, inspect the rubric first

Two reviewers inspect the same assistant answer. One marks it grounded because every claim appears in the retrieved document. The other marks it unsupported because the answer cites the wrong document version. The problem is not solved by averaging their labels.

Give reviewers the same object and the same decision

Agreement starts before anybody opens the labeling tool. Each reviewer needs the same evaluation unit, evidence bundle, rubric version, and allowed label set. If one person sees the final message while another sees the full tool trajectory, their labels describe different objects even when the case ID matches.

Define the unit in operational terms. For a retrieval answer, it might include the user question, exact document snapshot, assistant response, and citations. For an agent, it may include tool calls, arguments, returned observations, approval events, and final status. A screenshot of the last message cannot support a judgment about whether a destructive tool was authorized.

A rubric needs observable boundaries. “Good response” is not a label definition. “Grounded” becomes reviewable when it means that every externally verifiable claim is supported by the supplied evidence and no cited source contradicts the claim. Add examples near the boundary: a correct claim with the wrong citation, an incomplete but supported answer, and a source that implies rather than states the conclusion.

Tell reviewers what not to infer. If the case does not contain backend state, they cannot judge whether the order was actually refunded. They can judge whether the response matches the tool result shown. This rule prevents domain-savvy reviewers from silently adding evidence that less experienced reviewers do not have.

Label independence matters. Reviewers should not see each other’s decisions before submitting their first label. A chat thread where the first reviewer explains a choice can produce impressive agreement by social convergence. That may be useful during calibration, but it is not an independent reliability measurement.

Preserve abstention. “Cannot judge” is different from “fail” and different from a missing submission. It might mean corrupted evidence, an unfamiliar language, a rubric gap, or a genuine policy question. Require a reason code. An abstention rate concentrated in one slice often identifies a staffing or data problem sooner than any agreement statistic.

Use labels that match the decision. Binary pass or fail is appropriate for a crisp contract. Ordinal labels such as fully supported, partially supported, and unsupported retain severity when partial credit affects release decisions. Pairwise preference answers another question entirely: which of two acceptable responses is better. Mixing these tasks in one field makes the confusion matrix hard to interpret.

Reviewers also need a declared tie-breaking path. Adjudication can be a senior reviewer, a policy owner, or a panel. The adjudicator sees original labels and rationales, resolves the production label, and records the reason. Never replace the original labels. Those disagreements are the evidence needed to repair the rubric and detect reviewer drift later.

Calculate agreement without losing the confusion

Percent agreement answers a narrow question: on what fraction of independently labeled cases did the reviewers choose the same category? It is worth reporting because anyone can understand it. It is not enough by itself. A dataset with a dominant class can produce high agreement even when reviewers handle the rare, important class inconsistently.

Cohen’s kappa is commonly used for two categorical reviewers. It compares observed agreement with the agreement expected from each reviewer’s label proportions. The chance model is not a law of nature. When almost every item belongs to one class, kappa can be low or unstable despite high raw agreement. That is a reason to inspect prevalence and the confusion matrix, not a reason to hide either number.

The script below reads one CSV row per case with columns case_id, labeler_a, and labeler_b. It validates duplicates and missing values, prints a confusion matrix, computes observed agreement, and computes unweighted kappa. Keep any slice column in the export for the separate slice reports described below. The script uses only the Python standard library.

from __future__ import annotations

import argparse
import csv
from collections import Counter
from pathlib import Path


def divide(numerator: float, denominator: float) -> float:
    if denominator == 0:
        raise ValueError('agreement is undefined for an empty dataset')
    return numerator / denominator


def main(path: Path) -> None:
    rows: list[dict[str, str]] = []
    seen: set[str] = set()

    with path.open(newline='', encoding='utf-8') as handle:
        for line_number, row in enumerate(csv.DictReader(handle), start=2):
            case_id = (row.get('case_id') or '').strip()
            a = (row.get('labeler_a') or '').strip()
            b = (row.get('labeler_b') or '').strip()

            if not case_id or not a or not b:
                raise ValueError('line ' + str(line_number) + ': missing required value')
            if case_id in seen:
                raise ValueError('duplicate case_id: ' + case_id)

            seen.add(case_id)
            rows.append({'case_id': case_id, 'a': a, 'b': b})

    labels = sorted({row['a'] for row in rows} | {row['b'] for row in rows})
    matrix = Counter((row['a'], row['b']) for row in rows)
    total = len(rows)

    observed = divide(
        sum(matrix[(label, label)] for label in labels),
        total,
    )

    a_counts = Counter(row['a'] for row in rows)
    b_counts = Counter(row['b'] for row in rows)
    expected = sum(
        divide(a_counts[label], total) * divide(b_counts[label], total)
        for label in labels
    )

    kappa = (observed - expected) / (1 - expected) if expected != 1 else None

    print('labels:', labels)
    print('confusion matrix, rows=A and columns=B')
    print('A/B,' + ','.join(labels))
    for left in labels:
        print(left + ',' + ','.join(str(matrix[(left, right)]) for right in labels))
    print('observed_agreement:', round(observed, 4))
    print('expected_agreement:', round(expected, 4))
    print('cohen_kappa:', 'undefined' if kappa is None else round(kappa, 4))


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('labels_csv', type=Path)
    args = parser.parse_args()
    main(args.labels_csv)

An undefined kappa is not a software failure. If both reviewers assign the same single category to every case, expected agreement is one, so the denominator becomes zero. Report perfect observed agreement, the one-class distribution, and the fact that kappa is undefined. Then ask whether the dataset actually challenged the rubric.

A single overall kappa can conceal the release risk. Calculate agreement by meaningful slice as well as overall: language, misuse category, tool path, customer segment, or difficulty band. Do not publish a tiny slice’s coefficient without its case count and label distribution. A dramatic value from four cases is an invitation to inspect them, not a stable performance claim.

For ordinal labels, unweighted kappa treats adjacent and extreme disagreements the same. Weighted kappa can express that fully supported versus partially supported is less severe than fully supported versus fabricated, but the weights encode a value judgment. Write those weights into the rubric and review them with the product owner. Do not choose weights after seeing which scheme produces a reassuring result.

With more than two labelers, the calculation changes. The underlying questions do not. Who labeled each case? Were assignments balanced? Did all reviewers use the same rubric? Was missingness systematic? Use a method suited to the design, and always keep the category counts and disagreement rows accessible. A familiar statistic applied to the wrong assignment pattern is worse than a simple, honest table.

Learn from three disagreements that look alike in a dashboard

Imagine an eval for unsupported claims. In the first worked case, the assistant says, “The warranty lasts two years.” The supplied policy states two years, but the citation points to an older one-year document. Reviewer A judges claim correctness and chooses supported. Reviewer B judges citation correctness and chooses unsupported. Both followed plausible interpretations because the rubric combined two checks.

The repair is to split factual support from citation validity. A response can be factually supported by the evidence bundle and still attach a wrong citation. That separation gives engineering a better diagnosis: retrieval or citation selection failed, not necessarily generation. It also stops an adjudicator from making the same debate on every similar case.

In the second case, almost all outputs are harmless, and the rare label is “actionable credential theft.” Suppose an illustrative batch has 200 cases. The reviewers agree that 194 are harmless, disagree on four borderline cases, and agree that two are harmful. Raw agreement is high. The rare-class recall, however, depends heavily on those four disagreements. These figures are illustrative, not collected results.

A team that reports only overall agreement will miss the risk. Print the two-by-two confusion matrix and read every harmful or disputed row. Ask whether one reviewer requires executable steps while another treats intent plus partial steps as actionable. Add counterexamples to the rubric, then label a fresh calibration set. Relabeling the same four rows immediately can measure memory more than understanding.

In the third case, an agent says it cannot issue a refund and asks for supervisor approval. Reviewer A sees the final message and labels the behavior safe. Reviewer B sees the trajectory, notices that the refund tool was already called, and labels it unsafe. Their disagreement is caused by evidence scope, not a subtle policy boundary.

The fix is mechanical: bind a manifest of required artifacts to every task. The labeling UI should fail closed when a trajectory-required case lacks tool events. A reviewer can then select “incomplete evidence” instead of guessing. Adjudication cannot reconstruct missing telemetry after the fact.

These examples produce the same dashboard symbol, two different labels. They need different owners:

A broken review join can impersonate rubric confusion

There is another failure that produces the same off-diagonal confusion matrix. Reviewer A and reviewer B may each label their assigned evidence consistently, but an export job pairs A’s submission for one case with B’s submission for the next case. The agreement report then shows a sudden rise in pass-versus-fail rows. Rewriting the rubric cannot repair a join that compared different objects.

Check comparability before interpreting any label. A healthy paired row has one case version, one evidence-unit hash, one rubric version, two distinct assignment identities, and two independently submitted labels. A real rubric disagreement preserves all of those identities while the labels differ. A broken join has a mismatch in evidence hash, case version, or source assignment relationship even if the export has stamped one case ID onto the combined row.

The diagnostic output should put the report case ID beside each source submission’s case ID, evidence hash, rubric version, labeler identity, assignment identity, and label. In a healthy agreement, both source case IDs and hashes match, then the labels match. In a valid disagreement, the source IDs and hashes still match, but the labels differ. In the join defect, the paired hashes or source case IDs diverge. That row is invalid evidence, not disagreement.

Row number and submission time are misleading fields. Two reviewers can finish the same assignment hours apart, while two adjacent cases can be submitted seconds apart. Sorting each reviewer’s export and zipping by position creates plausible-looking pairs until one missing submission shifts every later row. A report can then show hundreds of disagreements from one absent record. Join on the immutable assignment-to-case relationship and reject one-to-many or missing relationships before calculating the matrix.

The repair owner depends on where the mismatch first appears. If source submissions contain the correct case and evidence identities but the combined report does not, the analytics or export owner fixes the join. If the assignment service issued the wrong evidence under the right case identity, the labeling-platform owner fixes packaging and affected work must be discarded. If identities match and rationales reveal different criteria, the rubric or policy owner resolves the real boundary.

A useful handoff contains the batch and rubric versions, the two source submission identities, both case versions and evidence hashes, assignment records, the first row where alignment diverges, missing or duplicated submission counts, and a small redacted reproduction. It should also state whether labels need to be re-exported or cases must be independently relabeled. Relabeling is required when a reviewer actually saw the wrong object. Re-exporting is enough when only the report paired correct submissions incorrectly.

Hashing and retaining assignment provenance adds storage, validation work, and another invalid state that operations must handle. Rejecting a malformed batch can delay a release while the export is rebuilt. The alternative is worse: paid reviewers appear unreliable, rubric authors change sound definitions, and adjudicators waste time resolving comparisons that never existed.

Rationales make this triage possible. Require a short evidence-based reason for risky labels and disagreements. Do not demand an essay on every obvious pass, because reviewer fatigue creates copied text. A citation to the relevant span, tool event, or rubric clause is usually more useful than generic prose.

A TypeScript validator can prevent mismatched units and rubric versions from entering the agreement report. It does not decide whether a label is correct. It proves that the two labels are comparable and writes a focused disagreement queue.

type Label = 'pass' | 'fail' | 'cannot_judge';

type Review = {
  caseId: string;
  labelerId: string;
  unitHash: string;
  rubricVersion: string;
  label: Label;
  rationale: string;
};

const reviews: Review[] = JSON.parse(process.argv[2] ?? '[]');
const byCase = new Map<string, Review[]>();

for (const review of reviews) {
  const group = byCase.get(review.caseId) ?? [];
  if (group.some(item => item.labelerId === review.labelerId)) {
    throw new Error(review.caseId + ': duplicate submission by ' + review.labelerId);
  }
  group.push(review);
  byCase.set(review.caseId, group);
}

const disagreements: object[] = [];

for (const [caseId, group] of byCase) {
  if (group.length !== 2) {
    throw new Error(caseId + ': expected exactly two independent reviews');
  }

  const [left, right] = group;
  if (left.unitHash !== right.unitHash) {
    throw new Error(caseId + ': reviewers saw different evidence units');
  }
  if (left.rubricVersion !== right.rubricVersion) {
    throw new Error(caseId + ': mixed rubric versions');
  }
  if (left.label !== right.label) {
    disagreements.push({
      caseId,
      labels: [left.label, right.label],
      rationales: [left.rationale, right.rationale],
      rubricVersion: left.rubricVersion,
    });
  }
}

console.log(JSON.stringify({ disagreements }, null, 2));

Passing JSON as a command-line argument is convenient for a small demonstration, not for a large or sensitive export. A production tool should read an access-controlled file or stream, redact sensitive content, and write only the fields needed by adjudicators.

Tell reviewer drift from model drift

A release lands on Tuesday and agreement drops on Wednesday. It is easy to blame unfamiliar model behavior. The reviewer pool may also have changed, the rubric UI may have cached an old version, or the assignment system may have sent all difficult cases to one specialist. Each cause needs different evidence.

Maintain an anchor set that does not change with the current candidate. Anchors should cover known boundary cases, include adjudicated reasoning, and remain hidden enough that reviewers cannot simply memorize an answer order. Send a sample throughout the project. Movement on anchors while production-case agreement changes points toward reviewer or process drift. Stable anchors with new disagreements concentrated in candidate outputs points toward behavior the rubric did not anticipate.

Do not use anchors as traps. A reviewer should receive feedback and a path to challenge an outdated gold label. Product policy changes. When an anchor changes, create a new version and preserve the old decision so historical agreement remains interpretable.

Look for asymmetry by labeler. Build pairwise confusion tables and per-label distributions. If one person chooses “cannot judge” far more often, inspect whether they receive incomplete evidence or need domain access. If one person never uses “partial,” their mental threshold may differ. Avoid ranking reviewers by a single agreement score; assignment mix and case difficulty can make such rankings unfair and operationally misleading.

Position and presentation can drift too. A UI update may truncate long retrieved passages, collapse tool arguments, or display a candidate response first in every pairwise task. Capture evidence-unit hashes and relevant UI version identifiers. Screenshots of a few disputed cases can reveal presentation defects that database exports cannot.

The following Python script creates a diagnostic report from original reviews and a separate adjudication file. It preserves independence, measures how often each reviewer is reversed by adjudication, and lists cases where both reviewers agreed but the adjudicator chose another label. That last condition is especially valuable because it points to a gold-set or rubric problem.

from __future__ import annotations

import argparse
import csv
from collections import Counter, defaultdict
from pathlib import Path


def read_rows(path: Path) -> list[dict[str, str]]:
    with path.open(newline='', encoding='utf-8') as handle:
        return list(csv.DictReader(handle))


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('reviews_csv', type=Path)
    parser.add_argument('adjudications_csv', type=Path)
    args = parser.parse_args()

    reviews = read_rows(args.reviews_csv)
    adjudications = {
        row['case_id']: row['final_label']
        for row in read_rows(args.adjudications_csv)
    }

    by_case: dict[str, list[dict[str, str]]] = defaultdict(list)
    for row in reviews:
        by_case[row['case_id']].append(row)

    reversed_by_labeler: Counter[str] = Counter()
    total_by_labeler: Counter[str] = Counter()
    unanimous_reversals: list[str] = []

    for case_id, group in sorted(by_case.items()):
        final = adjudications.get(case_id)
        if final is None:
            continue

        labels = {row['label'] for row in group}
        if len(labels) == 1 and final not in labels:
            unanimous_reversals.append(case_id)

        for row in group:
            labeler = row['labeler_id']
            total_by_labeler[labeler] += 1
            if row['label'] != final:
                reversed_by_labeler[labeler] += 1

    for labeler in sorted(total_by_labeler):
        print(
            labeler,
            'reversed',
            reversed_by_labeler[labeler],
            'of',
            total_by_labeler[labeler],
        )

    print('unanimous_reversals:', unanimous_reversals)

A high reversal count is a starting point for case review, not proof of a poor reviewer. Perhaps the adjudicator is applying a new policy. Perhaps one reviewer receives the hardest locale. Compare like assignments, read rationales, and let reviewers appeal.

Another near-miss is model variance. Two reviewers may receive different generations under the same case ID because the labeling export was assembled before results were frozen. Their labels can both be correct for what they saw. Evidence hashes expose this immediately. Without them, the team may rewrite a perfectly clear rubric to solve a data pipeline defect.

Roll out a labeling process that can be audited

Begin with a small calibration round drawn from every important slice. Reviewers label independently, then discuss disagreements together. The facilitator updates examples and definitions, versions the rubric, and starts another fresh round. Stop calibrating when the remaining disagreements are understood and acceptable for the release decision, not when discussion pressure produces unanimity.

Move into production labeling with overlap by design. Some cases can receive one label if the risk and rubric maturity permit it, but a planned subset needs independent duplicate review so agreement remains observable. Oversample risky and newly introduced behaviors. Random overlap limited to the dominant easy class will give a stable number while boundaries decay unnoticed.

The manifest below is illustrative. It records the process controls a labeling service or local scheduler should enforce. These are project-owned fields, not configuration keys for a named evaluation product.

rubric:
  id: grounded-answer
  version: 5
  labels:
    - supported
    - partially_supported
    - unsupported
    - cannot_judge
assignment:
  independent_reviews_per_overlap_case: 2
  hide_other_reviews_until_submission: true
  anchor_set_version: anchors-2026-08
  require_same_evidence_unit_hash: true
adjudication:
  preserve_original_labels: true
  require_reason_code: true
  route_unanimous_reversal_to_policy_owner: true
reporting:
  include_confusion_matrix: true
  include_label_prevalence: true
  include_slice_case_counts: true
  include_abstention_reasons: true

Store rubric version and evidence hash on each review, not only on the batch. Long-running batches cross deployments and policy updates. Row-level provenance lets you separate versions later without guessing from timestamps.

CI can validate file shape, duplicate submissions, known labels, and complete overlap assignments. It should not block a model release solely because a coefficient is below a copied internet threshold. Let the release policy state which slices require adjudication and how unresolved risky disagreements are handled. A safety disagreement may block until resolution; a tone preference disagreement may remain a tie.

Budget for adjudication. Duplicate labeling without time to resolve disagreements produces a beautiful metric and no better ground truth. Send adjudicators the original evidence, labels, rationales, and rubric version. Hide reviewer identity when practical to reduce hierarchy effects. Capture the final label and reason, then feed recurrent reasons into the next rubric revision.

Monitor operational burden. More independent labels improve visibility but cost money and calendar time. Long rationales help diagnosis but increase fatigue. Expert-only reviewers improve domain decisions but can become a queue bottleneck. Frequent rubric versions preserve accuracy while complicating historical comparison. These costs should shape the sampling plan.

A sensible migration from a single-label dataset is incremental. Keep existing labels as legacy annotations, select a stratified subset for independent relabeling under the new rubric, and compare old versus new definitions. Do not describe the old labels as reviewer A in a kappa calculation if they were created through discussion or copied from production outcomes. Independence and assignment conditions differ.

Before requesting the second label, change storage from one mutable value per case to append-only submissions plus a separate adjudicated value. Preserve a compatibility projection for consumers that still need the legacy label, but never let that projection write back into source submissions. Existing tools often overwrite the first reviewer when a second reviewer saves, or display the first decision in a generic “current value” control. Exercise both failure paths with synthetic assignments before paying for overlap.

Route a pilot overlap batch outside the release gate and deliberately omit one submission. The report must leave one case incomplete rather than pairing later rows by position. Also test a duplicated submission and an evidence-packaging rejection. Add adjudication only after those states pass through exports without mutating originals. The first threshold can then observe a structurally valid batch instead of rewarding a pipeline that manufactured complete pairs.

The first production break is usually workflow capacity. Duplicate review creates more assignments, but adjudication adds a serial queue that cannot be cleared by adding generalist labelers when policy expertise is scarce. Measure time from second submission to adjudication, unresolved risky rows by slice, evidence-packaging failures, and reviewer appeals. These operational fields show whether a nominally reliable process can finish before the release decision expires.

The rollout is working when every overlap row resolves to two independent source submissions, intentional hash mismatches are rejected, original labels survive adjudication, and anchor results remain interpretable across deployments. Agreement should be reported only after those structural checks pass. A target coefficient is not proof of installation because a trivial case mix can reach it while risky slices remain empty.

Ownership is divided deliberately. The labeling-platform team owns assignment independence, evidence delivery, and immutable submissions. The evaluation team owns sampling, comparable joins, metrics, and diagnostic artifacts. The product or policy owner owns label definitions and boundary decisions. Labeling operations owns reviewer access, scheduling, calibration attendance, and escalation. The release owner decides how unresolved risky cases affect shipment. The cutover packet should contain slice counts, label prevalence, invalid rows, disagreement reasons, adjudication age, anchor movement, rubric changes, reviewer access gaps, and the decision being requested from each owner.

Once the overlap process is stable, expand coverage where confusion remains costly. Add cases near observed boundaries rather than endless paraphrases of easy passes. When a new behavior appears, quarantine it as “rubric gap,” convene the policy owner, and add reviewed examples before forcing labelers to improvise.

Avoid agreement metrics when they cannot answer the question

Do not calculate agreement on labels produced collaboratively. The number will mostly measure that reviewers spoke to each other. Collaborative review can produce high-quality decisions, but describe it as panel adjudication rather than independent annotation.

Skip kappa when there is only one observed category and report the distribution honestly. The coefficient is undefined in that condition. More importantly, the set may lack negative or boundary examples. Add meaningful cases because the product needs coverage, not merely to make a statistic calculable.

Do not merge different rubric versions into one score. A revised definition of harmful assistance or groundedness changes the task. Report each version separately and use a versioned anchor bridge if historical comparison is important.

Avoid treating adjudicator agreement as objective truth. A senior reviewer can be inconsistent, underinformed, or bound to an obsolete policy. Unanimous reversals and recurring appeals are evidence about the adjudication layer. Keep that layer auditable too.

Do not chase perfect agreement by deleting hard cases. Those cases often represent the exact product boundaries most likely to generate incidents. A lower but explainable agreement rate with an owned adjudication path is more useful than a polished number built from obvious examples.

Agreement does not catch a shared misconception. Two reviewers can apply the same obsolete policy, overlook the same unsupported claim, or be influenced by the same misleading example and agree on every row. Neither percent agreement nor kappa compares them with product truth. Independent gold anchors, deterministic facts where available, appeals, incident review, and sampled expert adjudication are separate controls for correlated error.

Finally, do not automate the rubric with an LLM judge simply to remove human disagreement. First determine whether the humans lacked evidence, shared definitions, domain skill, or a coherent product policy. Automation belongs after the decision is specified and calibrated. Otherwise the system hides unresolved judgment behind consistent syntax and creates a new evaluator that nobody has earned the right to trust.