Measuring Annotator Agreement for Subjective LLM Rubrics

Annotator disagreement is diagnostic evidence. It can expose a vague construct, an underspecified rubric, missing context, poor training examples, genuine domain ambiguity, or a mislabeled task. Compressing those causes into one agreement percentage wastes the most useful information in a human evaluation.

A rigorous program separates independent annotation from adjudication, selects a metric that matches the label scale, analyzes disagreement by risk slice, and carries unresolved uncertainty into the release decision. The objective is not perfect consensus. It is a measurement process whose limits are visible and whose final labels are reproducible enough for the consequence at hand.

Define the Judgment Before Measuring Agreement

Name one observable criterion per rubric item. "Good answer" mixes correctness, relevance, completeness, style, policy compliance, and perhaps citation quality. Annotators may agree on their overall impression while using different hidden priorities. Break the decision into dimensions with explicit evidence rules, then define whether each dimension is nominal, ordinal, binary, or continuous.

The LangSmith annotation queue guide is a useful reference for organizing human review work. Queue mechanics do not resolve rubric ambiguity, so pair the operational workflow with a versioned rubric, blinded assignment, calibration cases, and a record of every adjudication change.

Preserve Data and Label Provenance

An agreement result is uninterpretable without knowing what each rater saw. Record example identity, source lineage, redaction and normalization steps, output ordering, rubric version, UI rendering version, assigned rater, timestamp, and whether reference material was available. If one annotator saw retrieved documents and another saw only the answer, they were not judging the same unit.

Keep raw independent labels immutable. Adjudicated labels, reviewer comments, and later corrections should be new records linked to the originals. This distinction lets the team calculate agreement before discussion, inspect systematic rater tendencies, and reconstruct why the final gold label changed.

Control leakage into annotation. Do not include candidate names, model families, expected release direction, user feedback labels, or previous grader scores unless the rubric explicitly requires them. Those signals can align raters through expectation rather than shared interpretation of the output.

Design a Calibration Batch

Build a small batch spanning obvious passes, obvious failures, boundary cases, expected abstentions, and every critical slice. Draw it from a development pool, not the sealed release holdout. Include cases that distinguish adjacent ordinal labels and cases where two rubric dimensions pull in different directions.

Annotators first label independently. A facilitator then compares rationales, not just labels. For each disagreement, ask whether the evidence was missing, a term was vague, an exception was absent, or the case required expertise the rater did not have. Rewrite the rubric and run a fresh calibration batch; repeated discussion of the same examples can create memorized agreement without improving general rules.

Calibration is role-specific. A medical-domain rubric, a policy taxonomy, and a writing-quality scale may need different qualification checks. Document which slices require specialists and which can be handled by general reviewers.

Choose a Metric That Matches the Scale

Percent agreement is the number of exact matches divided by jointly labeled items. It is transparent and useful, especially with a confusion matrix, but it can look high when one label dominates. Cohen's kappa for two raters adjusts observed agreement by agreement expected from their marginal label frequencies:

kappa = (observed_agreement - expected_agreement)
        / (1 - expected_agreement)

For more than two raters, missing assignments, ordinal scales, or continuous ratings, choose a statistic designed for that structure, such as an appropriate form of Krippendorff's alpha or an intraclass correlation coefficient. The choice must be specified before examining candidate outcomes. Do not shop among metrics for the most reassuring value.

Report the confusion matrix and label prevalence beside the summary. A single coefficient can hide that raters reliably distinguish clear failures but repeatedly confuse minor_issue with pass. That pattern affects a release gate differently from disagreement about the most severe category.

Quantify Uncertainty in Agreement

Agreement is estimated from a sample. Add an interval using a method suitable for the design, such as resampling complete examples while preserving their sets of rater labels. If examples are clustered by conversation or source document, resample clusters rather than pretending related rows are independent.

Avoid universal cutoffs such as declaring every rubric acceptable above one coefficient. An illustrative internal policy might require a tighter lower confidence bound for a critical safety label than for a stylistic preference. The values should reflect decision cost, label prevalence, and review capacity.

When a rare label appears only a handful of times, its agreement estimate will be unstable even if aggregate agreement is high. State that limitation and collect more targeted calibration evidence instead of treating absence of disagreement as proof.

Diagnose Disagreement by Slice and Rater

Break results down by language, input length, domain, tool-use path, policy category, source, and consequence severity. Also inspect intersections where there is enough evidence. Aggregate agreement can hide a rubric that works for short English answers but fails for multilingual retrieval responses.

Compare each rater's label distribution, abstention rate, and confusion pattern against peers and adjudicated outcomes. A rater who uses pass unusually often may be under-sensitive, or may have received a different case mix. Use assignment-balanced data before attributing a personal bias.

Slice definitions need provenance and versioning. If a model infers domain or severity, that classifier introduces another uncertain label. Sample its errors for human review and distinguish inferred slices from deterministic metadata slices in reports.

Run Adjudication as a Separate Stage

Route disagreements, abstentions, and high-severity cases to an adjudicator who can see the independent labels and rationales. The adjudicator should choose a final label, mark the rubric rule used, and classify the disagreement cause. Useful causes include missing context, ambiguous criterion, rater error, source corruption, legitimate multi-interpretation, and rubric gap.

Do not ask annotators to negotiate before their independent labels are saved. Group discussion can produce consensus but destroys the evidence needed to measure initial consistency. Likewise, do not replace all original labels with the adjudicated one.

When adjudication reveals a rubric change, decide whether earlier labels can be deterministically remapped or need re-review. Increment the rubric version and avoid mixing results from materially different definitions in one baseline.

Combine Human, Deterministic, and Model Graders

Deterministic graders should own requirements that can be checked exactly: valid JSON, required citations, allowed tool names, numerical constraints, or forbidden tokens. Asking humans to judge those repeatedly wastes attention and adds avoidable variance.

Model graders are useful for scaling semantic review, but agreement with humans must be evaluated like any other instrument. Freeze the grader prompt and configuration, compare against an adjudicated set, inspect confusion by slice, and define abstention or escalation behavior. A high aggregate match can coexist with unacceptable errors on a critical minority class.

Use humans where context and consequence justify judgment. A layered design can run deterministic checks first, a calibrated model grader on routine semantic cases, and human review for uncertain, novel, or severe cases. Report which instrument produced each label.

Connect Agreement to Release Decisions

The release report should include raw candidate metrics, agreement diagnostics, adjudication rate, abstentions, grader provenance, and uncertainty by protected slice. If the rubric cannot consistently identify the failure that would block release, the gate is not ready, regardless of the candidate score.

Predefine how ambiguity affects the decision. An illustrative rule could send a release to specialist review when a critical slice has too few double-labeled examples or when the lower agreement bound falls below the team's calibrated requirement. Such numbers are local policy choices, not general quality standards.

Avoid using adjudicated labels from candidate A to revise the rubric and then scoring candidate B under the new rubric without rescoring A. The comparison must use the same measurement instrument or explicitly become a new experiment.

Failure Modes and Tradeoffs

Maximizing agreement can make a rubric shallow: broad labels are easy to align on but may not guide fixes. Excessive rubric detail can overwhelm annotators and create rule conflicts. Specialist review improves validity but costs more and may reduce throughput. Blind review reduces expectation bias but can remove context necessary for a legitimate judgment.

Chance-corrected metrics can behave unexpectedly under extreme prevalence, while plain agreement ignores chance. Adjudication improves final-label usability but can concentrate one expert's bias. Use several views of the evidence and preserve the underlying labels so these tradeoffs remain auditable.

Operational Checklist

• Define one construct and an appropriate scale for each rubric item.
• Record exactly what content and context every rater saw.
• Keep independent, adjudicated, and corrected labels as separate records.
• Calibrate on representative boundary cases outside the sealed holdout.
• Collect labels blindly before any discussion.
• Report prevalence, confusion matrices, agreement statistics, and intervals.
• Diagnose disagreements by risk slice, assignment mix, and rater.
• Give annotators a documented abstention path.
• Reserve deterministic checks for exact properties.
• Calibrate model graders against adjudicated evidence and inspect minority errors.
• Freeze the measurement version for candidate comparisons.

Action Plan: Make the Rubric Release-Ready

Select one subjective release criterion and rewrite it as observable evidence with a bounded scale. Assemble a calibration batch that stresses each boundary, assign it independently to at least the raters planned for production, and compute agreement with a confusion matrix before holding a review session.

Classify every disagreement, revise the rubric, and repeat on fresh examples. Then freeze the rubric, assignment policy, metric, and adjudication process for the release run. Proceed to gating only when the team can explain where judgment remains uncertain, which slices need specialist review, and how that uncertainty changes the ship decision.