Mining Grader Disagreements to Improve LLM Eval Coverage

When two graders disagree, the useful question is not "which score should we average?" It is "what assumption just broke?" One grader may use different evidence, one rubric boundary may be vague, a parser may map labels incorrectly, or the output may expose a failure mode absent from the suite. Disagreement is a discovery queue, not a label.

A rigorous program preserves every component verdict, aligns only compatible criteria, ranks conflicts by risk, and sends a designed sample to independent human review. The LangSmith evaluation concepts describe code, human, model-based, pairwise, reference-based, and reference-free evaluators. Those methods can legitimately diverge because they observe different evidence; the analysis must retain that context.

Treat disagreement as an observation

Define disagreement at the criterion level. If a schema grader checks format while a model grader checks relevance, pass and fail are not contradictory. They describe different properties. A real conflict occurs when two evaluators claim to measure the same release criterion or when their separate results map to opposite final dispositions.

Preserve four categories: label disagreement, score gap, missing or invalid result, and decision-boundary conflict. A pass-versus-fail conflict deserves attention even when numeric scores differ only slightly. An invalid result is evaluator reliability evidence, not a vote against the candidate.

Never assume consensus is truth. Correlated graders can share prompts, references, parsing code, or model tendencies. Sample some unanimous cases for audit, especially high-risk cases and slices where all graders have limited calibration data.

Normalize evidence before comparing labels

Every grader result needs a common envelope: case ID, immutable output hash, criterion ID, grader ID and version, rubric version, evidence references, raw output, parsed label, confidence or abstention, latency, and error state. The envelope prevents a comparison between different candidate artifacts or rubric revisions.

{
  "caseId": "benefits-qa-204",
  "artifactHash": "sha256:example-placeholder",
  "criterion": "policy_grounding",
  "results": [
    {"grader": "citation-rule-v2", "label": "pass", "evidence": ["policy-2026:44"]},
    {"grader": "semantic-grounding-v5", "label": "fail", "confidence": "medium"},
    {"grader": "pairwise-review-v1", "label": "abstain", "reason": "reference_conflict"}
  ],
  "releaseBoundary": "pass-required",
  "slice": {"locale": "en-IN", "intent": "parental-leave", "risk": "high"}
}

Map scores to labels only through a versioned, calibrated rule. Do not compare a 0.7 helpfulness score with a 70 groundedness score because the numbers look compatible after scaling. Retain raw semantics and declare when two grader outputs cannot be combined.

Validate that every evaluator received the same candidate output and the evidence its contract expects. A disagreement caused by one grader receiving an older policy document is a pipeline defect, not a difficult model case.

Design a genuinely diverse grader panel

Assign evaluators complementary jobs. Use deterministic rules for schema, arithmetic, permissions, and exact references. Use model graders for bounded semantic criteria. Use pairwise comparison when relative preference answers the release question. Add human review for ambiguity and critical consequences.

Diversity comes from methods and evidence, not merely different grader names. Three prompts sent to similar configurations can repeat one failure. A citation-existence rule, a reference-grounding judge, and a human reviewer provide more diagnostic separation because they answer different subquestions.

Blind model identity and randomize pairwise order. Keep grader runs independent: do not show one grader another's rationale before it decides. Otherwise later graders become reviewers of the first result, and the disagreement rate measures anchoring rather than independent evidence.

Rank conflicts by risk and information value

Review capacity is finite, so prioritize without discarding coverage. A simple triage score can combine policy severity, release-boundary impact, novelty, recurrence, and uncertainty. Values and weights must be local, illustrative choices.

from dataclasses import dataclass

@dataclass(frozen=True)
class Conflict:
    severity: int
    crosses_release_boundary: bool
    novel_slice: bool
    recurrence_count: int
    invalid_grader_result: bool

def illustrative_priority(item: Conflict) -> int:
    return (
        4 * item.severity
        + 5 * int(item.crosses_release_boundary)
        + 3 * int(item.novel_slice)
        + min(item.recurrence_count, 5)
        + 2 * int(item.invalid_grader_result)
    )

Do not interpret this score as probability of model failure. It is queue ordering. Keep quotas for routine, low-score, and agreement samples so the review process does not see only dramatic edge cases. Randomize within equal-priority bands to reduce reviewer selection bias.

Show review denominators by slice. Ten conflicts from a high-volume intent can mean something different from ten conflicts among eleven cases in a rare intent. Report both count and rate with uncertainty where applicable.

Cluster by failure mechanism, not wording alone

Automated text clustering can help organize a large queue, but its groups are hypotheses. Start with structured features: criterion, label vector, evidence source, input intent, language, output format, parser status, and known error signatures. Then inspect candidate text or embeddings under the applicable privacy policy.

Useful mechanisms include ambiguous rubric boundary, missing reference evidence, stale policy version, candidate hallucination, judge verbosity preference, position sensitivity, parser mapping error, and unsupported language. A topic cluster such as "refunds" is less actionable unless it reveals why graders diverge.

Name a cluster only after reviewing representative cases and counterexamples. Maintain a miscellaneous bucket; forcing every item into an existing taxonomy can hide the next new failure. Track cluster purity through human review rather than claiming the algorithm discovered ground truth.

Adjudicate output, rubric, and grader behavior

Send selected cases to reviewers without model identity, candidate status, or grader majority. Give them the original input, trusted evidence, output, frozen criterion, and an option to mark rubric gap or insufficient evidence. Preliminary human labels should be independent before adjudication.

The LangSmith annotation queue documentation describes grouping runs, prescribing rubrics, tracking reviewer progress, and collecting pairwise or single-run feedback. Use comparable controls in any review system and preserve preliminary labels plus the final rationale.

Adjudication should assign error ownership: candidate, grader, reference data, rubric, pipeline, or unresolved. More than one owner may apply. This prevents every disagreement from becoming a new model test when the real repair belongs in evaluator code.

Convert findings into targeted coverage

For a confirmed candidate failure, minimize the example while retaining realistic evidence, then add a regression case with severity and mechanism tags. For a grader failure, add the case to that grader's calibration suite. For a rubric gap, create boundary examples that distinguish neighboring labels. For a pipeline defect, add an integration check on artifact and evidence identity.

Deduplicate by underlying source and mechanism. Ten paraphrases of one defect can dominate aggregate scores without expanding coverage. Keep related variants in the same dataset partition. Protect a held-out set from examples used during prompt, rubric, or grader repair.

Record provenance from production trace to adjudication and dataset version. A case can belong to multiple purpose-built suites, but reporting should avoid counting it twice in one release estimate.

Monitor disagreement as a measurement health signal

Publish pairwise confusion tables for graders that share a criterion, plus rates for invalid results, abstentions, and release-boundary conflicts. Slice by task, risk, language, response form, and evidence source. Include grader and rubric versions so changes do not look like unexplained drift.

Track adjudicated ownership over time. A rising global disagreement rate driven by parser errors requires a different response from a stable rate with more genuine boundary cases. Monitor the lag from disagreement detection to adjudication and suite promotion, because an unreviewed queue does not improve coverage.

Use confidence intervals for rates and avoid alerts on tiny slices without a minimum evidence policy. Still allow deterministic critical vetoes: one confirmed unauthorized disclosure can block a release even when interval estimation is not meaningful.

Execute a focused mining plan

1. Define criterion-level disagreement, invalid-result, and release-boundary conflict states before collecting data.
2. Normalize all results around immutable artifacts, versioned rubrics, evidence references, and raw plus parsed verdicts.
3. Run method-diverse graders independently, with identity blinding and randomized pairwise order where relevant.
4. Rank conflicts by local risk and information value while retaining random samples of agreements and low-priority cases.
5. Form mechanism hypotheses, then adjudicate output, rubric, references, grader, and pipeline without exposing the vote.
6. Promote each confirmed issue to the correct regression, calibration, rubric-boundary, or integration suite without contaminating held-out data.
7. Monitor conflict matrices, ownership, slices, uncertainty, and queue throughput by version; stop a release on confirmed critical failures rather than averaging them away.

The payoff is not a lower disagreement rate. It is a clearer map of what the product, rubric, graders, and data still fail to distinguish.