Debugging Flaky LLM-as-a-Judge Scores in CI

Flaky LLM-as-a-judge scores in CI should not be treated as random test noise. They are evidence that the evaluation decision is more precise than the measurement system supports. Preserve the candidate outputs, rerun only the grading layer, and identify whether variation enters through judge inference, rubric interpretation, data ordering, parsing, transport, or the release calculation.

Do not respond by raising retries until the build turns green. Retries can convert an explicit failure into an undocumented pass policy. The goal is a gate whose repeated decisions are explainable, whose uncertain cases are escalated, and whose hard safety checks do not depend on a probabilistic vote.

Freeze the failing CI evidence

Capture the exact application output once and prevent the system under test from regenerating it during grader diagnosis. Store dataset revision, case ID, input, reference data, candidate output, application version, rubric, grader prompt, grader identifier, sampling configuration, response schema, raw grader response, parsed value, request attempt, cache status, and final aggregate. Hash large text fields so accidental mutation is visible.

Separate three timestamps: application generation, grader request, and gate calculation. A CI rerun may regenerate a different candidate answer and then blame the grader for a legitimate score change. It may also evaluate the same answer after a backend or prompt alias changed. The artifact must establish what stayed constant.

Recalculate the gate locally from stored per-case scores. If the reproduced decision differs without any model call, the defect is in filtering, weighting, missing-value handling, rounding, or parallel result collection. That is usually the cheapest branch to eliminate.

Build a variance root-cause tree

Create branches for candidate data, judge execution, rubric and prompt, response parsing, orchestration, and decision policy. Candidate-data causes include regenerated outputs, unstable reference lookup, shuffled pairwise positions, or leaked metadata. Judge causes include sampling variation, changed configuration, and transient refusal behavior. Rubric causes include overlapping labels, hidden tie policy, and criteria that require unavailable context.

Parser causes include accepting prose around JSON, coercing missing fields to zero, locale-sensitive numbers, and mapping unknown labels inconsistently. Orchestration causes include rate-limit retries with altered requests, lost results, mixed cache entries, and cancellation races. Policy causes include a cutoff placed inside normal grader disagreement, unpaired aggregation, or a retry-until-pass rule.

For each leaf, specify a trace field and an isolation experiment. "The judge is random" is not a testable leaf. "The same prompt hash and candidate hash produce both pass and fail labels under uncached serial repeats" is.

Design a controlled repeat matrix

Start with fixed grader inputs and run controlled repetitions. Vary one dimension at a time: serial versus CI concurrency, cache enabled versus deliberately bypassed, baseline-first versus candidate-first ordering, old versus current parser, and current rubric versus a clarified development rubric. Preserve every raw response; summary statistics alone cannot reveal schema drift or explanation-label contradictions.

Use paired evaluation when comparing a release candidate with a baseline. Each repetition should score both outputs for the same case under the same declared conditions. Randomize or balance pairwise positions while retaining the position assignment in the trace. Group related turns or paraphrases so they are not counted as independent evidence.

The LangSmith evaluation concepts distinguish code, human, LLM-as-judge, and pairwise techniques and explicitly frame model outputs as non-deterministic. That is a reason to model disagreement, not a reason to accept an unstable build decision.

Instrument the grader as a test subject

Wrap the grader call so every attempt has a stable case ID and configuration fingerprint. The following Python shape is illustrative; the important feature is immutable input and raw-output retention:

from dataclasses import dataclass
from hashlib import sha256
import json

@dataclass(frozen=True)
class JudgeAttempt:
    case_id: str
    repetition: int
    request_hash: str
    raw_response: str
    parsed_label: str | None
    transport_status: str

def canonical_hash(payload: dict) -> str:
    encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return sha256(encoded.encode("utf-8")).hexdigest()

Record whether an attempt was served from cache and whether a retry reused an idempotency key. Never discard malformed or timed-out responses from the denominator without a declared policy. A grader that fails to return a usable judgment has produced an operational result, not a neutral absence.

The OpenAI graders API reference exposes grader definitions and sampling parameters for model-based graders alongside deterministic grader types. Pin every available field that affects execution, but do not state that a seed or low temperature guarantees identical output. Verify repeat behavior empirically for the exact evaluation contract.

Isolate inference, rubric, and parser effects

To test inference variation, send byte-identical requests from stored fixtures and compare raw outputs. To test rubric ambiguity, have blinded humans label the disagreement set and mark cases where multiple labels are defensible. To test parsing, replay all stored raw responses through old and new parsers without calling a model. To test candidate contamination, replace candidate text with controlled minimal examples that exercise each label boundary.

Run deterministic assertions before model grading. Schema validity, required citations, prohibited data, tool authorization, and exact business invariants should fail through code when code can decide them. This removes needless judge surface and makes severe violations reproducible.

For pairwise graders, swap A and B while keeping content fixed. A preference that follows position is evidence of positional sensitivity. For scalar rubrics, permute irrelevant metadata and delimiter styles. Material score movement under a semantically irrelevant change identifies a prompt or parser weakness rather than an application regression.

Calibrate labels against human decisions

Build a blinded calibration panel that includes clear passes, clear failures, boundary cases, adversarial candidate text, and representative slices. Retain individual human labels before adjudication. The judge should be assessed by criterion and severity, because agreement on formatting does not establish agreement on factual support or safety.

Write rubric criteria as observable questions. State what evidence the grader receives, how ties and insufficient context are handled, and whether one severe defect overrides otherwise strong quality. If a criterion needs product or domain context unavailable in CI, route it to a reviewer instead of inviting the judge to infer policy.

Keep calibration data separate from examples used to revise the grader prompt. Every prompt change creates a new evaluator version and requires rerunning the sealed panel. Human review is not a decorative final layer; it defines ambiguous boundaries, audits systematic judge errors, and owns exceptions.

Replace brittle thresholds with a decision policy

A single mean cutoff often oscillates because it ignores per-case disagreement and severity. Define deterministic vetoes first, then stable quality evidence, then an explicit inconclusive state. Near the boundary, gather a declared number of additional repetitions or request human review. Do not repeatedly sample until one favorable result appears.

Illustrative policy configuration:

{
  "illustrativeThresholds": true,
  "deterministicVetoes": ["schema-invalid", "secret-exposed", "unauthorized-action"],
  "modelJudge": {
    "repetitionsPerCase": 3,
    "maximumCriticalLabelDisagreement": 0,
    "minimumPairedWinRateLowerBound": 0.5
  },
  "inconclusive": {
    "action": "human-review",
    "retryUntilPass": false
  }
}

These numbers are examples, not recommendations. Select thresholds from calibration evidence, sample dependence, consequence of error, and available review capacity. Publish missing and malformed judgment rates beside quality results.

Define ownership and safety boundaries

The product team owns the meaning and severity of quality criteria. The evaluation team owns datasets, rubrics, calibration, and gate calculations. The platform team owns request reliability, caching, rate limits, and artifact retention. Model providers own their service behavior, but the application team remains responsible for designing a gate robust to documented uncertainty. Release engineering owns the enforcement path and exception expiry.

Treat candidate output as untrusted input. Delimit it from grader instructions, deny the judge unnecessary tools and network access, validate structured outputs, and redact sensitive fields before sending them. Include prompt-injection cases in calibration. A grader persuaded by the candidate to award a high score is a security defect in the evaluation system.

Prove the CI gate is stable

Replay a sealed panel across multiple clean CI runs and compare final decisions, not only average scores. Force transport failures, malformed responses, empty subsets, duplicate case IDs, and partial cancellation. The gate must fail closed or become explicitly inconclusive according to policy, with enough artifacts to reproduce the calculation.

Before enabling a changed grader as a blocking check, run it in shadow mode and review disagreements with the current gate. Promotion requires acceptable calibration, operational error handling, cost and latency budgets, and no unresolved severe slice regression. Keep a rapid switch back to the previous evaluator version.

Execute the stabilization plan

Freeze one failing run, reproduce the aggregate from stored scores, build the root-cause matrix, repeat byte-identical grader requests, replay parsers, test position and irrelevant-input sensitivity, and adjudicate the boundary set. Then revise the smallest responsible layer and validate it on a sealed panel.

A stable judge pipeline does not promise identical prose from every call. It produces release decisions that remain defensible under measured variation, preserves uncertainty instead of hiding it, and sends genuinely ambiguous cases to the people who own the product judgment.