Confidence Intervals for LLM Eval Release Gates

An LLM eval score is an estimate, not a property engraved into the model. Change the sampled cases, grader decisions, or execution randomness and the score can move. A release gate that compares only point estimates turns ordinary sampling variation into false confidence.

Confidence intervals make the decision rule honest. They show whether the evaluation can distinguish a material regression, a meaningful improvement, or an unresolved region. The interval must match the estimand, pairing, clustering, weighting, and grader design; adding a generic error bar after aggregation is not enough.

State the Decision and Estimand First

Define what quantity the gate uses. It might be candidate pass rate on a production-weighted population, paired change in rubric score, severe-failure risk within a protected slice, or difference in tool-task completion. Specify the population, unit of analysis, scoring rule, aggregation, and material effect before calculating sample size.

The LangSmith evaluation documentation organizes resources for running and analyzing evaluations. Whatever execution system you choose, bind each run to a frozen dataset, candidate, baseline, grader bundle, and analysis plan so the interval answers the question written before results were visible.

Preserve Provenance and Pairing

Each row needs a stable example identity, source and family lineage, collection window, transformation history, slice labels, reference and rubric versions, and exposure status. Each score needs candidate identity, prompt or agent configuration, execution timestamp, dependency versions, grader identity, and any retry or abstention status.

Evaluate baseline and candidate on the same examples whenever the comparison permits. Pair by example identity, not row order. Exclude a pair only through predefined rules, such as both executions being unavailable because a shared external dependency failed. Candidate-only parse failures are outcomes, not missing data to discard.

Related turns, variants, and examples from one document are not independent units. Give them a cluster identifier and use cluster-level resampling or aggregation. Otherwise a large conversation can create an artificially narrow interval.

Prevent Leakage Before Estimating Precision

A precise estimate on a contaminated holdout is precisely misleading. Check exact and near duplicates, shared source families, prompt-development records, fine-tuning data, retrieval content, and grader-calibration cases. Keep transformed siblings in one split. Record who has inspected sealed failures and when their status changed.

Do not choose the candidate after repeatedly querying the same release holdout without accounting for adaptation. Confidence intervals for the final pair do not capture selection over many attempted prompts or models. Use development data for iteration and reserve a fresh or tightly controlled confirmation set for the release claim.

Data exclusions selected after seeing outcomes also bias the interval. Freeze corruption, infrastructure, and abstention handling rules before comparing systems.

Use Paired Differences for Candidate Comparisons

For a binary criterion, encode each example's baseline and candidate result as 0 or 1. The paired difference is candidate - baseline, taking values -1, 0, or 1. Its mean estimates the change in pass probability over the sampled population. Pairing uses the fact that hard and easy cases affect both systems.

For continuous rubric scores, use the within-example score difference. If weights represent a target population, apply the same documented weights to both sides and use an uncertainty method compatible with the weighting design.

The most informative companion table counts improved, unchanged, and regressed pairs by slice. Two candidates can have the same mean change while one makes many small improvements and the other introduces a few severe regressions.

Match the Interval to the Outcome

For one unpaired binary pass rate, a Wilson or exact binomial interval is generally more defensible than the simple normal approximation near extreme rates or with small samples. For a paired binary comparison, use a method based on discordant pairs or a paired bootstrap. For bounded or ordinal scores, resample paired units and recompute the complete statistic.

If sampling is stratified, either report stratum-specific estimates or apply known target weights during every resample. If examples are clustered, resample clusters. If the release gate uses a percentile or worst-slice statistic, bootstrap that exact statistic rather than an easier mean.

No interval method rescues a sample that omits the product risk. Statistical coverage and behavioral coverage are separate requirements.

Implement a Reproducible Paired Bootstrap

The following example resamples complete paired rows and returns a percentile interval for the mean candidate-baseline change. It is suitable as a transparent starting point for independent pairs, not a universal implementation for weighted or clustered data.

from random import Random

def paired_bootstrap(baseline, candidate, repeats=10_000, seed=17):
    if len(baseline) != len(candidate) or not baseline:
        raise ValueError("paired non-empty scores required")

    differences = [c - b for b, c in zip(baseline, candidate)]
    rng = Random(seed)
    estimates = []
    for _ in range(repeats):
        sample = [differences[rng.randrange(len(differences))]
                  for _ in differences]
        estimates.append(sum(sample) / len(sample))

    estimates.sort()
    lower = estimates[int(0.025 * repeats)]
    upper = estimates[int(0.975 * repeats)]
    return {
        "estimate": sum(differences) / len(differences),
        "lower": lower,
        "upper": upper,
        "seed": seed,
    }

The nominal 95 percent level and resample count are illustrative choices. Predeclare them, verify index handling, and test the function on known synthetic cases. For clustered data, sample cluster IDs and include all rows in selected clusters.

Plan Sample Size Around Material Harm

Do not ask only, "How many eval cases do we need?" Ask what regression size must be detectable in the slice that drives the decision. Required evidence depends on outcome variance, pairing correlation, confidence level, desired power, slice prevalence, clustering, and tolerated harm.

Pilot data can estimate variance, but a pilot selected from known failures will not represent the release population. Use conservative assumptions and state them. Rare critical outcomes may require targeted sampling, longer collection windows, or a veto rule based on confirmed failures rather than a precise rate estimate.

More rows are not always more independent evidence. Ten transformations from one source case improve debugging breadth but should not count like ten unrelated users for statistical precision.

Include Grader and Annotation Uncertainty

The basic interval treats recorded scores as correct. Human disagreement and model-grader error violate that assumption. Calibrate annotators on shared cases, measure pre-adjudication agreement, preserve abstentions, and adjudicate severe disagreements. Report how many final labels required adjudication.

Deterministic graders should handle exact properties such as parsing, field constraints, tool arguments, and reference membership. Model graders should use narrow rubrics, frozen configurations, and independent human calibration. Audit false-pass and false-fail patterns by slice.

For high-stakes gates, run sensitivity analysis: recompute the candidate-baseline effect after plausible disputed labels are reversed, or use repeated grading to estimate measurement stability. Do not fold disagreement into sampling error without an explicit model; label it as a separate uncertainty source.

Protect Critical Slices and Intersections

Predeclare slices tied to product risk: language, domain, user role, tool path, context length, policy category, retrieval condition, or consequence severity. Report estimate, interval, denominator, abstentions, and integrity status for each. A top-line interval can be narrow while a rare slice remains unresolved.

Testing many slices increases the chance that at least one appears extreme by chance. Decide whether the gate controls a family of comparisons, uses hierarchical analysis, or treats selected critical slices as separate safety claims. Do not add slices only after finding a favorable or alarming pattern and then present them as confirmatory.

Intersections often reveal failures but have smaller samples. Use them for investigation and targeted collection; avoid confident release claims from a denominator that cannot support them.

Translate Bounds Into Ship, Hold, or Review

Use an interval against a predeclared non-inferiority margin or minimum improvement. Suppose an illustrative policy allows at most a two-percentage-point regression on a reversible routine slice. Shipping would require the lower bound of candidate - baseline to remain above -0.02. The value is an example, not a universal standard.

For improvement claims, require the interval to clear the minimum effect worth acting on, not merely zero. For critical failures, a confirmed event may trigger a veto regardless of the aggregate interval. If the interval crosses both acceptable and unacceptable regions, return review or collect more data; do not relabel inconclusive evidence as no regression.

Release records should include the analysis version, all thresholds, missing slices, integrity caveats, grader changes, and accountable exceptions.

Failure Modes and Tradeoffs

Very wide intervals make a gate indecisive but honestly expose weak evidence. Very large datasets improve precision while costing more and becoming stale. Overweighting rare risks improves sensitivity but changes the aggregate's interpretation. Multiple-comparison corrections reduce false alarms but can reduce power for genuine slice regressions.

Bootstrap intervals are flexible but inherit the sampling design and may behave poorly with tiny or highly discrete samples. Parametric methods make stronger assumptions. Choose transparently, run sensitivity checks, and focus on whether the decision changes under reasonable alternatives.

Operational Checklist

• Define the population, unit, estimand, and material effect before execution.
• Freeze dataset, baseline, candidate, grader, and analysis versions.
• Pair results by stable example identity.
• Preserve cluster and source-family lineage.
• Complete leakage and adaptive-overfitting controls before scoring.
• Use an interval suited to binary, paired, weighted, or clustered data.
• Report denominators, abstentions, and integrity by protected slice.
• Keep grader uncertainty separate from sampling uncertainty unless modeled.
• Predeclare treatment of missing, invalid, and disputed outcomes.
• Return ship, hold, or review when bounds cross policy regions.
• Store code, seeds, inputs, and outputs needed to reproduce the interval.

Action Plan: Replace One Point Gate

Select one release metric currently judged by a point estimate. Rewrite it as a paired candidate-baseline estimand, define the largest acceptable regression from product consequence, and identify the slices that can veto the aggregate. Audit data lineage and leakage before running both systems on the same examples.

Implement and test the matching interval, add grader sensitivity analysis, and produce a decision table for ship, hold, and review. Trial the policy on historical runs without changing past decisions, then freeze it for the next release. The gate is ready when an inconclusive interval produces an explicit next action instead of a hopeful interpretation.