Estimating Retrieval Recall When Relevance Labels Are Incomplete

Retrieval recall is not directly observable when the relevance set is incomplete. A score computed from partial judgments may look precise while hiding relevant documents that no assessor ever saw. The practical response is to define the evaluation population, preserve unknown labels, expand judgments with a defensible design, and report how much the conclusion depends on missing evidence.

This matters most when comparing a new retriever with systems that created the original pool. The new system can discover genuinely relevant chunks outside that pool and still appear worse if every unjudged result is treated as a failure. A trustworthy evaluation separates known relevance, known non-relevance, and unknown status all the way through analysis.

Define the recall estimand first

Start with the exact unit: a query evaluated against a named corpus snapshot. Specify whether relevance belongs to documents, passages, chunks, or canonical facts. If several chunks repeat the same source fact, counting each as a separate relevant item can make duplication look like recall. A canonical document or evidence identifier is often needed alongside chunk identifiers.

Also define rank depth and relevance grade. Binary recall at 20 answers a different question from graded coverage of all evidence required to answer. For a support assistant, a document may be relevant only if it applies to the user's product, region, and effective date. Write those conditions into the judgment rubric before assessors see system identity.

Known-label recall at k can be written as the number of known relevant items retrieved in the first k positions divided by all known relevant items for that query. With incomplete labels, that is a statistic over the known set, not automatically an estimate of true corpus recall. Missing relevance is rarely random, so the distinction must appear in the metric name and report.

Freeze corpus and query splits

Create train, development, and test query groups before tuning chunking, embeddings, filters, pool depth, or fusion parameters. Group related queries by originating conversation, intent template, entity, and source document so paraphrases cannot cross splits. Otherwise, judgment expansion on a development paraphrase can reveal the exact evidence expected by a supposedly held-out test case.

Version the corpus independently. Store source version, chunking configuration, access policy, ingestion time, and canonical document mapping. A changed corpus changes the denominator of possible relevant evidence even when queries stay fixed. Historical comparisons should either rerun against the same snapshot or state that the population changed.

The LangChain retrieval documentation separates the indexing and runtime retrieval components of a RAG pipeline. That separation is useful for evaluation provenance: corpus preparation, candidate retrieval, and answer generation should have distinct configuration identifiers so a recall change can be attributed to the correct stage.

Build a diverse judgment pool

A pool should combine top results from retrieval systems with meaningfully different error patterns. Include a lexical baseline, one or more dense configurations, the current production retriever, and controlled query variants where appropriate. Deduplicate by canonical evidence identity before assessment, but retain every system and rank that contributed an item.

Choose pool depth on development data and annotation capacity, then freeze it for the test run. A shallow pool creates many unknowns; an extremely deep pool consumes judgments on low-probability material. Neither choice is universally correct. The coverage report should show pool depth per contributing system, unique candidates before and after deduplication, and how many judgments were unavailable or abstained.

Assessors should see the query, candidate passage, necessary source context, and rubric, but not the retriever name or rank. Preserve graded labels such as fully_relevant, partially_relevant, not_relevant, and cannot_judge. A forced negative when context is insufficient creates the false negatives the process is intended to control.

Preserve unknown labels in storage

Do not overload false to mean both judged non-relevant and not yet reviewed. The data model needs an explicit state plus provenance:

{
  "queryId": "q-184",
  "corpusSnapshot": "policies-2026-07-01",
  "evidenceId": "policy-42:section-7",
  "label": "unknown",
  "judgmentPool": [
    {"retriever": "lexical-baseline", "rank": 18},
    {"retriever": "dense-candidate", "rank": 4}
  ],
  "assessment": null
}

Keep raw assessor labels, adjudicated labels, rubric version, and correction history as separate records. If a source is later found to be superseded or malformed, append a versioned correction rather than rewriting the original evaluation without trace. This lets an older report be reconstructed from the labels it actually used.

Unknowns should remain visible in metric inputs. A metric implementation that accepts only booleans should receive a filtered judged set or an explicit sensitivity scenario, never an implicit conversion performed during serialization.

Report recall with judgment coverage

For every query and slice, publish known-label recall at k beside judged coverage at k, defined as the fraction of returned ranks with a relevance judgment. Also show the known relevant count. A query with one known relevant item and a query with twenty should not look equally well characterized merely because each has full coverage of its returned top five.

The Ragas metrics documentation distinguishes retrieval-oriented context measures from answer-oriented measures. Use that conceptual boundary: retrieval recall diagnoses whether evidence entered the candidate set, while answer faithfulness or correctness evaluates later behavior. An answer score cannot repair an unidentified retrieval denominator.

Add a sensitivity table. One scenario can treat all unknown top-k items as non-relevant; another can use newly sampled judgments; a deliberately conservative scenario can examine how many unknown relevant items would be needed to reverse the decision. These are assumptions, not interchangeable estimates. Label them plainly.

Expand judgments with probability sampling

Judging every corpus item for every query is usually infeasible. A probability sample from a defined candidate frame can estimate missing relevance if inclusion probabilities are known. Stratify by query slice, rank band, retriever contribution, and disagreement pattern. Oversample high-ranked unknowns and candidate-only results, but retain the probability attached to each selection.

from collections import defaultdict

def weighted_relevant_total(sampled_rows):
    totals = defaultdict(float)
    for row in sampled_rows:
        if row["label"] == "relevant":
            totals[row["query_id"]] += 1.0 / row["inclusion_probability"]
    return totals

The function is illustrative analysis code. A production estimator must combine already known relevant items with sampled unknowns, account for strata and finite populations, and calculate uncertainty using the actual sampling design. A ratio of weighted retrieved relevance to weighted total relevance can be unstable when few positives are observed, so report its interval and the unweighted counts.

Be precise about the frame. Sampling from the union of top results estimates relevance within that union, not across the entire corpus. Calling the result corpus recall would overstate what was measured. Expand the frame with random corpus samples or targeted source strata only when that wider claim matters and the annotation budget supports it.

Detect pooling bias and evaluation leakage

Compare judged coverage by system at each rank. If the production retriever contributed most pool documents, it will usually have better coverage than a novel candidate. Report candidate-only unknowns and have them assessed through the same blind process before final comparison. Do not let one team hand-pick only the candidate results that look promising.

Keep pool development separate from final scoring. Tuning the retriever after inspecting test unknowns converts those judgments into development data. Freeze the test pool and candidate configuration, complete blind judgments, and evaluate once. Corrections for rubric errors are legitimate, but they must apply to every system and be documented.

Leakage can also enter through synthetic queries derived from the exact wording of target chunks. Split source documents before generating queries, and keep generators from seeing sealed test documents when the goal is generalization. Near-duplicate queries and chunks require hash, semantic, and provenance checks rather than filename comparison alone.

Compare systems with paired uncertainty

Retrievers are evaluated on the same queries, so comparisons should remain paired. Compute each query's metric difference, then use a paired bootstrap or another method suitable for the sampling and clustering design. Resample conversations or source groups when multiple queries share them. Treating correlated paraphrases as independent narrows intervals without adding independent evidence.

Show macro averages so large judgment pools do not dominate every result, and inspect risk slices separately. A weighted production estimate may be useful, but retain unweighted slice metrics for rare, consequential intents. Predeclare whether a critical slice is a veto, a review trigger, or part of an aggregate objective.

If the interval spans the release boundary, the result is inconclusive. Do not select the favorable endpoint. Identify whether uncertainty comes from too few queries, incomplete judgments, rare relevant items, or unstable labels, then collect evidence targeted at that cause.

Attribute errors before changing retrieval

Review false negatives by failure class: missing source in the index, unsuitable chunk boundary, lexical mismatch, embedding confusion, metadata filter, stale version, permission filter, or rank truncation. Keep unknown as a separate queue. Mixing indexing and ranking failures leads teams to retune embeddings for documents that were never eligible candidates.

Use ablations on development queries to isolate changes. Compare the same index with and without a filter, the same candidates under alternate rankers, and alternate chunking with a stable retrieval method. Evaluate each ablation against the same judgment revision when possible. If the candidate set changes substantially, expand judgments before interpreting a score movement.

Latency and quality should be reported together. Deeper retrieval may increase measured recall while raising retrieval, reranking, and prompt costs. Capture candidate depth, returned depth, timeout behavior, and cache state so a quality improvement is not credited to a configuration that cannot satisfy the product's latency budget.

Execute a decision-ready recall plan

1. Freeze the query groups, corpus snapshot, evidence identity rule, relevance rubric, and rank depths.
2. Build a blind pool from diverse retrievers and record every contribution, rank, and deduplication decision.
3. Preserve relevant, non-relevant, unknown, and abstained states without boolean coercion.
4. Report known-label recall with judged coverage, known relevant counts, and candidate-only unknowns by slice.
5. Expand important gaps through a documented probability sample and calculate design-aware uncertainty.
6. Run paired comparisons and sensitivity analysis against a release rule written before final judgments are opened.
7. Block the decision when missing labels can plausibly reverse it; assign judgment expansion or targeted collection instead of manufacturing certainty.

Incomplete labels do not make retrieval evaluation impossible. They make the observation process part of the system under test. Once pool construction, unknown status, sampling, and uncertainty are visible, recall becomes evidence a release owner can reason about rather than a deceptively exact fraction.