RAG Recall Labeling and Metric Tradeoff Interview Scenarios

Senior RAG QA interviews should expose whether a candidate understands that retrieval metrics depend on human judgments, corpus state, and the answer task. Recall is not a self-validating number. A missing document can mean a search failure, a stale index, an incomplete label set, or an assumption that the question had only one valid source.

Use these scenarios to show how you would build relevance evidence, choose cutoffs, diagnose failures, and prevent metric optimization from damaging end-to-end answers. Name the query, claim, corpus snapshot, labels, retrieved ranks, and downstream consequence in every investigation.

Anchor Recall to the User Task

The official Ragas metric catalog distinguishes retrieval and response evaluation concepts. A credible interview answer starts by identifying which evidence the answer must contain and which retrieval behavior makes that evidence available.

An interviewer should reject answers that optimize a retrieval score without describing what happens to groundedness, completeness, latency, and context use.

Define Relevance and the Unit of Recall

1. Why must recall be defined against required answer evidence?

Topic similarity is not enough. For a refund question, the answer may require eligibility, deadline, and exception evidence from different passages. I would decompose the expected response into claims, label which passages support each claim, and measure whether retrieval exposes the required set. Failure evidence is a fluent answer missing a required claim even though a generic topical document ranked highly.

2. How would you choose between document-level and passage-level relevance?

Match the label unit to what the retriever returns and the generator consumes. Document labels can hide that only one small section is useful, while passage labels can punish a system that retrieves a larger parent containing the same evidence. I would preserve parent-child identifiers and compute both evidence availability and retrieval granularity where architecture requires it. The choice should not make one implementation artificially correct.

3. Why should contradictory evidence receive its own label?

A passage can be highly topical while asserting the opposite of current policy. Binary relevant/not-relevant labels may reward its retrieval. I would label support, contradiction, superseded status, and applicability, then test whether ranking and generation resolve conflicts using effective dates or source authority. A retrieved contradiction is not automatically bad, but unmarked conflict creates unsafe context and misleading recall.

Build and Govern Relevance Labels

4. How would you create qrels for a large changing corpus?

Start with representative and risk-enriched queries, pool candidates from diverse retrievers, and have qualified reviewers label claim-level relevance with source evidence. Store corpus snapshot, document revision, query interpretation, grade, and adjudication state. New systems can surface unpooled candidates for review. The set remains versioned because relevance changes when documents, policies, or user intent definitions change.

5. Why is pooling from one retriever dangerous?

The judgment pool inherits that retriever's blind spots. A different system may find valid evidence that was never labeled and be penalized as if it returned noise. I would pool across lexical, dense, hybrid, and known production variants where available, then judge newly retrieved unassessed documents during comparison. System identity stays hidden from reviewers to prevent preference from contaminating labels.

6. How would you label a passage that supports only half of a compound question?

Break the query into required claims and attach relevance per claim. Mark the passage as partially useful and not independently sufficient. The metric can then assess claim coverage rather than pretending the passage is fully relevant or irrelevant. In the answer evaluation, verify that the generator combines compatible evidence. This avoids rewarding a retriever that repeatedly covers the easy half while missing the deciding condition.

7. Why retain annotator rationales with relevance grades?

Rationales reveal whether reviewers used topic overlap, source authority, factual support, or sufficiency. During disagreement, the team can repair the rubric instead of averaging unexplained labels. Rationales also help diagnose apparent retrieval regressions after document revisions. They should cite claim and source spans, avoid personal data, and remain linked to the exact query and corpus snapshot.

Select Metrics and Cutoffs

8. How would you choose k for recall at k?

Evaluate candidate cutoffs through retrieval and answer stages. For each k, inspect required-claim coverage, irrelevant and contradictory context, latency, context budget, and answer quality. The selected k may vary by query class if routing is testable. Any threshold proposed during an interview is illustrative until measured on the organization's traffic and constraints; larger k is not free recall.

9. Why report precision or context relevance beside recall?

A retriever can obtain high recall by returning much of the corpus. Precision-like evidence shows how much distracting material accompanies the useful passages. In RAG, distraction can change source selection, latency, and citations. I would report both by slice and inspect answer outcomes. Neither metric alone proves success: sparse high-precision results may miss required evidence, while broad high-recall results may overwhelm the generator.

10. How would you compare binary recall with graded-gain metrics?

Binary recall answers whether labeled relevant evidence appeared. Graded metrics can value highly sufficient passages above weakly supportive ones and account for rank. I would compute both when the rubric supports grades, then inspect cases where they disagree. A system retrieving many marginal passages may look acceptable under binary counting but poorly under graded utility; the reverse can happen when one decisive passage satisfies the task.

11. Why should answer correctness not be used as a substitute for retrieval recall?

The model may answer from prior knowledge despite missing corpus evidence, or fail to use correctly retrieved passages. Capture retrieval independently, then evaluate grounded answer claims and attribution. A wrong answer with good recall implicates generation or conflict handling; a correct unsupported answer still violates a corpus-grounded contract. Stage metrics make remediation ownership possible without claiming they are independent in user impact.

Handle Incomplete and Ambiguous Judgments

12. How would you detect that relevance labels are incomplete?

Sample unjudged top-ranked results from multiple systems, especially documents unique to a challenger. Review semantic near-duplicates, citations from correct answers, and user-selected sources. A high rate of newly validated evidence indicates pool incompleteness. Recompute affected metrics under a new qrel version and report the change. Do not retroactively alter prior release artifacts without preserving their original label snapshot.

13. Why is treating every unjudged document as irrelevant risky?

It creates a false negative label by default and favors systems resembling the one that built the pool. I would distinguish judged irrelevant from unjudged, report judgment coverage at each cutoff, and prioritize unique high-rank items for review. Depending on the decision, use lower and upper bounds or exclude insufficiently judged queries. The uncertainty must remain visible rather than being coerced into precision.

14. How would you compare two retrievers when the judgment pool is incomplete?

Create a blinded union of their top results, add candidates from a baseline, and adjudicate unique and disputed documents. Compare on queries with adequate judgment coverage, then run sensitivity analysis for remaining unjudged items. Inspect wins by claim and risk slice. I would avoid declaring a winner if the result depends on assumptions about a large unjudged region; the next action is labeling, not deployment.

def judged_recall(retrieved_ids, relevant_ids, k):
    """Recall over adjudicated relevance only; report pool coverage separately."""
    top_k = set(retrieved_ids[:k])
    if not relevant_ids:
        return None
    return len(top_k.intersection(relevant_ids)) / len(relevant_ids)

Diagnose Retrieval Failures

15. How would you investigate zero recall for one product language?

Freeze the query, locale, corpus revision, index configuration, and retrieved candidates. Verify that translated or native documents were ingested, chunked, and permission-visible. Compare lexical and embedding retrieval, inspect language normalization, and relabel with fluent reviewers. The failure may be missing content rather than ranking. A fix is verified on unseen queries and adjacent languages, not only the reported sentence.

16. Why analyze recall by query intent rather than only globally?

Navigational queries, broad synthesis, entity lookup, and policy exception questions need different evidence patterns. Global averages can hide systematic failure on a small but consequential intent. I would maintain an intent taxonomy with reviewer guidance, report confidence or sample sufficiency, and inspect failure clusters. Routing by intent is justified only if intent classification errors are evaluated as part of the pipeline.

17. How would you distinguish an indexing defect from a ranking defect?

Check whether the required passage exists in the source corpus, transformed corpus, index, and candidate set. Query the index directly by identifier or distinctive text, then compare pre-rerank and post-rerank positions. Missing from the index points to ingestion, permissions, or freshness; present but low-ranked points to retrieval features or query representation. Preserve each stage's candidate IDs and scores as evidence.

18. Why inspect duplicate passages in recall failures?

Near-duplicate chunks can occupy top ranks, making apparent coverage look broad while one claim dominates the context. Group candidates by source and semantic family, then compute unique claim and source coverage. Deduplication may improve diversity but can remove legitimately repeated authoritative evidence. Test the rule on versioned documents, templates, and mirrored content before using it as a blanket ranking fix.

Make Release Tradeoffs Explicit

19. How would you set a release gate for a high-risk RAG slice?

Define required claims and unacceptable contradictory sources, then require adequate label coverage and inspect every critical miss. Use stable and fresh query sets, report retrieval and answer evidence together, and establish a deterministic veto for absent mandatory policy. Numeric cutoffs are organization-specific and should be labeled illustrative. A passing aggregate cannot override repeated misses in the designated critical slice.

20. Why might higher recall reduce end-to-end answer quality?

Increasing k can add stale, redundant, or conflicting context that competes with decisive evidence. I would run an ablation across cutoffs while holding query, corpus, model, and prompt fixed, then compare claim support, citation validity, latency, and abstention. If answer quality drops, options include reranking, diversity constraints, metadata filters, or query decomposition rather than simply reverting the retriever.

21. How would you present a retrieval regression to a release board?

Show affected user intent and risk, corpus and qrel versions, missing claims, retrieved ranks before and after, judgment coverage, and downstream answer impact. State whether the cause is indexing, ranking, labeling, or unknown, plus the containment and retest plan. Recommend release, limited rollout, or block decisively. A metric delta without example-level evidence is not enough for that decision.

Decide on Evidence, Not a Favorite Metric

A strong candidate treats labels as versioned hypotheses about relevance and actively searches for where those hypotheses are incomplete. They connect recall to required claims, report unjudged candidates, and preserve stage artifacts so indexing and ranking failures do not collapse into one score.

The final decision is simple: do not approve a retrieval change until critical evidence is available to the answer pipeline and the label pool is complete enough to support the comparison. Optimize the user task, then use metrics to explain why the decision is defensible.