How to Test RAG Retrieval: QA Guide for Accurate Context

A benefits assistant cited the correct employee handbook but answered a parental-leave question from a chunk describing unpaid caregiver leave. Both passages shared “leave,” “family,” and “twelve weeks,” so the top result looked plausible. The generator did exactly what it was asked to do with the wrong context.

This is why retrieval must be tested separately from generation. A fluent final answer cannot tell you whether the required evidence was absent, buried below the context cutoff, filtered out, or contradicted by a newer source.

Define the retrieval contract

Write down what the retriever must return for each query type. For a policy assistant, a successful result might require at least one chunk containing the applicable rule, from the correct jurisdiction and effective date, within the first eight candidates. It may also require exclusion of superseded documents.

Separate stages in the contract:

• Ingestion turns source documents into addressable chunks.
• Candidate retrieval finds a broad set using lexical, vector, or hybrid search.
• Metadata filters enforce tenant, locale, permissions, and time boundaries.
• Reranking orders candidates for the context window.
• Context assembly removes duplicates and fits the token budget.

Record configuration versions for every stage. If chunking and embedding change together, you may know the candidate improved but not why. Controlled experiments change one major variable at a time, then test the integrated configuration before release.

Create query-to-evidence judgments

A retrieval case needs evidence labels, not an ideal generated answer. Store required passages or facts, acceptable alternatives, harmful sources, and metadata constraints.

{
  "queryId": "leave-ca-birth-parent",
  "query": "How much leave do I get after giving birth in California?",
  "mustRetrieve": [
    { "factId": "ca-birth-leave-duration", "minGrade": 2 }
  ],
  "acceptableDocs": ["leave-policy-us-2026", "ca-addendum-2026"],
  "mustNotUse": ["leave-policy-us-2024", "caregiver-unpaid-faq"],
  "filters": { "country": "US", "state": "CA", "effectiveOn": "2026-05-01" },
  "tags": ["policy", "temporal", "hard-negative"]
}

Label at passage level when chunking is under evaluation. A document-level label can produce a false pass even when the returned chunk omits the sentence needed to answer. Use graded relevance: direct answer, supporting context, tangential, irrelevant, and harmful.

Build cases from search logs, unanswered questions, user reformulations, and expert-authored boundaries. Include queries that should return no answer, because forced retrieval on unsupported questions creates confident hallucination. Pool results from multiple candidate systems for labeling so the current retriever does not define its own ground truth.

Verify ingestion and chunk integrity first

Retrieval quality cannot recover content that was never indexed correctly. Add deterministic ingestion tests for document counts, stable IDs, source URLs, checksums, permissions, locale, effective dates, and deletion propagation.

Inspect chunk boundaries around tables, headings, lists, and conditional clauses. A chunk containing “employees receive twelve weeks” without the preceding eligibility condition is dangerous even though it matches well. Track parent document, section path, character offsets, and neighboring chunks so investigators can reconstruct context.

Use fixtures that expose common parser failures: scanned pages with OCR noise, repeated headers, multi-column PDFs, footnotes, tables, code blocks, and empty sections. Assert that protected documents stay in the proper tenant or access namespace. Verify that an updated policy invalidates or versions the old chunks rather than silently leaving both active.

Chunk size is a tradeoff. Smaller chunks improve targeting but may lose conditions; larger chunks preserve context but dilute relevance and consume more tokens. Evaluate sizes against the same passage-level dataset instead of selecting by intuition.

Measure candidate recall and reranking separately

Candidate recall@k asks whether at least one required evidence item appears in the broad candidate set. This is the primary metric for the first stage because a reranker cannot restore missing evidence. Measure at the actual candidate count sent to the reranker.

For final ordering, use hit rate, reciprocal rank, precision@k, or normalized discounted cumulative gain depending on the labels. Measure harmful-source rate and stale-source rate explicitly. A system can improve relevance while increasing retrieval of an obsolete policy that uses similar terms.

The following Python is a small evaluation helper, not a framework API:

def recall_at_k(ranked_fact_ids, required_fact_ids, k):
    required = set(required_fact_ids)
    found = required.intersection(ranked_fact_ids[:k])
    return len(found) / len(required) if required else 1.0

def contains_harmful(ranked_doc_ids, blocked_doc_ids, k):
    return bool(set(ranked_doc_ids[:k]).intersection(blocked_doc_ids))

If a question requires two independent facts, measure both. “One relevant chunk found” is insufficient when eligibility and duration live in different sections. For no-answer cases, measure whether the system abstains from assembling misleading context, not recall.

Test filters, freshness, and access boundaries

Metadata filters are part of retrieval correctness. Create paired cases differing only in tenant, role, locale, product version, or effective date. Confirm that the correct evidence is returned and disallowed evidence is absent from candidates, reranker input, context, logs, and caches.

Test missing and malformed metadata. Decide whether the system fails closed, falls back to a safe global source, or asks for clarification. Silent removal of a filter can create a high-scoring but unauthorized result.

For freshness, use controlled document versions with overlapping language. Query before, on, and after an effective date. Verify deletion and update service-level objectives. If ingestion is asynchronous, measure the delay from source publication to searchable evidence and define what the assistant does during the gap.

Cache tests should include permission changes and policy updates. A cached candidate list must not outlive the access or freshness constraint that made it valid.

Test query rewriting at these boundaries as well. A rewriter may remove a state, product version, or negation that the filter cannot reconstruct. Save both original and rewritten queries, then assert that mandatory constraints survive. For ambiguous queries, compare forced retrieval with a clarification path. Retrieving broadly can look like higher recall while increasing the chance that an inapplicable source reaches the generator.

Evaluate assembled context and citations

The generator sees assembled context, not raw rankings. Test deduplication, ordering, truncation, token budgeting, and source attribution. Record which chunks were dropped and why. Required evidence at rank eight still fails if the context builder only fits six chunks.

Measure context coverage as the fraction of required facts present in the final prompt. Measure context precision as how much supplied material is relevant enough to support the answer. Excess irrelevant context can distract the model and increase cost even when recall is perfect.

Check that citation identifiers survive transformations. If adjacent chunks are merged, the displayed citation must still point to a source that contains the claim. Deterministically verify citation IDs against supplied context, then use human or model review to judge whether each claim is actually entailed by that source.

Run answer generation as a secondary end-to-end evaluation. Compare retrieval failure with generation failure: missing evidence, present but ignored evidence, conflicting evidence, or unsupported synthesis. This distinction determines whether to fix search, prompting, or policy logic.

Diagnose retrieval failures by stage

For every regressed query, save the rewritten query, applied filters, initial candidates with scores, reranked order, assembled context, index version, and timing. Classify the first stage where expected evidence disappears.

Useful failure classes include corpus gap, parser loss, poor chunk boundary, query-rewrite drift, embedding miss, lexical miss, overstrict filter, reranker demotion, context truncation, and stale duplicate. Human reviewers should be able to correct labels when the supposed gold passage is outdated or insufficient.

Use counterfactual runs. Disable query rewriting, increase candidate count, apply lexical-only search, bypass the reranker, or force the expected passage into context. If forced context produces the right answer, retrieval is implicated. If it still fails, investigate generation rather than tuning search blindly.

Promote the smallest reproducible failure to the appropriate suite. An ingestion defect deserves a parser fixture; a stale-filter defect needs an integration test; an ambiguous user query may need a clarification expectation.

Gate releases on critical slices and budgets

Compare baseline and candidate on the same corpus snapshot and dataset version. Count paired fixes and regressions, then slice by query type, source format, locale, access rule, freshness, and difficulty. Rerun severe or borderline cases to estimate variance from approximate search or model-based rewriting.

A release gate can require zero access-boundary violations, zero active use of superseded critical policies, stable recall for required safety facts, and a minimum improvement in the target slice. Add limits for P95 retrieval latency, context tokens, empty-result rate, and index cost. A candidate that gains one relevance point by doubling candidates and context may not be economical.

Before sign-off, experts should review new high-risk misses and harmful-source hits. The report should state corpus and index versions, configuration diff, stage metrics, slice outcomes, trace evidence, latency, token impact, and rollback plan. Ship when required evidence reliably reaches the model under real filters and budgets, not merely when the final answer occasionally looks correct.