Building Multi-Hop RAG Evals with Evidence-Chain Labels

Multi-hop RAG evaluation fails when a dataset stores only a question, a final answer, and a bag of documents. That format cannot prove that the system retrieved each dependency, followed a valid evidence path, or merely found a passage that leaked the conclusion. A useful eval represents the evidence chain itself.

The goal is not to reward a hidden chain-of-thought. It is to label externally verifiable facts and their dependencies. With that graph, teams can measure candidate recall at each hop, complete evidence coverage, answer attribution, shortcut leakage, and the latency cost of iterative retrieval without requiring private reasoning text.

Define a verifiable evidence graph

Represent evidence nodes as source-backed facts, not whole documents. Each node should contain a canonical fact ID, exact supporting spans, source and version, applicable time, entities, and relevance grade. Directed edges describe dependency: fact B can be sought or interpreted only after fact A supplies a bridge entity, date, category, or other needed value.

Mark the minimal sufficient fact sets for the answer. A two-hop query may have three relevant documents but require only two facts. Conversely, one document may contain several facts; if it alone exposes the final answer, the case may not test multi-document retrieval even though its annotation lists multiple nodes.

Keep answer aliases and normalization separate from evidence. Exact string matching is appropriate only when the answer has a closed form and normalization rules are deterministic. For explanatory answers, use claim-level reference facts and citations so a fluent paraphrase can pass without accepting unsupported additions.

Construct questions that require dependency

Begin from a real information need or a reviewed graph path. Write the question naturally, then perform a shortcut audit. Search the full corpus using the question, answer aliases, bridge terms, and distinctive phrases. If one passage states the final answer directly, either relabel the case as single-hop, exclude that source under a documented as-of snapshot, or redesign the query.

Do not make difficulty depend on obscurity alone. A good multi-hop case requires composition, such as identifying which policy applies and then finding that policy's deadline. An awkward query that hides a simple entity lookup tests query interpretation, not evidence chaining.

The DeepEval golden synthesizer documentation is an official reference for generating evaluation goldens. Synthetic generation can help propose paths and distractors, but every case still needs provenance, shortcut review, answerability checks, and a split policy. Generated fluency is not evidence that the chain is valid.

Store required and alternative chains

One gold path is often incomplete. Two documents may independently establish the same bridge fact, or a direct primary source may replace two secondary sources. Store alternative sufficient sets or define required fact IDs whose evidence can come from any approved source. Evaluation should reward valid support, not fidelity to one curator's search route.

{
  "caseId": "mh-072",
  "query": "Which retention rule applies to the plan acquired with Project Orion?",
  "requiredFacts": ["orion-plan", "plan-retention-rule"],
  "validChains": [
    ["contract-14:orion-plan", "policy-8:plan-retention"],
    ["catalog-3:orion-plan", "policy-8:plan-retention"]
  ],
  "dependency": {
    "from": "orion-plan",
    "to": "plan-retention-rule",
    "bridgeField": "planId"
  },
  "corpusSnapshot": "enterprise-docs-2026-06-30"
}

This illustrative record exposes answer-bearing labels only to evaluation tooling, never to the system under test. Add exact source spans, effective dates, ACL class, and adjudication history in a production dataset.

Label distractors by the error they invite: same entity with wrong date, same policy family with wrong plan, obsolete source, bridge-name collision, and partially valid chain. Those slices explain failures better than a generic non-relevant label.

Prevent split and generation leakage

Split source families, entities, path templates, conversations, and generated seed graphs before creating paraphrases. If the same bridge entity or nearly identical policy appears in train and test, a system can memorize the mapping rather than retrieve it. Keep all variants derived from one graph path in one partition.

Generate or curate development cases for prompt, planner, retrieval, and threshold changes. Seal test graphs and source snapshots. Assessors may see gold chains; pipeline developers tuning the candidate should not. Once a test chain drives a fix, migrate it to the next development set and replace it with an independent held-out path.

Check answer leakage in document titles, chunk metadata, URLs, and synthetic query wording. A source filename containing the answer can let metadata ranking bypass the intended hop. Keep a record of discovered shortcuts because corpus updates can introduce new ones later.

Instrument every observable hop

Assign a step ID to each retrieval request. Store the query or structured search state, filters, returned evidence IDs and ranks, selected evidence, intermediate factual state, tool errors, timeout outcome, and cumulative latency. Redact sensitive user content while preserving stable test identifiers and enough trace to reproduce the decision.

Do not require internal chain-of-thought. The observable contract can be a sequence of retrieval calls, selected source IDs, cited answer claims, and structured intermediate variables such as planId. Systems that solve the case in one retrieval call can still pass if their evidence is sufficient and no shortcut invalidates the case.

Capture branch behavior for planners that issue parallel searches. Hop number may be a graph layer rather than a strict sequence. The evaluator can match retrieved facts to dependencies without assuming one exact call order.

Score hops and complete chains

Candidate recall at hop one asks whether the initial candidates contain an approved starting fact. Conditional recall at hop two measures retrieval when the correct bridge fact is supplied. Free-running hop-two recall uses the system's actual prior state. The gap between conditional and free-running results localizes upstream propagation errors.

Complete-chain coverage requires at least one sufficient evidence set in the selected context. Path precision measures how much selected evidence participates in a valid chain, but irrelevant extras should be interpreted in light of context budget and security risk. Final answer correctness and claim-level citation entailment remain separate outcomes.

The Ragas customization documentation is an official starting point for adapting evaluation behavior. Multi-hop scoring still needs a local chain schema because the product must define acceptable evidence sets, dependency semantics, and severity. Store metric configuration with every run.

Report the earliest observable failure: missing corpus fact, hop-one retrieval miss, wrong bridge extraction, hop-two retrieval miss, selection loss, unsupported composition, wrong answer, or citation failure. Aggregate metrics are useful for release; earliest-failure counts are useful for repair.

Use oracle ablations for attribution

An oracle-hop ablation supplies the correct prior fact while leaving the next retrieval stage unchanged. If hop two succeeds with the oracle bridge but fails free-running, the defect is upstream. An oracle-evidence ablation provides all gold evidence to the generator; if the answer remains wrong, retrieval is not the only problem.

Other useful ablations include single-shot versus iterative retrieval, planner enabled versus fixed query templates, reranker on versus off, and full context versus minimum sufficient evidence. Change one stage at a time and replay stable inputs where possible.

Oracle tests are diagnostics, not release candidates. Their labels must be excluded from normal system input and their scores clearly separated. A perfect oracle-answer score does not imply the deployed retriever can assemble the chain.

Calibrate judgments and uncertainty

Human reviewers should independently validate answerability, sufficient fact sets, alternative paths, and final claims. Adjudicate disagreements about whether a passage supports a bridge or whether a shortcut exists. Preserve raw judgments and the reason for each chain revision.

Model graders can assist with answer semantics and claim attribution after calibration on held-out adjudicated examples. Use deterministic graph matching for evidence IDs and required fact coverage. Do not ask a semantic judge to infer whether an unlogged document was retrieved.

Use paired comparisons at the case level and cluster related graph paths during resampling. Publish intervals for complete-chain and answer differences. Sparse path templates or rare domains may require qualitative review instead of an unstable automated gate. Mark uncertain labels and missing judgments explicitly.

Test latency and stopping behavior

Each retrieval hop adds potential network, ranking, and generation work. Measure per-hop and cumulative latency, candidate counts, retries, parallel branches, and timeout cause. Compare quality at the same end-to-end budget, not with unlimited time for the candidate and a production timeout for the baseline.

Define stopping rules. The system should halt when evidence is sufficient, the search budget is exhausted, no new evidence appears, or a safety condition blocks continuation. Test loops caused by repeated bridge terms, duplicate evidence, and alternating queries. A correct answer produced after an unbounded search is not an acceptable operational result.

Fallback behavior needs scoring. If the final hop times out, should the answer abstain, present partial evidence with uncertainty, or use a single-hop result? The release gate should evaluate that actual policy and forbid unsupported completion from incomplete chains.

Execute a decisive multi-hop plan

1. Build fact-level evidence graphs with source spans, versions, dependency edges, and sufficient evidence sets.
2. Create natural queries, audit the corpus for single-document shortcuts, and label meaningful distractors.
3. Split source families, entities, templates, and generated seeds before tuning or query expansion.
4. Instrument retrieval calls, selected evidence, structured bridge state, citations, errors, and cumulative latency.
5. Score candidate recall by hop, complete-chain coverage, answer correctness, and claim attribution with paired uncertainty.
6. Run oracle-hop and oracle-evidence ablations to identify the earliest broken stage without exposing gold labels to the candidate.
7. Release only when critical chains, timeout behavior, attribution, and latency meet the frozen policy; otherwise repair the localized stage and version the dataset.

A multi-hop eval is credible when another reviewer can reconstruct which facts were necessary, which evidence paths were acceptable, and where the pipeline lost them. Evidence-chain labels turn an opaque wrong answer into a testable sequence of retrieval and attribution contracts.