Testing Reciprocal Rank Fusion in Hybrid RAG Search

Reciprocal rank fusion can improve hybrid search only when the lexical and vector rankings contribute complementary evidence. A higher aggregate score does not prove that complementarity. It may come from deeper candidate retrieval, duplicate chunks, favorable judgment coverage, or parameters tuned against the final test labels.

A strong RRF evaluation therefore treats fusion as a ranking experiment. It freezes branch behavior, uses common relevance judgments, verifies document identity, compares branch and union baselines, and carries quality, latency, and partial-failure behavior into one release decision.

Write the fusion contract

Specify the ranked-list semantics before calculating a score. For each branch, record candidate depth, whether ranks start at one, how ties are ordered, which metadata filters apply, and what happens when fewer candidates are returned. The fusion layer needs a rank constant, optional branch weights, output depth, and a policy for documents missing from one list.

For document d, a weighted formulation can be expressed as:

rrf_score(d) = sum(weight_j / (rank_constant + rank_j(d)))

The sum includes only branches that returned the document. This formula does not make parameter choices universal. A smaller rank constant emphasizes top-rank differences more strongly; weights change branch influence; truncated branch lists remove contributions beyond their depth. Every value should be tuned on development data and printed in the run artifact.

Define whether fusion occurs at chunk, section, or canonical document level. The downstream generator may consume chunks, but fusing duplicate chunks as independent documents can over-reward a repeated source. A practical design computes branch contributions on a canonical evidence ID, keeps the best representative chunk or a deterministic set, and logs all collapsed members.

Establish fair branch baselines

Run lexical-only and vector-only systems with their actual production filters and comparable candidate budgets. Also include the current production retriever and a deterministic union baseline. If RRF retrieves twice as many candidates as either branch baseline, any recall gain could be a depth effect. Add deeper single-branch baselines or equalize total candidates to separate fusion value from additional work.

The LangChain retrieval documentation presents retrieval as a modular pipeline with search components that can be composed. Use that modularity in the experiment log: branch query, index version, search parameters, filters, and returned identities should be inspectable before fusion. A fused list alone cannot reveal which branch produced an error.

An oracle-union diagnostic is useful: ask whether the union contains relevant evidence even when RRF orders it poorly. It is not a deployable baseline because it uses relevance labels to choose ordering, but it separates candidate-generation loss from fusion-order loss. Label it diagnostic, not a candidate system.

Implement rank and identity handling deterministically

The fusion implementation should reject duplicate IDs inside one branch, non-finite weights, and invalid ranks. Ties in fused score need a stable secondary key, such as best branch rank followed by canonical ID. Otherwise repeated runs can reorder equal-scoring evidence and create noisy metrics.

type RankedItem = { evidenceId: string; rank: number };

export function fuseRrf(
  branches: Array<{ weight: number; items: RankedItem[] }>,
  rankConstant: number,
) {
  const scores = new Map<string, { score: number; bestRank: number }>();
  for (const branch of branches) {
    for (const item of branch.items) {
      const current = scores.get(item.evidenceId) ?? {
        score: 0,
        bestRank: Number.POSITIVE_INFINITY,
      };
      current.score += branch.weight / (rankConstant + item.rank);
      current.bestRank = Math.min(current.bestRank, item.rank);
      scores.set(item.evidenceId, current);
    }
  }
  return [...scores.entries()].sort(
    (a, b) => b[1].score - a[1].score ||
      a[1].bestRank - b[1].bestRank ||
      a[0].localeCompare(b[0]),
  );
}

This example assumes validation happened at the boundary. Unit tests should cover one empty branch, an item appearing in both lists, duplicate input IDs, unequal depths, equal fused scores, extreme but permitted constants, and deterministic ordering. Property tests can assert that adding an additional positive branch contribution never lowers that item's raw RRF score.

Protect corpus and query splits

Group queries by conversation, user intent template, entity, and source family before splitting. Keep generated paraphrases with their seed query. If near-identical policy chunks cross corpus partitions, branch weights may be tuned against evidence patterns that repeat in test.

Use training data for model or index construction, development data for branch depth, rank constant, weights, deduplication, and query preprocessing, then evaluate one frozen selection on test. Trying many RRF variants on test and publishing the best is multiple-comparison leakage even if no model weights changed.

Version relevance judgments with the corpus snapshot. A lexical branch may surface exact identifiers that were well represented in the pool, while a vector branch finds unjudged semantic alternatives. Expand a common judgment pool from both branches, fused results, and independent baselines before comparing them.

Build shared relevance judgments

Pool candidates deeply enough on development data to choose an assessment budget, then freeze the test pooling protocol. Assessors should be blind to branch and fused rank. Preserve graded relevance, unknowns, abstentions, and canonical duplicate relationships.

Report judged coverage at every evaluated depth for every system. Do not convert unjudged vector-only results to negatives while lexical results are mostly judged. The companion guide on incomplete relevance labels explains why that asymmetry can reverse a comparison; in an RRF study it can also make a branch seem harmful when it is merely novel.

Judgment pools should include cases where exact terms matter, paraphrases dominate, both branches agree, branches disagree, filters eliminate candidates, and no relevant source exists. The final class is important: a fusion algorithm cannot manufacture answerability, and a forced relevant label encourages topical noise.

Choose metrics for the downstream context budget

If the generator receives five contexts, evaluate NDCG at five or another order-sensitive measure at that boundary, plus recall at the initial candidate depth. Reciprocal rank is useful when the first relevant item dominates. When an answer needs several independent facts, measure coverage of required evidence identities rather than only whether one positive appeared.

The Ragas metrics documentation provides retrieval and response evaluation concepts that can be kept distinct. Score fusion first on ranked evidence. Then, in a separate end-to-end experiment, measure whether the generator uses that evidence correctly. A better answer from one stochastic generation is not enough to attribute a ranking improvement to RRF.

Report per-query deltas and win, loss, and tie counts beside averages. Fusion can make identifier-heavy queries worse while improving broad semantic queries. Slice by query length, exact identifier presence, locale, temporal constraint, entity ambiguity, branch overlap, and number of required evidence items.

Run ablations that explain the gain

At minimum compare lexical only, vector only, simple deduplicated union, unweighted RRF, the proposed weighted RRF, and each branch with the other's candidate depth budget. Keep the same qrels and output depth. Add branch dropout to show behavior when a service times out or returns empty.

Ablate canonical deduplication separately. If quality falls after deduplication, inspect whether repeated chunks were legitimately covering distinct evidence or simply occupying discounted ranks. Also test query rewriting independently from fusion; combining both changes in one candidate prevents attribution.

Create a branch-complementarity table on development and sealed test data. Count queries where relevant evidence appears only in lexical, only in vector, in both, or in neither. This does not replace ranking metrics, but it explains whether fusion has material to combine. If both branches fail on the same slice, tuning the RRF constant is unlikely to solve the underlying retrieval gap.

Measure latency and degraded modes

Record latency for each branch, fusion, deduplication, and any metadata fetch. Branches may execute concurrently, but end-to-end behavior depends on orchestration, timeouts, connection pools, and fallback rules. Measure cold and warm conditions under the same protocol used for baselines, and report quality among timed-out as well as successful requests.

Define degraded behavior explicitly. If vector search fails, does the lexical list pass through unchanged? If one branch returns stale results, can the fusion layer detect the snapshot mismatch? If ACL filtering empties one branch, does the system broaden retrieval unsafely? Tests should assert both ranking and authorization invariants.

Branch depth is a latency-quality parameter. Tune it with a constrained objective or a gate that requires both relevance and latency conditions. Values used in a local policy are illustrative until measured against the application's workload; do not present them as general RRF defaults.

Quantify uncertainty and decision sensitivity

Because every configuration runs on the same queries, calculate paired differences. Bootstrap query groups or use another paired method that respects conversation and source clustering. Publish intervals by critical slice where evidence permits. A small aggregate gain with an interval crossing zero is not a reliable reason to add another service dependency.

Test parameter sensitivity on development data. Plot or tabulate performance over a predeclared grid of rank constants, weights, and branch depths. Prefer a stable region over a single narrow optimum. Freeze one configuration before opening test results, and preserve the full development sweep for review.

For release, combine a primary ranking boundary, critical-slice vetoes, no authorization regression, and latency limits. If fusion improves NDCG but increases empty results during branch timeout, the system has not passed its operational contract.

Execute a decisive RRF test plan

1. Freeze query groups, corpus snapshot, relevance grades, canonical evidence identity, and context depth.
2. Record branch contracts, candidate budgets, filters, rank semantics, failure handling, and latency instrumentation.
3. Build blind pooled judgments from every branch, independent baselines, and fused candidates; retain unknown status.
4. Tune rank constant, weights, depth, and deduplication only on development queries, then lock the test configuration.
5. Compare branch-only, union, RRF, weighted, and dropout ablations with paired ranking metrics and slices.
6. Exercise timeout, empty, stale, and filter-conflict paths while measuring end-to-end quality and latency.
7. Ship only when the frozen fusion clears relevance, uncertainty, security, and latency gates; otherwise fix the branch or fusion failure identified by the ablation.

RRF is easy to calculate and easy to evaluate badly. A valid test proves not just that a fused number increased, but that complementary branches improved the ranked evidence under a common corpus, common judgments, stable identity rules, and the operating constraints users will actually encounter.