Evaluating Query Rewriting Before and After Retrieval

Query rewriting is valuable only when it retrieves better evidence for the user's actual request. A smoother-looking query is not the objective. The test target is a policy that decides whether to keep the original, use one rewrite, issue several variants, or stop because the transformation changed meaning or security scope.

Evaluate that policy with paired runs. Hold the corpus, index, retriever settings, and trusted authorization context constant; vary only the query sent to retrieval. This makes gains attributable to rewriting and exposes regressions that an end-to-end answer score can conceal.

Define the Evaluation Unit Before Running Retrieval

Treat one unit as the original request, its trusted context, candidate rewrites, retrieval outputs, and relevance judgments. The LangChain retrieval overview separates retrievers from the surrounding RAG workflow and describes query enhancement as a hybrid RAG step. That boundary is useful for testing because it lets the team inspect retrieval before generation adds another source of variation.

Record user intent as constraints rather than as a reference sentence. A support query such as "Which retention rule applies to our EU workspace after renewal?" contains jurisdiction, tenant, lifecycle event, and requested policy type. A rewrite that preserves the topic but drops "our" or "after renewal" is not equivalent.

Build Cases Around Intent-Preserving Transformations

Create cases from production-shaped queries, known retrieval misses, abbreviations, conversational follow-ups, multilingual wording, and underspecified requests. For each case, label facts that a valid rewrite may add, facts it must preserve, and assumptions it must not invent. Link every transformed query to its original family so variants do not inflate the apparent sample size.

Include a no-rewrite control. Without it, the experiment can select the least bad rewrite even when the original query performs better. Also include deliberately harmful candidates: changed dates, negated terms, broader entities, missing product names, and injected document identifiers. These mutants test whether the selector recognizes semantic damage.

A compact fixture can make those obligations reviewable:

{
  "caseId": "renewal-policy-014",
  "original": "Which retention rule applies to our EU workspace after renewal?",
  "mustPreserve": ["tenant", "EU", "after renewal", "retention rule"],
  "allowedExpansion": ["data retention", "contract renewal"],
  "forbiddenAssumptions": ["consumer account", "US region"],
  "relevantChunkIds": ["policy-eu-17", "renewal-terms-4"]
}

Keep Tenant Authorization Outside the Rewrite

Permission-aware RAG requires two independent inputs: semantic query text and a trusted authorization filter. Never ask a rewriter to reproduce tenant IDs, role scopes, ACLs, or allowed document sets in natural language. The application should derive those values from authenticated request context and pass the same filter to every retrieval branch.

Test with neighboring tenants that have confusingly similar documents. Give Tenant A and Tenant B near-duplicate policy titles, shared product vocabulary, and unique canary phrases. A rewrite succeeds only if it improves authorized retrieval without returning any chunk that fails the principal's access check. Also test stale ACL caches, removed group membership, missing tenant context, and a forged tenant name inside the user prompt.

Security assertions are vetoes. A candidate with excellent relevance and one unauthorized hit is not a net improvement. The failure is in retrieval scope even if the generator later ignores the leaked chunk.

Compare Retrieval Before Comparing Answers

Run the original and rewritten query over the same index snapshot with identical top-k, hybrid weights, reranker, filters, and timeout budget. Capture ranked chunk IDs, document versions, raw scores, filter decisions, and retrieval errors. If approximate search is used, repeat paired branches in an interleaved order so infrastructure drift does not consistently favor one side.

Measure judged recall, precision at the consumption depth, first relevant rank, forbidden-hit count, and evidence sufficiency. The Ragas metric catalog lists context precision, context recall, response relevancy, and faithfulness as separate concerns. Use that separation as a reminder that a better answer score does not identify where a rewrite helped.

Compare per-query deltas, not only aggregate means. Ten small wins must not silently offset one severe loss on a regulatory or cross-tenant slice. Preserve retrieval failures and empty results as first-class outcomes instead of converting them to zero-length context and losing the cause.

Score Intent Preservation With Layered Oracles

Use deterministic checks for explicit entities, dates, negation, locale, product, and requested output constraints. Use a narrowly calibrated semantic grader only for obligations that cannot be parsed reliably. Give the grader the original, rewrite, protected constraints, and a three-way output of preserved, changed, or uncertain; do not force uncertainty into a pass.

The selection function should reject first and rank second:

type RewriteEvidence = {
  intent: "preserved" | "changed" | "uncertain";
  forbiddenHits: number;
  relevantFound: number;
  relevantExpected: number;
};

export function eligible(candidate: RewriteEvidence): boolean {
  if (candidate.intent !== "preserved") return false;
  if (candidate.forbiddenHits !== 0) return false;
  return candidate.relevantFound > 0 && candidate.relevantExpected > 0;
}

Keep grader development examples separate from the evaluation set. Review false-preserved and false-changed decisions by slice, because the cost of accepting semantic drift differs from the cost of rejecting a harmless paraphrase.

Run Ablations That Isolate Policy Choices

Evaluate at least four branches: original only, unconditional single rewrite, original plus rewrite, and selector-controlled rewriting. If the production design uses decomposition or multi-query expansion, add those as separate branches. This reveals whether gains come from the transformation itself, extra retrieval budget, rank fusion, or the selector.

Match total retrieval budgets when comparing policies. A five-query expansion should not be credited solely for searching five times as many candidates unless that added cost is part of the intended tradeoff. Log latency and downstream context size, but avoid turning them into universal limits.

For an illustrative local gate, a team might require zero authorization violations, no confirmed intent changes in critical slices, and a positive paired recall delta whose uncertainty interval excludes its tolerated regression boundary. Those thresholds are examples to calibrate against business risk, not model facts.

Inspect Slices and Conflict Cases

Break results down by short versus conversational queries, follow-ups requiring chat context, explicit identifiers, locale, language, negation, time qualifiers, acronym density, tenant size, and document ACL shape. Query rewriting often helps one ambiguity class while harming already-specific requests, so one global enablement flag can be too coarse.

Create conflict cases where two plausible rewrites emphasize different interpretations. The correct policy may retrieve both branches and abstain from choosing, ask for clarification, or keep the original. Labeling only one rewrite as canonical punishes legitimate ambiguity handling and encourages premature certainty.

Inspect retrieval overlap as diagnostic evidence, not as a quality score. Low overlap can be beneficial when the original missed the right evidence; high overlap can still contain the wrong chunks. Human review should see the query pair, protected constraints, ranked evidence, and authorization decisions together.

Model the Failures You Expect in Production

The main failure classes are semantic drift, scope broadening, lost qualifiers, fabricated specificity, context leakage from conversation history, cross-tenant retrieval, rewrite loops, and budget exhaustion. Add infrastructure cases too: one branch times out, the index version changes mid-run, the reranker fails, or the authorization service returns an indeterminate decision.

Specify fallback behavior for every class. An unavailable rewrite service may fall back to original retrieval; a missing authorization context must fail closed; conflicting high-quality rewrites may trigger clarification. Do not let an exception path silently remove filters or switch to an unrestricted index.

Tradeoffs should remain visible. Multi-query retrieval may increase coverage and cost, while strict intent vetoes can reject useful expansions. A conservative selector may protect high-risk traffic but leave recall gains unrealized. Report each outcome rather than compressing safety, quality, and latency into one opaque score.

Operational Checklist

• Freeze the corpus, index, retriever configuration, and authorization policy version.
• Store original and rewritten queries under one family identifier.
• Label protected constraints, permitted expansions, and forbidden assumptions.
• Include original-only and no-rewrite controls.
• Apply trusted tenant and ACL filters identically to every branch.
• Capture ranked chunk IDs, document versions, filter verdicts, and errors.
• Score retrieval and intent before running answer generation.
• Treat unauthorized hits and confirmed intent changes as vetoes.
• Compare paired deltas by risk and query slice.
• Test timeout, stale policy, missing identity, and partial-branch failures.

Action Plan: Ship a Rewrite Policy With Evidence

Start with one failure-rich query family, such as acronym expansion or conversational follow-ups. Freeze a labeled corpus and build original-only, unconditional-rewrite, and selector-controlled branches. Add tenant twins and forged-scope prompts before tuning the selector, then review every intent change and forbidden hit.

Choose rollout rules from paired slice evidence. Enable rewriting first where it improves relevant retrieval without weakening constraints, preserve the original path as a measured fallback, and shadow uncertain slices until adjudication is stable. Promote the policy only when its trace explains which rewrite was used, which trusted filters were applied, and why the selected evidence was allowed.