Debugging a RAG Faithfulness Drop After a Reranker Rollout

A faithfulness drop immediately after a reranker rollout is an attribution problem before it is a model problem. Freeze the incident window, preserve comparable traces, and determine whether support disappeared before generation, during context assembly, in the answer itself, or only inside the evaluator. Changing prompts, chunking, and thresholds together destroys the evidence needed to answer that question.

The fastest useful investigation compares the old and new paths on identical inputs. Treat query text, eligibility filters, first-stage candidates, corpus snapshot, generator settings, and grader configuration as controlled variables. The reranker and its downstream ordering policy should be the first deliberate difference.

Establish the incident boundary

Start with deployment facts, not aggregate score interpretation. Record the reranker artifact, scoring configuration, candidate depth, score threshold, tie breaker, timeout and fallback policy, context budget, corpus revision, and exact activation time. Overlay that time on faithfulness by tenant, language, query intent, source family, answer length, and fallback outcome. A global average can hide a severe loss in one policy collection or a harmless traffic-mix shift.

Verify that the alert represents newly generated observations. Delayed evaluation jobs can make a regression appear after deployment even when the underlying traces predate it. Conversely, a cache can continue serving old answers after the switch. Every evaluated row should carry request time, generation time, evaluation time, pipeline version, and cache status.

Contain exposure while evidence is collected. Route the affected slice to the previous ordering, lower traffic to the new reranker, or require abstention when adequate support is absent. Do not silently widen the context window as an emergency fix; that can hide rank loss, increase irrelevant context, and create a second uncontrolled change.

Build a root-cause tree before testing fixes

Organize hypotheses by stage. Under candidate generation, consider corpus drift, changed filters, embedding or query-rewrite differences, and low first-stage recall. Under reranking, consider score-direction inversion, text-field mismatch, aggressive thresholding, truncation before scoring, nondeterministic ties, or timeout fallback. Under context assembly, inspect deduplication, per-document caps, token budgeting, serialization, and order reversal. Under generation, inspect prompt or model configuration drift. Under evaluation, inspect grader, rubric, parsing, sampling, and dataset changes.

This tree prevents a common mistake: blaming a reranker for evidence that never entered its candidate set. A pure reranker can only reorder or filter what it receives. The LangChain retrieval documentation presents retrieval as modular building blocks; trace those boundaries separately so candidate production is not conflated with final ordering.

Attach an owner and a disconfirming observation to every branch. For example, the search owner can disprove candidate loss by showing identical frozen candidate IDs and content hashes. The serving owner can disprove timeout fallback by showing completed reranker spans and the selected-policy label. A hypothesis without a measurable disproof tends to become an argument.

Preserve claim-to-evidence trace data

For each sampled request, persist the normalized query, authorization and metadata filters, ordered candidate IDs before reranking, source scores, reranker scores, post-policy IDs, final serialized context, chunk content hashes, answer, citations, and evaluator inputs. Store sensitive text according to the product's retention controls; hashes and stable evidence IDs can support many comparisons when raw text cannot be retained.

The final serialized context is essential. A trace showing a supporting chunk at rank three is misleading if a per-source cap removed it or a token budget cut it before the prompt. Record included byte or token ranges and a reason for every excluded chunk. Also preserve duplicate groups so one document split into several similar chunks does not crowd out independent evidence.

Use a compact paired record to make omissions explicit:

type EvidencePlacement = {
  evidenceId: string;
  contentHash: string;
  candidateRank: number;
  finalRank: number | null;
  exclusionReason?: "below-cutoff" | "deduplicated" | "token-budget" | "timeout-fallback";
};

type PairedRetrievalTrace = {
  queryId: string;
  corpusRevision: string;
  baseline: EvidencePlacement[];
  candidate: EvidencePlacement[];
  answerClaimIds: string[];
};

Run isolation experiments in a strict order

First, replay identical frozen candidate lists through both rerankers. If support moves below the final context cutoff only in the candidate path, the failure belongs to scoring or selection. Next, bypass reranking and retain the original candidate order. Recovery implicates the reranker path; no recovery points toward first-stage retrieval, assembly, generation, or grading.

Then cross the contexts and generators: old context with the current generator, and new context with the previous generator configuration. If both generators fail on the new context and both succeed on the old context, evidence quality is the stronger explanation. Follow with one-variable ablations for candidate depth, score threshold, chunk field, truncation length, deduplication, and tie handling. Do not tune on the incident's held-out confirmation sample.

Finally, force timeout and malformed-score paths. Some regressions exist only when the reranker partially responds, emits non-finite values, or exceeds its deadline. Confirm that fallback preserves eligible evidence and emits an explicit trace label rather than masquerading as normal reranking.

Separate evidence loss from evaluator instability

Faithfulness is not the same as answer correctness or retrieval relevance. It asks whether answer claims are supported by the supplied context. The Ragas metric catalog lists faithfulness alongside context precision, context recall, noise sensitivity, and other distinct measures. Use that separation diagnostically: context recall loss with stable grading suggests missing evidence, while only one grader moving suggests evaluation drift.

Rescore stored answer-context pairs without regenerating them. Pin the old and current grader configuration, rerun parsing, and send a blinded stratified sample to humans using a claim-level rubric. Include unsupported, contradicted, and not-assessable outcomes instead of forcing every claim into a binary score. Check whether the same claims change labels across repeated judge calls.

Inspect the denominator too. A generator that becomes more verbose can create more atomic claims and lower a faithfulness aggregate even when retrieved evidence is unchanged. Report claim count, supported count, severity, and per-query distribution rather than relying on one mean.

Assign ownership at stage boundaries

The retrieval team owns candidate eligibility, stable evidence identity, corpus revisions, and pre-rerank recall. The reranking team owns input text construction, model or service configuration, score semantics, deterministic ordering, thresholds, and timeout behavior. The application team owns final context assembly and generator inputs. The evaluation team owns datasets, claim segmentation, rubrics, grader configuration, calibration, and alert logic. The incident commander owns the shared timeline and release disposition.

No team should overwrite another stage's raw evidence. Derived dashboards may join records by trace ID, but immutable stage outputs are the basis for attribution. A schema change at one boundary needs a compatibility check and an owner-approved migration because a missing field can look exactly like a quality regression.

Apply safety controls during diagnosis

Keep authorization filters outside model discretion and revalidate them after every ordering step. A reranker must never introduce an evidence ID that was not eligible in the first-stage set. Redact secrets and personal data before evaluator prompts, and separate production-trace access from model configuration privileges.

For high-risk domains, a low-support condition should produce an abstention, a constrained answer, or human review rather than a best-effort fluent response. Add a kill switch for the reranker, cap retries, and make fallback behavior observable. Operationally, these controls may reduce answer rate or relevance while the incident is open; that is an explicit tradeoff for limiting unsupported claims.

Gate the repair with paired evidence

Release criteria should combine retrieval, claim support, severe slices, latency, and fallback correctness. Threshold values below are illustrative, not recommended defaults:

{
  "illustrativeThresholds": true,
  "pairedGate": {
    "minimumFaithfulnessDeltaLowerBound": 0.0,
    "maximumFinalEvidenceRecallRegression": 0.01,
    "maximumSevereUnsupportedClaimIncrease": 0,
    "requiredCriticalSliceDisposition": "pass-or-reviewed-exception"
  },
  "operations": {
    "maximumRerankerTimeoutRate": 0.005,
    "fallbackMustBeTraceLabeled": true,
    "unknownEvidenceIdsAllowed": 0
  }
}

Run the gate on a sealed incident-confirmation set and a broader regression set. Require paired query-level results, confidence or uncertainty reporting appropriate to the sample design, and manual review of severe losses. Canary with automatic rollback on the same trace fields used in diagnosis. A passing average cannot waive an authorization violation or a new unsupported high-impact claim.

Close with an evidence-led action plan

Freeze the incident inputs; compare old and new ranked evidence; map answer claims to the context actually sent; execute bypass, crossed-context, and one-variable ablations; verify grader stability; then repair only the stage supported by those results. Document the causal chain from deployment to rank delta to evidence loss to answer behavior.

The investigation is complete when the team can reproduce the failure, remove it with one controlled change, and show that the repaired path passes quality, safety, fallback, and latency gates. Until then, keep the safer baseline active and the hypotheses open.