Ragas Faithfulness and Context Recall Interview Scenarios

Ragas scores are useful only when a QA lead can explain what evidence each metric consumed and which RAG layer it can indict. Faithfulness asks whether the response is supported by retrieved context. Context recall asks whether retrieval supplied the evidence needed for the expected answer. Confusing those questions leads teams to rewrite prompts for retrieval defects or retune search for generation defects.

These 21 scenarios demand claim-level diagnosis, dataset scrutiny, metric calibration, and release judgment. A senior answer names the sample fields, inspects the documents and reference, and chooses a controlled experiment rather than treating a low aggregate as a self-explanatory bug report.

Map Metrics to the RAG Evidence Chain

Ragas publishes an evaluation API that accepts a dataset and selected metrics, and its official metric catalog lists faithfulness and context recall among RAG measures. The framework computes observations; QA still owns source truth, sample validity, and the inference from score to system change.

The fastest useful interview probe is to show the same response with two different retrieved context sets. A strong candidate immediately asks which claims remain supported and which required facts disappeared.

Interpret Faithfulness at Claim Level

1. Why can a fluent and factually plausible answer receive poor faithfulness evidence?

Faithfulness is bounded by the supplied retrieved context, not by the evaluator's general knowledge. If the answer states a correct fact absent from the contexts, it is unsupported for this RAG execution. I would decompose the response into verifiable claims, map each to context passages, and inspect the application's grounding contract. The product may allow external knowledge, but then the metric and source policy must represent that explicitly rather than calling every plausible addition faithful.

2. How would you investigate a low-faithfulness answer when retrieval looks correct?

Freeze the question, ranked contexts, prompt, and response. Mark each unsupported or contradicted claim, then replay generation with the same contexts and a stricter grounding instruction. Also test context ordering and conflicting distractors. If the retrieval contains authoritative evidence but generation ignores or embellishes it, the defect is downstream. The tradeoff is that stronger grounding can increase abstention or reduce helpful synthesis, so evaluate completeness alongside faithfulness.

3. What evidence distinguishes an unsupported claim from a citation-format defect?

Check whether the claim is entailed by any retrieved passage independently of the citation marker. A supported claim with a missing or wrong document ID is a citation attribution failure; an unsupported claim remains a grounding failure even if it points to a plausible source. I would score these separately because remediation differs. Combining them can hide fabricated content behind correct formatting or punish grounded prose for a renderer defect.

Diagnose Context Recall

4. How would you explain low context recall to a retrieval engineer?

List the reference facts needed for the answer and identify which were absent from the retrieved set. Then provide query, filters, corpus snapshot, rank positions, and relevant document IDs. Re-run retrieval with controlled changes to query transformation, top-k, filters, or reranking one at a time. A score alone is not actionable. The evidence should distinguish a corpus omission from an indexing, authorization, query, ranking, or chunk-boundary failure.

5. Why does context recall depend on trustworthy reference information?

The metric cannot determine whether retrieval contains required evidence if the expected answer is obsolete, incomplete, or unsupported by the corpus. I would trace each reference fact to an authoritative source and effective date, then adjudicate conflicts before scoring. A low result caused by a wrong reference is a dataset defect. Maintaining references adds governance cost, but skipping it turns metric output into a precise-looking comparison against unverified truth.

6. How would you evaluate recall for a multi-hop question?

Represent the required evidence as multiple facts and their source relationships, not one polished reference sentence. Verify that retrieval supplies every necessary hop and that tenant or policy filters remain correct. Report partial evidence explicitly, then test query decomposition or iterative retrieval. Increasing top-k may recover a missing hop but also add noise and latency, so pair context recall with precision and end-to-end answer checks.

Read Metrics Together

7. What does high faithfulness with low context recall suggest?

The generator may be behaving cautiously within an incomplete retrieved set. Inspect whether the response omits required facts or abstains appropriately, then focus on corpus coverage, filters, query construction, and ranking. Do not weaken grounding merely to make the answer more complete. A candidate should also challenge the reference validity because false low recall can produce the same pattern. The next experiment holds generation fixed while improving only the retrieved evidence.

8. What does low faithfulness with high context recall suggest?

Retrieval likely supplied the necessary information, but generation contradicted, ignored, or embellished it. Freeze the contexts and compare prompt variants, context order, conflict handling, and output claim extraction. Check whether noisy but technically relevant passages created ambiguity. The likely owner is generation or context assembly, yet the candidate should inspect metric reasons before concluding. High recall does not mean the context is precise, current, or internally consistent.

9. Why is a single RAG aggregate dangerous for release decisions?

Faithfulness and recall diagnose different contracts, and averages can hide severe failures in policy, locale, tenant, or multi-hop slices. I would report per-metric distributions, named risk slices, failed sample families, and uncertainty or missing results. Critical unsupported claims may be vetoes regardless of the mean. More detailed reporting can slow executive review, so provide a summary disposition while keeping the distinct evidence visible beneath it.

Challenge Dataset and Context Quality

10. How would you handle retrieved contexts that contain conflicting policies?

Annotate authority, effective date, and conflict resolution rules in the sample. The expected behavior may be to choose the current authoritative source, disclose uncertainty, or refuse. Evaluate raw support and policy resolution separately; a response can quote a retrieved but obsolete passage and still be unacceptable. The retrieval test should check version filters and ranking. Simply removing the difficult case would hide a production-relevant knowledge governance defect.

11. Why should retrieved chunk order be preserved in evaluation artifacts?

Order can change which passage the model attends to and can expose ranking regressions even when the same set of chunks is present. Store rank, document and chunk identifiers, safe content references, and retrieval scores where available. Compare faithfulness across reordered contexts in a controlled probe. Full text may require restricted storage, but losing order makes it impossible to distinguish set recall from the context actually presented to generation.

12. How would you prevent duplicate chunks from inflating confidence in retrieval quality?

Detect exact and semantic duplicates by document family, preserve one representative for scoring, and record duplication as its own retrieval defect. Repeated copies do not add independent evidence and can crowd out other required facts. I would compare metrics before and after deduplication and inspect rank diversity. Aggressive deduplication can merge legitimately distinct policy clauses, so candidate matches need source-aware review rather than one global similarity cutoff.

{
  "sample_id": "rag-case-042",
  "question": "Which approval is required?",
  "reference_facts": ["manager approval", "security review for privileged access"],
  "retrieved_context_ids": ["policy-7#c3", "policy-7#c3-copy", "policy-2#c8"],
  "diagnostic_tags": ["multi_hop", "duplicate_context", "access_policy"]
}

Customize Metrics Responsibly

13. When would you customize a Ragas metric instead of changing the product?

Customize when the product's evidence contract differs from the metric's assumptions, such as domain-specific authority, multilingual criteria, allowed external knowledge, or structured abstention. First prove the mismatch on human-labeled cases. Then version the prompt, evaluator, parser, and input mapping and recalibrate. Customization is not a way to make a candidate score higher; it is justified only when the revised measure better represents independently established product truth.

14. How would you validate a custom faithfulness rubric?

Build pairs covering supported paraphrase, unsupported addition, contradiction, synthesis across passages, obsolete evidence, and valid abstention. Have domain reviewers label claim support and severity. Compare custom metric decisions with those labels by slice, review reasons, and adversarially vary answer length and wording. Keep a holdout for the rubric revision. A higher apparent agreement on the tuning examples is insufficient because judge prompts can overfit their own demonstrations.

15. Why might a different evaluator model change Ragas results?

LLM-based metrics depend on the evaluator's interpretation, prompt following, and output parsing. A model change can shift claim decomposition or rubric decisions without any RAG application change. I would run both evaluators on a frozen calibration set, inspect disagreements, and version the metric configuration. Cost or latency gains may justify migration, but historical comparisons require a bridge study; renaming the new series as continuous would misstate the evidence.

Make Runs Reproducible and Diagnosable

16. How would you reproduce a surprising Ragas score?

Preserve the exact sample fields, ordered contexts, reference, selected metric configuration, evaluator identity, prompt or rubric version, and raw reason or intermediate output allowed by policy. Re-run evaluation without regenerating the application response. If the score changes, investigate evaluator variability or infrastructure; if stable, inspect data and criterion. Re-running the whole pipeline changes too many variables and can replace the evidence that triggered the investigation.

17. What would you do with metric execution errors in an aggregate report?

Report them as missing evaluations with counts and affected slices, not zeros and not silently dropped rows. Diagnose rate limits, malformed samples, parsing, and timeouts separately from low quality. A release rule should specify maximum missing evidence for critical slices. Treating an error as zero confuses reliability with quality; omitting it creates survivor bias. Both practices can change an aggregate without any model behavior change.

18. How would you compare two RAG candidates fairly?

Run both on the same versioned questions and corpus snapshot, preserving candidate-specific retrieved contexts and outputs. Use paired per-sample deltas and inspect regressions by risk slice. Pin the metric configuration and randomization controls where applicable. If one candidate changes corpus eligibility or latency enough to affect the product contract, report those differences rather than forcing a narrow score comparison. Fairness means shared decision conditions, not pretending architectures are identical.

Gate Releases and Score Seniority

19. How would you design a release rule around faithfulness and recall?

Use critical claim-support vetoes, calibrated slice boundaries, and paired non-regression rules rather than one universal average. Any numeric boundary shown before calibration is labeled illustrative. Require sample-level review for severe failures and record dataset, corpus, retrieval, generation, and metric versions. Recall improvements do not excuse unsupported answers, and faithfulness improvements do not excuse missing required evidence. The rule should preserve both contracts.

20. What should happen when context recall improves but latency becomes unacceptable?

Inspect whether the gain came from retrieving more chunks, slower reranking, or iterative search. Test targeted alternatives such as query decomposition, metadata filtering, chunk repair, or a smaller reranked set. Evaluate quality and latency on the same workload slices. The product chooses an operating point under explicit constraints; QA should not call the higher recall candidate superior if users cannot complete the workflow within its service objective.

21. How would you rate a candidate who recommends prompt changes for every low score?

Rate the response below senior level until the candidate maps each anomaly to retrieval, context assembly, generation, metric, or dataset evidence. Ask for one controlled isolation experiment and the artifact that would change ownership. Prompt changes may solve low faithfulness, but they cannot create a missing source document or repair a false reference. Seniority is visible in causal boundaries, not in the speed of proposing a familiar lever.

End With the Failed Sample, Not the Dashboard

Ragas is most valuable when it forces a team to inspect how evidence traveled from corpus to retrieval to response. Faithfulness protects claim support; context recall protects evidence completeness; precision, relevance, and product checks fill other parts of the diagnosis.

The decisive QA action is to preserve the failed sample, identify the broken contract, change one layer, and re-evaluate the same risk slice. A release should move only when the underlying examples show that unsupported claims and missing evidence are controlled, not because a blended score became easier to celebrate.