Testing RAG Abstention on Unanswerable and Conflicting Evidence

A RAG system should not answer merely because retrieval returned text. It should answer when authorized evidence is sufficient for the requested claim, resolve disagreements only when an explicit rule permits resolution, and abstain when the evidence cannot support a responsible conclusion.

Testing that behavior requires more than a set of empty-context prompts. Build fixtures that distinguish absent evidence, irrelevant evidence, partial support, genuine contradiction, superseded documents, and retrieval failures. Then score the decision, the explanation, and any supported partial answer separately.

Define Answerability as an Evidence Contract

For each test case, specify the claims requested, the minimum evidence needed, permitted source hierarchy, time scope, authorization scope, and acceptable response action. The answerability label is about the evidence available to this run, not whether a knowledgeable person could answer from memory.

The Ragas metric catalog separates context recall, context precision, response relevancy, and faithfulness. Those dimensions help diagnose RAG behavior, but abstention needs an additional decision label: answer, partial answer, clarify, or abstain. Keep that label visible instead of reducing it to generic response quality.

Build a Taxonomy of Unanswerable Cases

Use distinct fixture classes because they imply different fixes. corpus_absent means the required fact is not indexed. retrieval_miss means relevant authorized evidence exists but was not returned. access_denied means evidence exists but the current principal may not see it. underspecified means multiple interpretations remain. future_unknown asks for information that does not yet exist. unsupported_inference asks for a conclusion the sources do not justify.

Add partial cases where one subquestion is supported and another is not. Include plausible distractors that share keywords but cannot answer the claim. Include adversarial wording that pressures the model to guess, ignore missing evidence, or treat a draft as approved policy.

Do not collapse these classes into one refusal reference. A secure refusal for access_denied should not reveal the restricted document's existence or title. A retrieval_miss should create an operational alert. An underspecified request should ask a concise clarifying question.

Represent Conflicting Evidence Explicitly

Conflict evidence needs structure. Label the exact claims that disagree, each source's authority, effective interval, jurisdiction, audience, document state, and supersession relationship. Two different values are not necessarily unresolved: one may be newer, one may apply to contractors, or one may be a draft.

Use three expected outcomes. resolvable allows an answer using a documented precedence rule and requires the response to cite the winning evidence. unresolved requires abstention or escalation. apparent_only means the documents concern different scopes and the response should explain the distinction.

{
  "caseId": "expense-limit-conflict-07",
  "claim": "meal reimbursement limit for EU contractors",
  "conflictClass": "unresolved",
  "evidence": [
    {"chunkId": "policy-41", "value": "60 EUR", "state": "approved", "effective": "2026-01-01"},
    {"chunkId": "memo-19", "value": "75 EUR", "state": "approved", "effective": "2026-01-01"}
  ],
  "expectedAction": "abstain_and_escalate"
}

Keep intentionally conflicting fixtures separate from accidental duplicate content. If ingestion has lost effective-date or status metadata, the test should report that pipeline defect rather than pretending the model alone can resolve it.

Capture Retrieval and Authorization Causes

Store ranked chunk IDs, document versions, retrieval scores, authorization filter input, filter decisions, query rewrite, reranker output, and errors. An empty context is an observation, not a root cause. The same answer behavior can be correct for missing knowledge and dangerously misleading when a retriever outage was hidden.

Create tenant-paired cases. Put a complete answer in Tenant B and no answer in Tenant A, then query as Tenant A using vocabulary from both. The correct behavior is to abstain without exposing Tenant B's fact. Repeat with revoked access, stale group membership, and a document that becomes unauthorized between retrieval and generation.

Fail closed when authorization cannot be evaluated. However, label the user experience separately: a generic unsupported answer, an access-safe explanation, and an operational error may all avoid leakage but have different usefulness.

Score the Decision and the Response Separately

First score the action against the fixture label: answer, partial, clarify, or abstain. Then inspect unsupported claims, conflict disclosure, cited evidence, and next-step quality. This prevents a polished refusal from passing when the case was fully answerable, and prevents a correct answer from passing when it includes an extra unsupported claim.

Use deterministic checks for cited chunk IDs, forbidden entities, required escalation codes, and claim coverage. For semantic judgments, define a narrow rubric and permit uncertain. The OpenAI graders API reference documents programmable grader resources; any model-based grader used here still needs calibration against independently adjudicated abstention cases.

A simple outcome record keeps errors decomposable:

from dataclasses import dataclass

@dataclass(frozen=True)
class AbstentionResult:
    expected_action: str
    observed_action: str
    unsupported_claims: int
    forbidden_disclosures: int
    cited_supporting_chunks: int

def critical_failure(result: AbstentionResult) -> bool:
    return result.forbidden_disclosures > 0 or result.unsupported_claims > 0

Calibrate Thresholds With Decision Curves

If the system computes evidence sufficiency or conflict scores, preserve the raw values and evaluate candidate decision rules after execution. Plot false answers against unnecessary abstentions by slice. A threshold that is appropriate for product tips may be unacceptable for employment policy or account permissions.

Label thresholds as illustrative until local data and risk owners approve them. For example, a team might require all critical claims to have at least one authorized supporting source, zero unresolved contradictions, and no retrieval error before answering. That is a transparent policy example, not a universal measure of truth.

Do not tune and report on the same cases. Split by source family and incident lineage so near-duplicate documents do not cross calibration and audit sets. Recheck the threshold after corpus, retriever, metadata, prompt, or conflict-resolution changes.

Test Failure Paths and Partial Answers

Inject retriever timeouts, malformed metadata, unavailable rerankers, stale indexes, duplicate chunks, and context truncation. Verify that the generator receives an explicit evidence-state signal rather than a context that merely looks short. A timeout must not be interpreted as proof that no policy exists.

For multi-part requests, map each requested claim to evidence. Permit the system to answer supported claims while marking unresolved claims, provided the response does not imply completeness. Test whether one well-supported answer causes the model to improvise the remaining fields.

Also test recovery. When a user supplies a missing date or selects one of two interpretations, the next turn should reconsider answerability with the updated constraints. A prior abstention should not become a sticky conversation state.

Establish Risk-Based Release Gates

Report false-answer rate, unnecessary-abstention rate, clarification accuracy, partial-answer precision, conflict-resolution accuracy, and forbidden disclosures by fixture class. Show counts alongside rates, especially for rare high-risk slices. Review every confirmed leakage or unsupported high-consequence claim independently of the aggregate.

Candidate and baseline should run on the same retrieval snapshot and fixture set. If evidence changes between runs, compare on a shared core or declare the result non-comparable. Separate model behavior from retrieval coverage so the owner of each regression can act.

The central tradeoff is usefulness versus unsupported certainty. Over-abstention frustrates users and can hide retrieval defects; under-abstention invents confidence. A four-way action policy gives more room than a binary answer/refuse rule, but it also needs richer labels and UI handling.

Operational Checklist

• Define requested claims and minimum authorized evidence per case.
• Separate corpus absence, retrieval miss, denied access, ambiguity, and unsupported inference.
• Label conflicts with authority, status, scope, dates, and supersession.
• Include answerable controls and partial-answer fixtures.
• Capture retrieval, filtering, reranking, and failure traces.
• Test tenant twins without revealing restricted document existence.
• Score action correctness before response style.
• Use deterministic checks for disclosures and unsupported structured claims.
• Calibrate semantic graders and decision thresholds on separate data.
• Gate critical unsupported answers and forbidden disclosures independently.

Action Plan: Build an Abstention Gate That Explains Failures

Begin with a small adjudicated set spanning fully answerable, corpus-absent, retrieval-miss, unresolved conflict, and partial-answer cases. Add source authority and time metadata, instrument the complete retrieval path, and implement a four-way action label. Run the current system before changing prompts so the baseline reveals where errors originate.

Calibrate one risk-specific decision rule, fault-inject the retrieval path, and review all false answers plus a sample of unnecessary abstentions. Release only when critical slices have explicit owners and the runtime trace can show why the system answered, clarified, partially answered, or withheld a conclusion.