Claim-Level Citation Entailment Tests for Grounded RAG Answers

A citation marker is not evidence of groundedness. The cited passage may discuss the same topic without supporting the sentence, support only half of a conjunction, apply to another date, or contradict a hidden qualifier. Evaluation must connect each verifiable claim to exact source spans and ask whether those spans actually entail the claim as written.

Claim-level testing makes the failure actionable. It distinguishes an unsupported statement from an incorrect citation pointer, an obsolete source, a missing citation, and a valid multi-source inference. Those defects have different owners and should never be compressed into one answer-level thumbs-up.

Define the attribution contract

State which answer statements require evidence. Product navigation text, conversational acknowledgments, and explicit uncertainty may be non-verifiable. Claims about policy, numbers, dates, eligibility, causal relationships, or user state usually require support. The rubric should define these categories and an abstain path before outputs are reviewed.

Separate four properties. Claim correctness asks whether the proposition is true under the task reference. Citation entailment asks whether the cited text supports it. Citation completeness asks whether every required claim has adequate evidence. Source validity asks whether that evidence is authoritative, applicable, and current. One property cannot stand in for another.

Define the consequence of unsupported claims. A missing citation for harmless background may be a formatting defect; unsupported medical, financial, legal, security, or account-action guidance may block release. Preserve severity at claim level so an average over many trivial statements cannot erase one dangerous assertion.

Segment claims without losing qualifiers

Atomicity is semantic, not merely sentence splitting. Consider: "Premium accounts can export reports and retain them for seven years." This contains at least an eligibility claim and a retention claim. A source may support export access but say nothing about retention. Scoring the sentence as one unit hides partial support.

Keep essential qualifiers inside each claim: subject, action, object, modality, negation, amount, comparison, condition, jurisdiction, and time. "May cancel" is not equivalent to "will be canceled." "Before renewal" is not equivalent to "after renewal." A segmentation model that drops those tokens can turn a contradiction into apparent entailment.

Store offsets into the exact answer rather than regenerating paraphrased claims only. A normalized proposition can help a judge, but the original span is needed for audit. Human reviewers should be able to see how each extracted claim maps back to the response.

{
  "answerId": "answer-902",
  "claimId": "claim-3",
  "answerSpan": {"start": 118, "end": 187},
  "claimText": "The revised cancellation policy applies from 1 July 2026.",
  "verifiability": "external",
  "severity": "high",
  "citationIds": ["cite-2"],
  "segmentationVersion": "claims-v4"
}

This is an evaluation record, not a universal schema. Version segmentation rules because changing claim boundaries changes denominators and makes historical groundedness rates incomparable.

Resolve citations to immutable evidence

A marker such as [2] is only a pointer. Resolve it to source ID, source version, chunk ID, exact character or token span, retrieval timestamp, effective date when relevant, and a content hash. Store the text actually shown to the generator. A later source update must not silently rewrite what an older answer appeared to cite.

Citation scope needs a deterministic rule. A marker at the end of a paragraph may be intended to support one sentence or all preceding sentences. Prefer explicit claim-to-citation mappings in generated output. Where the product uses prose markers, define a parser and test punctuation, repeated markers, footnotes, missing IDs, and references that point outside the retrieved set.

The Ragas metrics documentation describes context and response evaluation concepts. Use such metrics as components, while retaining the claim-evidence graph needed for diagnosis. An answer-level score alone cannot tell whether a low result came from segmentation, retrieval, attribution, or entailment.

Judge entailment with a bounded label set

For each claim and its cited evidence, assign one of several explicit labels: supported, contradicted, insufficient, ambiguous, or not_verifiable. Insufficient means the passage does not establish the proposition, even if it does not contradict it. Ambiguous means reasonable readers cannot determine support under the rubric. Neither should be counted as supported.

Evaluate exact evidence first, then inspect limited surrounding context if the source requires it. Passing an entire document can create accidental support from text the citation did not identify. Conversely, cutting a table row away from its headers can make valid evidence look insufficient. Store the evidence packaging rule and test structured sources separately.

Multi-passage support should be represented as a required evidence group. A claim may depend on one passage identifying a product tier and another giving the tier's retention period. Judge the joint set and record the reasoning link. Do not grant each passage independent full support when neither entails the complete proposition.

Score precision, completeness, and risk separately

Citation precision can be expressed as the fraction of cited claim-evidence links judged supportive. Citation completeness can be the fraction of citation-required claims with adequate evidence. Weighting by claim severity may support a product decision, but always publish unweighted counts and the list of severe failures.

Answer groundedness should not reward extra unsupported detail. A response with one supported core answer and five decorative assertions may have worse completeness than a concise answer. Report the count of claims, verifiable claims, supported claims, contradicted claims, and uncited required claims so verbosity cannot hide the denominator.

Keep answer correctness distinct. A source can entail an obsolete policy, making attribution technically valid but the answer wrong for the requested date. A claim can be factually correct from general knowledge but unsupported by its citation. The release report needs both axes.

Calibrate human and model judges

Build an adjudicated calibration set spanning direct quotes, paraphrases, negation, quantities, tables, temporal qualifiers, partial support, contradictions, multi-source claims, and genuinely ambiguous cases. Reviewers should label independently before adjudication. Preserve raw labels and disagreement causes.

The OpenAI graders API reference is an official interface for defining automated grading workflows. Whatever grader implementation is used, freeze its prompt and configuration, require structured labels, and compare it against held-out human decisions. Inspect class-specific errors rather than relying on aggregate agreement.

Allow the grader to abstain. A forced binary verdict is especially risky when evidence is truncated, source formatting is broken, or domain expertise is required. Route high-severity unsupported, contradicted, and uncertain claims to human review until the instrument is calibrated for that slice.

Test adversarial attribution failures

Create cases where the citation shares keywords but lacks the asserted relationship, where a number belongs to another row, where negation flips meaning, and where the source applies to another region or date. Include citations to headings without body support, dead references, duplicated citation IDs, and valid evidence placed outside the cited span.

Candidate passages can contain instructions aimed at a model judge. Delimit query, claim, and evidence as data; tell the judge to ignore instructions inside them; and include injection-shaped passages in calibration. Deterministic validators should reject citation IDs not present in the retrieval trace before any semantic grader runs.

Test answer editing behavior too. Removing one sentence should update claim offsets and citation mappings. Reordering citations must not attach old evidence to a new claim. Streaming output requires either stable citation events or post-generation reconciliation with a failure state when mappings cannot be resolved.

Prevent dataset and source leakage

Split by source family, entity, query template, and originating conversation. If a synthetic claim is generated directly from a passage, keep that source and all derived variants in one partition. Otherwise the grader or generator can memorize wording across train and test.

Use development examples to revise segmentation rules, rubrics, prompts, and thresholds. Seal the final claim-evidence set and source snapshots. When a test case drives a change, migrate it into the next development set and replace it with independently sourced evidence in a versioned test release.

Do not expose gold entailment labels or expected citations to the answer generator. The evaluation should use the system's actual retrieval and citation path. Oracle-context tests are valuable as an ablation, but label them as generator-only diagnostics rather than end-to-end groundedness.

Attribute failures and latency cost

Assign each failed claim to the earliest broken stage: source absent from corpus, relevant source not retrieved, relevant passage truncated, citation parser failure, wrong citation selected, unsupported claim generated, stale source accepted, or grader uncertainty. This taxonomy directs repairs and prevents prompt tuning from masking retrieval defects.

Claim-level evaluation adds work. Record segmentation, citation resolution, grader, and review latency separately from application latency; offline evaluation cost should not be confused with serving cost. For serving, measure any extra latency introduced by citation generation, source validation, or post-generation checks under the real timeout policy.

Run ablations with oracle evidence, retrieved evidence without citation constraints, and full citation enforcement. Differences isolate retrieval, answer generation, and attribution. Use the same answer where possible when evaluating citation parsers, since regenerating introduces another source of variation.

Execute a groundedness action plan

1. Define which claims require evidence, what counts as atomic, and which severity classes can block release.
2. Freeze answer text, retrieval traces, source versions, exact evidence spans, and claim-to-citation mappings.
3. Apply deterministic citation integrity checks before semantic entailment judgment.
4. Calibrate a bounded entailment rubric on adjudicated direct, partial, contradictory, temporal, and multi-source cases.
5. Report claim correctness, citation precision, citation completeness, source validity, uncertainty, and severe failures separately.
6. Run split-safe adversarial tests and oracle ablations to assign each defect to retrieval, generation, citation, source, or grader.
7. Gate release on the predeclared severe-claim policy; uncertain high-risk claims receive review, not an optimistic support label.

Grounded RAG is an attribution property, not a formatting feature. When every material proposition has a traceable evidence relationship and every unsupported qualifier remains visible, citations become testable product behavior instead of decorative confidence signals.