The citation is present, but does the source support the claim?

Your RAG answer names the right policy and shows [policy-3], but source policy-3 contains a different rule. A broad groundedness score can still look healthy because most of the answer came from the retrieved set. The user sees a citation and trusts a claim that the cited passage does not support.

That failure is easy to miss when a team treats every evidence problem as one metric. Citation syntax, source identity, claim support, and citation coverage are different contracts. They break for different reasons, produce different evidence, and need different fixes.

Why one citation score hides the defect you need to find

A citation marker is only a pointer. Before the pointer has any quality meaning, your application must define what it points to. In one product, [3] means the third item in the current retrieval result. In another, [policy-3] is a durable document identifier. Some systems emit URLs, while others attach structured source IDs that the user interface turns into footnotes. An evaluator that does not know this contract cannot tell whether the marker is valid.

Start with three questions for every cited claim:

1. Does the marker resolve to a source that was available to the model for this response?
2. Does that particular source support the claim next to the marker?
3. Did every claim that requires evidence receive at least one supporting citation?

The first question is deterministic. Compare the parsed marker with the source manifest saved beside the response. There is no reason to ask an LLM whether [policy-3] exists when a set lookup can answer exactly.

The second question can be deterministic for exact quotations, identifiers, dates, or values copied from structured data. Paraphrases usually need semantic judgment. Even then, judge the claim against its cited passage, not against every retrieved chunk. Supplying the whole retrieval set answers a broader question: whether the answer is supported somewhere. It does not validate the pointer the reader will open.

The third question measures coverage. Suppose an answer contains four material claims. Three have good citations and the last has none. Citation precision can be perfect for the three links that exist, while coverage is incomplete. A single average hides whether the system cited the wrong source or simply failed to cite a claim.

Current Ragas documentation lists metrics for RAG quality, natural-language comparison, and agent behavior, but it does not present a universal class named CitationAccuracy that understands your product's citation grammar. Do not invent that import. Ragas can evaluate useful pieces of the problem. Faithfulness evaluates whether response claims are consistent with retrieved context. QuotedSpansAlignment checks qualifying quoted spans against retrieved sources. Your application still owns marker parsing, source resolution, claim boundaries, and the policy that says which claims require citations.

Consider three failures that often collapse into one dashboard label.

In the first, the retriever returns two policy chunks in this order: refund-2025 and refund-2024. The model correctly uses the 2025 rule, but the rendering layer sorts the source cards by title after generation. A positional marker [1] now opens the 2024 text. The answer remains supported by the original retrieval snapshot, yet the displayed citation is wrong.

In the second, [refund-2025] resolves correctly. The cited chunk explains the refund window, while the nearby claim states an exception for damaged goods that appears only in a different chunk. A whole-context faithfulness check may find the exception elsewhere and pass. A per-citation check correctly fails the mapping.

In the third, every emitted citation is valid and supportive, but the final sentence introduces a new eligibility rule without a marker. Precision over existing citations remains high. Claim coverage catches the missing evidence.

Keep the retrieval snapshot as part of the test record. At minimum, store the source ID, document version, chunk ID, rank, and the text actually sent to the model in an access-controlled store. Add a content hash so you can detect corruption or accidental replacement. If policy forbids copying the text into a CI artifact, retain a durable reference to the controlled snapshot instead. A live URL or hash alone cannot reconstruct changed evidence.

Build deterministic checks before asking an LLM to judge support

The cheapest bugs should fail first. Parse the exact citation syntax your product promises, reject unknown IDs, and record uncited claims before spending tokens on semantic scoring. This also gives the person debugging the run a precise error instead of an unexplained decimal.

The following module validates a structured answer made of claims. Each claim carries the citation IDs produced by the application. Structured citations are preferable to recovering claim boundaries from rendered prose, but the same source-manifest check applies if your parser extracts bracketed markers.

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class Source:
    source_id: str
    document_version: str
    text: str


@dataclass(frozen=True)
class Claim:
    claim_id: str
    text: str
    citation_ids: tuple[str, ...]
    requires_citation: bool = True


@dataclass(frozen=True)
class ReferenceAudit:
    unknown_by_claim: dict[str, tuple[str, ...]]
    uncited_claims: tuple[str, ...]
    duplicate_source_ids: tuple[str, ...]

    @property
    def passed(self) -> bool:
        return not (
            self.unknown_by_claim
            or self.uncited_claims
            or self.duplicate_source_ids
        )


def audit_references(
    claims: Iterable[Claim],
    sources: Iterable[Source],
) -> ReferenceAudit:
    source_list = list(sources)
    source_ids = [source.source_id for source in source_list]
    known_ids = set(source_ids)

    duplicates = tuple(
        sorted(source_id for source_id in known_ids if source_ids.count(source_id) > 1)
    )
    unknown: dict[str, tuple[str, ...]] = {}
    uncited: list[str] = []

    for claim in claims:
        if claim.requires_citation and not claim.citation_ids:
            uncited.append(claim.claim_id)

        missing = tuple(
            sorted(citation_id for citation_id in set(claim.citation_ids)
                   if citation_id not in known_ids)
        )
        if missing:
            unknown[claim.claim_id] = missing

    return ReferenceAudit(
        unknown_by_claim=unknown,
        uncited_claims=tuple(sorted(uncited)),
        duplicate_source_ids=duplicates,
    )

This code does not claim that a known source supports a claim. It deliberately stops at referential integrity. Mixing entailment into this function would make a deterministic failure depend on a model response.

A small test suite should cover more than the happy path. The diagnostic case below contains one unknown marker and one uncited material claim. It also proves that a conversational sentence can be exempt when your product policy does not require evidence for it.

import unittest

from citation_contract import Claim, Source, audit_references


class CitationContractTests(unittest.TestCase):
    def setUp(self) -> None:
        self.sources = [
            Source(
                source_id="refund-2025",
                document_version="2025-06-01",
                text="Standard purchases may be returned within 30 days.",
            ),
            Source(
                source_id="damaged-items",
                document_version="2025-04-12",
                text="Damaged items must be reported within 48 hours.",
            ),
        ]

    def test_reports_unknown_and_uncited_claims_separately(self) -> None:
        claims = [
            Claim(
                claim_id="c1",
                text="Standard purchases have a 30-day return window.",
                citation_ids=("refund-2025",),
            ),
            Claim(
                claim_id="c2",
                text="Damaged items must be reported within 48 hours.",
                citation_ids=("damaged-2024",),
            ),
            Claim(
                claim_id="c3",
                text="Sale items use a separate eligibility rule.",
                citation_ids=(),
            ),
            Claim(
                claim_id="c4",
                text="I can help you check an order.",
                citation_ids=(),
                requires_citation=False,
            ),
        ]

        audit = audit_references(claims, self.sources)

        self.assertFalse(audit.passed)
        self.assertEqual(
            audit.unknown_by_claim,
            {"c2": ("damaged-2024",)},
        )
        self.assertEqual(audit.uncited_claims, ("c3",))
        self.assertEqual(audit.duplicate_source_ids, ())

    def test_rejects_ambiguous_source_manifest(self) -> None:
        duplicated = [self.sources[0], self.sources[0]]
        audit = audit_references([], duplicated)
        self.assertEqual(audit.duplicate_source_ids, ("refund-2025",))


if __name__ == "__main__":
    unittest.main()

Run these tests before any model-backed evaluator. The expected output below is the standard unittest success shape for the two fixtures, not a performance measurement.

python -m unittest -v tests.test_citation_contract

# test_rejects_ambiguous_source_manifest ... ok
# test_reports_unknown_and_uncited_claims_separately ... ok
# Ran 2 tests
# OK

If your application emits prose instead of structured claims, preserve the raw structured model output before rendering whenever possible. Recovering citations from Markdown is fragile. Brackets also appear in arrays, math, and user-provided text. If parsing rendered text is unavoidable, freeze the grammar and test escaped brackets, adjacent markers, repeated markers, punctuation after markers, and markers inside code blocks.

Do not assign meaning by list position after the response has been generated. A stable source ID should survive reranking in the interface, deduplication, and pagination. If product design requires numeric footnotes, keep an immutable map such as {"1": "refund-2025"} beside the answer. The UI can display [1], but evaluation should resolve it through the saved map rather than reconstructing the order.

There is a maintenance cost. Structured claims and durable IDs add fields to your generation and storage path. The payoff is localization. When the evaluator says unknown source refund-2024, the team checks data flow and rendering. When it says claim c2 is unsupported by damaged-items, the team checks generation or evidence selection. Those are different owners.

Use Ragas only for the semantic layer it actually covers

Once reference integrity passes, decide what kind of support the claim needs. Exact quotations and paraphrased claims are not interchangeable.

Ragas documents QuotedSpansAlignment for text inside quotation marks. The metric compares qualifying quoted spans with retrieved_contexts, normalizing case and whitespace according to its documented options. Its default minimum span length is three words. The same documentation says that an answer with no qualifying quoted spans receives a value of 1.0 because there is nothing to verify. That behavior is reasonable for quote alignment, but it means the metric cannot detect an uncited paraphrase or enforce that citations exist.

The code below uses the collections API shown in the current official documentation. It runs quote alignment without an LLM, then runs Faithfulness for one claim against only the source cited by that claim. It prints the returned values and reasons rather than asserting a universal threshold.

import asyncio

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness, QuotedSpansAlignment


async def inspect_claim_support() -> None:
    client = AsyncOpenAI()
    evaluator_llm = llm_factory("gpt-4o-mini", client=client)

    quoted = QuotedSpansAlignment()
    quote_result = await quoted.ascore(
        response='The policy says "returned within 30 days" [refund-2025].',
        retrieved_contexts=[
            "Standard purchases may be returned within 30 days."
        ],
    )

    faithfulness = Faithfulness(llm=evaluator_llm)
    support_result = await faithfulness.ascore(
        user_input="What is the return window?",
        response="Standard purchases have a 30-day return window.",
        retrieved_contexts=[
            "Standard purchases may be returned within 30 days."
        ],
    )

    print(
        {
            "quoted_spans_alignment": quote_result.value,
            "quoted_spans_reason": quote_result.reason,
            "claim_support": support_result.value,
            "claim_support_reason": support_result.reason,
        }
    )


if __name__ == "__main__":
    asyncio.run(inspect_claim_support())

Calling Faithfulness with one claim and its cited source is a diagnostic pattern, not a new Ragas citation metric. It narrows the evidence supplied to the documented faithfulness calculation. Keep the exact evaluator model, Ragas version, prompt configuration, input claim, and source text with the result. A later model or prompt can judge a borderline paraphrase differently.

Do not strip qualifiers while creating claim units. “Returns are allowed within 30 days” and “unopened returns are allowed within 30 days” are not the same proposition. An over-aggressive sentence splitter can turn a supported qualified claim into a broader unsupported one. Claim extraction therefore needs its own reviewed fixtures, especially for tables, bullet lists, headings, and sentences with multiple clauses.

Also keep citation correctness separate from factual correctness. A passage can support a statement that is outdated or wrong in the real world. Citation evaluation asks whether the answer accurately represents its supplied evidence. Verifying the evidence itself requires authoritative-source selection, freshness checks, or domain review.

For a result that will block a release, calibrate semantic judgments against human-labeled examples from your domain. Include clear support, clear contradiction, partial support, and insufficient evidence. A few easy examples prove that the pipeline runs, not that the judge handles the boundary your product cares about.

Read the evidence that separates citation bugs from their near-misses

Begin an investigation with the saved response artifact, not the aggregate score. You want to see the raw claim, its marker, the resolved source ID, the exact source text, and the retrieval snapshot before any UI transformation.

A useful deterministic failure record looks like this:

case_id: damaged-item-window
status: failed
response_version: answer-0187
claim:
  id: c2
  text: "Damaged items must be reported within 48 hours."
  citation_ids:
    - damaged-2024
reference_integrity:
  known_source_ids:
    - refund-2025
    - damaged-items
  unknown_source_ids:
    - damaged-2024
semantic_support:
  status: not_run
  reason: "Reference integrity failed before semantic evaluation."

Those values are an illustrative fixture, not production measurements. The important part is the state transition. A semantic score is absent because the deterministic precondition failed. Recording zero would be misleading: zero would imply the support evaluator ran and rejected the claim.

Now compare four problems that look like “bad citation” in a user report.

The renderer changed the mapping. The raw model output cites source s1, and the saved manifest maps s1 to the correct chunk. The browser shows [1] next to a different source after client-side sorting. The evaluator should pass the generation artifact and fail an end-to-end rendering assertion. Fix the stable mapping in the presentation path. Changing the retriever or judge will not help.

The retriever missed the needed document. The answer either guesses a claim or omits it because no supporting source reached the model. The saved top-k snapshot lacks the gold document ID. This is a retrieval recall problem first. Adding harsher citation prompting may produce more refusals, but it cannot cite evidence that was never present.

The index contains a stale version. The source ID resolves and its text supports the answer, but both reflect last year's policy. Compare the stored document version and content hash with the expected corpus manifest. Citation mapping is internally correct. Freshness or ingestion failed. A semantic judge operating only on retrieved text has no basis to know the source is obsolete.

The support judge is unstable. Reference integrity, source snapshots, and claim extraction remain byte-for-byte identical, while repeated model-backed evaluations disagree on a borderline paraphrase. Inspect the reason and per-claim decision, not only the total. Pin the evaluator configuration, add the case to calibration data, and send genuine ambiguity to review. Retrying until a pass turns uncertainty into hidden flakiness.

One more failure produces almost the same "unsupported claim" record but needs an infrastructure fix. The generator may have received the complete cited passage while the semantic evaluator received a truncated, normalized, or stale copy under the same source ID. In that case the claim can be supported in the saved response view and unsupported in the evaluator input. Rewriting the answer prompt or relaxing the support threshold would treat artifact divergence as model behavior.

Carry the source identity and content hash across retrieval, generation, evaluation, and rendering. A healthy record shows one resolved source ID and the same hash at every stage that is supposed to consume identical text. A genuine unsupported mapping shows matching hashes, a completed support decision, and a reason tied to content that does not entail the claim. An evaluation-wiring defect shows the expected source ID but a different hash or length at the judge boundary. A wrong source mapping usually changes both the resolved ID and hash. These patterns distinguish evidence quality from evidence transport.

The misleading value is often the live source. An investigator opens today's URL, sees a passage that supports the claim, and concludes that the failed judge was wrong. The frozen evaluator hash may show that the run used an older revision, a partial chunk, or text before a rendering transformation. Read the case ID, claim ID, resolved source ID, document version, per-stage hashes, evaluation status, score, and reason in that order. A decimal without those comparability fields cannot establish which text was judged.

Hash equality proves sameness, not support, and hash inequality proves difference, not which copy is correct. Retaining stage-level hashes is cheap compared with retaining every duplicate passage, but it adds join fields and operational discipline. Reviewers still need controlled access to the actual frozen text for semantic disputes. Do not present a content hash as if it lets a human assess meaning.

The Ragas quote metric has another near-miss worth testing. An answer can contain no quotation marks and still display a citation after a paraphrase. QuotedSpansAlignment has no qualifying span to compare, so its documented no-quote behavior is not evidence that the citation is accurate. Your report should label that row “not applicable to paraphrase” or route it to claim support evaluation, rather than interpreting the quote score as a pass for the whole answer.

When logs expose only the final response and URLs, add instrumentation before adjusting thresholds. Capture:

• the retrieval request and stable result IDs in rank order
• the exact chunks sent to the generator
• the model's structured citations before rendering
• the numeric-footnote-to-source map, if the UI uses numbers
• the rendered answer and source-card order
• every deterministic validation error
• the semantic evaluator version, result, and reason

Redact sensitive text according to your data policy, but retain enough identity to join the artifacts. Hashes can prove two chunks differ, yet a reviewer still needs approved access to content when deciding semantic support.

Roll the checks into CI without turning judge noise into flakiness

Introduce the evaluator in stages. First run the deterministic contract on a small set of reviewed cases. Include one unknown ID, one duplicate manifest ID, one required claim without citations, one optional conversational claim, and one correct multi-citation claim. These cases verify your own data flow without an external model.

Next run semantic evaluation in shadow mode. Save per-claim results but do not block a release. Compare disagreements with labels from engineers or subject-matter reviewers. The goal is to find where the evaluator's interpretation differs from your product contract. Do not choose a threshold simply because it makes the current build green.

After calibration, gate failures that are both reliable and actionable. Unknown source IDs, ambiguous manifests, missing required citations, and broken renderer mappings are strong blocking candidates because they are deterministic. Borderline semantic support may begin as a review state. Clear contradictions can become blocking once the labeled set shows the evaluator recognizes them consistently enough for your risk tolerance.

A CI job can keep deterministic and model-backed steps visibly separate. This workflow assumes the first two examples are saved as citation_contract.py and tests/test_citation_contract.py, and the Ragas example is saved as scripts/run_semantic_support.py:

name: citation-evaluation

on:
  pull_request:
    paths:
      - ".github/workflows/citation-evaluation.yml"
      - "rag/**"
      - "evaluation/citations/**"
      - "citation_contract.py"
      - "tests/test_citation_contract.py"
      - "scripts/run_semantic_support.py"
      - "requirements-eval.txt"

jobs:
  citation-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-eval.txt
      - name: Run deterministic citation contract
        run: python -m unittest -v tests.test_citation_contract

  semantic-support:
    needs: citation-contract
    if: github.event.pull_request.head.repo.fork == false
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-eval.txt
      - name: Evaluate support on reviewed claims
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          set -o pipefail
          python scripts/run_semantic_support.py | tee semantic-support-report.txt
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: semantic-support-report
          path: semantic-support-report.txt

The secret-dependent job is skipped for forked pull requests in this example. The exact policy depends on your CI provider, but never expose evaluator credentials to untrusted changes. Keep the deterministic job available because it needs no model key and gives contributors fast feedback.

The two-job design costs more wall-clock time than one bundled script. It also prevents a provider timeout from hiding a broken source ID. Run the fast contract on every relevant change. Reserve the model-backed suite for reviewed cases, scheduled runs, or changes that affect retrieval and generation. This reduces expense while preserving a clear release signal.

Migrate an existing suite slice by slice:

1. Inventory citation formats and declare one canonical structured representation.
2. Backfill stable source IDs and document versions for reviewed fixtures.
3. Save the renderer map for numeric footnotes.
4. Add deterministic failures in reporting-only mode and assign each category an owner.
5. Fix data-shape failures before collecting semantic baselines.
6. Human-label the support boundary cases that matter to your domain.
7. Pin evaluator dependencies and configuration.
8. Enable blocking only for categories with a documented response.

Version the dataset when a claim, source snapshot, citation requirement, or expected outcome changes. Do not silently replace a source passage under the same test-case version. The old result answered a question about the old evidence. Preserving that lineage lets you distinguish product regression from corpus correction.

For a suite that already stores only final answers, land provenance capture before adding semantic gates. First save source IDs, document versions, and the pre-render citation map without changing release policy. Then add per-stage hashes and verify in shadow runs that generation, evaluation, and rendering receive the intended snapshots. Backfill older fixtures as new reviewed revisions rather than guessing their missing maps. The first break is often a cached fixture whose numeric markers depended on a source order nobody preserved. Mark it non-comparable until a reviewer rebuilds the mapping from protected evidence.

The rollout is working when deterministic failures keep their category across repeated runs, stage hashes agree where equality is required, and semantic review receives the exact cited passage rather than the whole convenient retrieval bundle. A lower mean support score is not necessarily a rollout regression. It can be the honest result of replacing an overly broad evidence input with the source the user actually opens.

Ownership follows the first disagreement. Ingestion owns document version and source identity. Retrieval owns the ranked snapshot delivered to generation. The application team owns structured markers and the renderer map. The evaluation team owns claim extraction, judge input, and result status. The content owner decides whether a passage genuinely supports a domain claim. A handoff needs the case and claim IDs, raw marker, resolved ID, document version, all relevant hashes, frozen cited text under approved access, judge configuration and reason, renderer order, and the first stage whose artifact differs. Without that packet, teams can each demonstrate a healthy local view while the end-to-end citation remains broken.

Avoid reporting only a mean. Release owners need counts for invalid references, missing required citations, unsupported mappings, evaluation errors, and review cases. A row that could not be evaluated must not disappear into an average or become zero by default.

Know the costs and the cases where this evaluation is the wrong tool

Fine-grained citation tests add storage and review work. Saving exact chunks increases artifact size and may increase privacy obligations. Stable identifiers require coordination between ingestion, retrieval, generation, storage, and rendering. Per-claim semantic checks create more evaluator calls than scoring one complete answer against all context. Human calibration consumes specialist time.

Those costs buy a more useful failure. You can decide whether that value is warranted by the product risk.

Do not require citations for every assistant utterance. Greetings, navigation hints, explicit refusals, and content generated from fields your product labels as uncited may not need evidence markers. Mark those cases as exempt through reviewed policy. Letting the model decide ad hoc whether a claim is exempt makes coverage impossible to audit.

Do not use quote alignment to evaluate ordinary paraphrases. It answers whether quoted spans occur in the supplied sources. If the product discourages direct quotations, that metric may have little work to do. Use claim support against the cited passage instead.

Do not use citation evaluation as a substitute for retriever evaluation. If the relevant source never appears in the retrieval snapshot, inspect recall, filters, permissions, indexing, and query construction. A citation checker can describe the consequence, but it cannot identify which eligible documents the retriever should have returned without reference data.

Do not treat internal support as proof of real-world truth. A model can cite an obsolete, low-authority, or corrupted document accurately. Corpus governance, source authority, effective dates, and conflict resolution sit upstream.

Do not make an LLM judge the blocking oracle when a deterministic comparison exists. Product IDs, source membership, URL allowlists, content hashes, required fields, and duplicate keys should remain code. Semantic evaluation is valuable where language meaning matters, not where a set comparison is sufficient.

Finally, do not auto-fail genuinely ambiguous evidence without a review path. A passage may imply a claim without stating it, or two approved sources may conflict. Preserve the claim, cited text, evaluator reason, and reviewer decision. That disputed row is useful calibration data. Hiding it behind retries or averaging removes the exact case your evaluation needs to learn from.

This evaluation does not prove that the cited source is accessible to the user who received the answer. A stable ID may resolve inside the evaluator while the browser link expires, crosses a tenant boundary, or requires permissions the user lacks. Add an end-to-end access check under the intended user persona for that contract. Citation support can be perfectly measured while the product still presents unusable evidence.

A practical release rule can stay narrow: block unresolved source IDs and missing mandatory citations immediately, review disputed semantic support, and route missing gold evidence to retrieval testing. Each outcome has a different owner and a different repair.