Testing RAG Freshness Against Superseded Documents

RAG freshness is not the age of a document. It is whether the retrieved evidence was valid for the user's requested time, jurisdiction, product, and authority. A policy published yesterday may not be effective until next month, while an older amendment may remain the governing source for a historical question.

Testing freshness requires a temporal corpus, not a list sorted by updated_at. Preserve the document lifecycle, issue time-bound queries, verify retrieved versions and citations, and exercise caches and deletion paths. When source authority is ambiguous, the correct system behavior is explicit uncertainty rather than a confident guess.

Define the temporal contract

Choose which time the question asks about. A current query uses the system's declared evaluation clock. A historical query includes an as-of instant. A future-effective query may ask what will apply later. If the user provides no date, the product must define whether "current" means request time, business date in a jurisdiction, or another clock.

Then define applicability beyond time: tenant, region, product version, plan, language, authority, and document status. Supersession can be total or partial. One amendment may replace only a section, while the rest of the older policy remains active. Store validity at the smallest evidence unit used for retrieval when partial replacement matters.

Do not use file modification time as legal or business validity. Publication time says when a source became visible. Effective time says when its rule applies. Ingestion time says when the RAG system learned about it. Retrieval time says when the user asked. Their differences produce distinct defects.

Model document versions and supersession

Give every logical document family a stable ID and every revision a version ID. Record source URI, content hash, publication time, effective interval, ingestion time, status, authority, supersedes relationship, and any tombstone. Chunk records should inherit or reference this metadata, not duplicate an unversioned current=true flag with no lineage.

CREATE TABLE document_versions (
  family_id TEXT NOT NULL,
  version_id TEXT PRIMARY KEY,
  published_at TIMESTAMPTZ NOT NULL,
  effective_from TIMESTAMPTZ NOT NULL,
  effective_to TIMESTAMPTZ,
  ingested_at TIMESTAMPTZ NOT NULL,
  status TEXT NOT NULL,
  superseded_by TEXT,
  content_hash TEXT NOT NULL
);

SELECT version_id
FROM document_versions
WHERE family_id = $1
  AND status = 'approved'
  AND effective_from <= $2
  AND (effective_to IS NULL OR $2 < effective_to);

This illustrative query assumes non-overlapping approved intervals. Production validation should reject accidental overlap or classify it as a conflict. A result set with two governing versions may be legitimate in different jurisdictions, but that distinction belongs in explicit applicability metadata.

Version chunks together. If an old chunk remains in the vector index after its parent is superseded, document-table correctness alone will not prevent retrieval. Build referential checks from source version through every index and cache representation.

Build a temporal scenario matrix

Include ordinary current questions, historical as-of questions, a request just before and at an effective boundary, a future policy published but not active, a revoked source, a partially superseded section, and a conflict between authorities. Test both explicit dates and natural phrases such as "before the renewal" after normalizing them to a stored interpretation.

Add ingestion-lag scenarios. At time T, the authoritative source may be effective but not yet indexed. The expected behavior depends on the service contract: abstain, use a trusted fallback, or state that records may be delayed. Returning the known obsolete rule without disclosure should not pass merely because the pipeline had not ingested the update.

Use a controllable clock for deterministic tests. Each case stores query time, expected applicable versions, prohibited stale versions, and acceptable uncertainty behavior. Never make a test depend on the machine's real current date when a fixture time can express the requirement.

Freeze corpus snapshots and temporal splits

For regression tests, package the document event timeline and all derived chunks under a corpus snapshot ID. The LangChain retrieval documentation distinguishes indexing from runtime retrieval. Apply that boundary to freshness: test event ingestion and index construction separately from query-time filtering and ranking.

Use development timelines to tune metadata filters, recency signals, query parsing, and conflict behavior. Hold out document families and later time windows for final evaluation. Randomly splitting versions from one family lets the system see the exact language and lineage of a future test revision.

Forward validation is useful when the claim is performance on future updates: tune on events before a cutoff and test on later events. Preserve a stable sentinel suite as well, because a moving time window alone makes release-to-release comparisons difficult.

Evaluate retrieval freshness directly

Start with temporal candidate recall: did the candidate set contain the relevant version applicable at query time? Then calculate fresh relevance at the final context depth. A result can be temporally eligible but irrelevant, or relevant in topic but stale. Keep these dimensions separate.

Define stale exposure at k for the product. It might count any superseded evidence in the top ranks, weight higher positions more heavily, or focus only on stale evidence that contradicts the current rule. Publish the formula, rank cutoff, and treatment of duplicates. This is a local diagnostic, not a universal standard.

Also report first fresh rank, stale-to-fresh ordering conflicts, and cases with no applicable source. For multi-source answers, require all time-sensitive evidence to be compatible at the requested instant. A fresh main policy combined with a stale exception can still produce a wrong answer.

Pool judgments across temporal and non-temporal retrieval configurations. Assessors need the query time and applicability rules but should not see candidate system identity. Preserve unknowns when source authority or effective interval cannot be resolved.

Test answer and citation time alignment

Retrieval freshness is necessary but not sufficient. The generator may ignore the current source, merge incompatible versions, or cite the fresh document while repeating a stale rule from another context. Extract time-sensitive claims and bind each to exact versioned citations.

The LangSmith evaluation concepts guide distinguishes offline and online evaluation settings. Use offline versioned cases for controlled release comparisons, then monitor online traces for new source families, ingestion lag, stale citations, and user corrections. Production findings should be reviewed before joining a sealed benchmark.

Check the answer's stated time. If a historical question asks about 2024, responding with today's policy is not fresh; it is temporally wrong. If the source timeline is ambiguous, an answer that names the conflict and requests clarification may be the expected outcome.

Exercise caches, deletes, and partial updates

Freshness defects frequently survive in caches after the primary index is correct. Test query-result caches, embedding caches, reranker caches, generated-answer caches, CDN layers, and application memory. Cache keys should include the policy or corpus version needed to invalidate obsolete results.

Simulate a supersession event through the complete pipeline. Assert that new chunks become eligible, old chunks become ineligible for current queries, historical access remains available where permitted, and all affected caches stop serving the old current answer. Measure propagation delay against the product's declared objective.

Deletion and revocation require a stricter path. A tombstoned source must not remain retrievable through a stale vector entry or cached answer. Tests should search by exact old wording and identifiers, not only typical user phrasing. Logs and evaluation artifacts must follow the same retention policy.

Run temporal ablations and diagnose failures

Compare ranking without temporal metadata, hard validity filtering, soft recency scoring, and the proposed combined policy on development data. Hard filtering can remove relevant historical evidence if query-time parsing fails. Soft scoring can leave an obsolete document above the valid version. The ablation reveals which mechanism creates each error.

Assign failures to event ingestion, metadata extraction, version linking, chunk propagation, index deletion, query-time interpretation, eligibility filtering, ranking, generation, citation, or cache invalidation. Track the earliest broken stage. Retuning embeddings is not an answer to an unprocessed supersession event.

When metadata is incomplete, test sensitivity. Evaluate under each plausible effective interval or authority ordering and ask whether the decision changes. Cases that remain ambiguous should be excluded from numeric claims or counted in a declared uncertainty category, not assigned the label most favorable to the candidate.

Include latency and availability tradeoffs

Temporal filters, version lookups, and cache invalidation can add latency or reduce cache hit rates. Measure query parsing, metadata lookup, retrieval, reranking, and source validation separately under realistic corpus sizes and concurrency. Compare candidates with the same clock, corpus, and timeout policy.

Test the quality of degraded paths. If the version service is unavailable, fail closed, abstain, or use a clearly disclosed fallback according to product risk. Silently dropping the temporal filter can turn an availability incident into a correctness incident.

Gate tail latency and stale exposure together. A fast response from a superseded source is not a successful request, and a fresh response that routinely times out may not meet the service contract. Threshold values must come from local risk and operating objectives and should be labeled illustrative during design.

Execute a freshness release plan

1. Define query-time semantics, applicability fields, source authority, and partial-supersession behavior.
2. Build immutable document and chunk timelines with publication, effective, ingestion, revocation, and deletion events.
3. Freeze temporal scenarios and corpus snapshots, using family and forward-time splits for held-out evaluation.
4. Score temporal candidate recall, fresh top-rank relevance, stale exposure, answer claims, and exact version citations.
5. Run complete supersession, cache invalidation, tombstone, conflict, and service-degradation tests.
6. Use ablations and earliest-stage attribution to separate ingestion, filtering, ranking, generation, and citation failures.
7. Release only when current and historical slices, uncertainty behavior, latency, and revocation rules meet the predeclared policy.

Freshness becomes testable when time and authority are data, not assumptions. A versioned corpus plus controlled clocks can prove which source the system should have used, which source it actually used, and whether the answer preserved that temporal boundary.