Testing Semantic Caches for Stale or Unsafe LLM Responses

A semantic cache is a decision system, not a faster dictionary. It decides that a prior response is safe and correct enough to reuse for a new request whose wording is merely similar. That decision can leak another tenant's data, replay revoked guidance, or bypass a newer safety rule.

Test cache eligibility before testing hit rate. A valid hit must match trusted scope and response contract, depend on evidence that is still current, survive current policy checks, and represent the same user intent. Similarity should rank eligible candidates; it must not create eligibility.

Define a Cache Entry as a Bound Artifact

Store more than prompt text, embedding, and answer. A defensible record includes tenant or public scope, principal class where relevant, authorization-policy version, normalized request contract, tool and prompt versions, evidence IDs and revisions, safety-policy version, creation time, expiry, and response schema.

The LangChain retrieval overview explains how RAG fetches external knowledge at query time. Caching a RAG answer freezes the result of that retrieval step, so the cache must retain enough lineage to determine whether the supporting knowledge and access decision still apply.

Partition by Trusted Security Context

Derive tenant ID, account, role, groups, region, and policy version from authenticated server context. Do not extract them from the prompt or let an embedding decide that two tenants are equivalent. The initial cache search should occur inside the authorized partition, followed by an entry-level authorization check before use.

Build tenant twins with nearly identical questions and distinct canary facts. Seed a response for Tenant A, ask the paraphrased question as Tenant B, and assert a miss even at high semantic similarity. Repeat after role revocation, group removal, tenant transfer, and session replacement. Verify logs do not expose cached snippets from rejected candidates.

A shared cache can exist for intentionally public material, but it needs an explicit public classification and separate namespace. Accidentally omitting a tenant field must fail closed rather than fall through to a global partition.

Build Equivalent and Dangerous Near-Miss Pairs

Create positive pairs that differ in harmless wording while preserving entities, time range, locale, task, and output contract. Create hard negatives that are lexically close but alter negation, amount, unit, date, jurisdiction, tenant, product tier, or requested action. Include conversational follow-ups whose meaning depends on prior turns.

Label each pair as reusable, non-reusable, or uncertain, with a reason code. Group variants by seed prompt so evaluation splits do not leak paraphrases. Sample real prompts, but scrub and classify them before they enter a cache benchmark.

Examples such as "How do I cancel a scheduled transfer?" and "How do I cancel a completed transfer?" should be near in embedding space but operationally different. The benchmark must reward the miss. Cache precision on these boundary pairs matters more than a large hit count on duplicate FAQs.

Make Freshness a Read-Time Requirement

Attach every evidence-backed response to document IDs, revisions, effective dates, and retrieval-policy version. On document update, deletion, supersession, or ACL change, emit invalidation events for dependent entries. Also verify dependencies when reading, because queues can be delayed or messages can be lost.

Use explicit expiry rules for volatile sources, but do not treat time-to-live as the only freshness mechanism. A year-long policy can be revoked minutes after caching; a stable glossary may remain valid for much longer. Event invalidation and read-time version checks cover different failure modes.

Test clock boundaries, delayed events, duplicate events, out-of-order updates, partial dependency records, and a cache node that missed the invalidation broadcast. The correct response to uncertain freshness is a miss and fresh execution, not reuse with a warning added afterward.

Bind Hits to Policy and Response Contracts

A response generated under an old system prompt, tool set, output schema, or safety policy may no longer be valid. Include those contract versions in eligibility. When a response schema adds a required field, old cached JSON should miss or pass through a tested adapter rather than causing downstream parse failures.

The MCP security best-practices guide documents authorization-related attack classes such as confused-deputy and session-hijacking risks. A semantic cache can amplify similar scope mistakes by replaying an already-authorized result into a different session. Bind entries to the resource decision that made them legal and recheck that decision for the current caller.

Reapply mandatory output safety checks on hits. The check may be cheaper than generation, but it must use the current policy. Store whether the original response required human approval or contained redacted fields; those obligations do not disappear when served from cache.

Implement Hard Eligibility Before Similarity

Design the lookup as filter, retrieve, verify, then serve. Similarity is considered only after hard dimensions match. The final verification should compare dependency and policy versions against authoritative state.

type Scope = {
  tenantId: string;
  authzVersion: string;
  safetyVersion: string;
  responseSchema: string;
};

type CacheEntry = Scope & {
  similarity: number;
  expiresAtMs: number;
  evidenceCurrent: boolean;
};

export function reusable(entry: CacheEntry, request: Scope, nowMs: number): boolean {
  return entry.tenantId === request.tenantId
    && entry.authzVersion === request.authzVersion
    && entry.safetyVersion === request.safetyVersion
    && entry.responseSchema === request.responseSchema
    && entry.expiresAtMs > nowMs
    && entry.evidenceCurrent;
}

Treat this as an eligibility skeleton, not a complete authorization system. The caller still needs an authoritative permission check, and the similarity threshold belongs after this predicate.

Calibrate Similarity by Risk Slice

Run labeled positive and hard-negative pairs through the exact embedding and distance function used in production. Inspect score distributions by task, language, prompt length, and consequence. Pick candidate thresholds against the tolerated false-hit rate, then validate on a held-out family split.

An illustrative local rule might demand a stricter threshold for account-changing guidance than for public documentation, while requiring zero cross-tenant reuse in every slice. Label such numbers as illustrative and preserve the raw scores so the rule can be recomputed after an embedding migration.

Measure hit precision, miss rate on true equivalents, stale-hit count, unsafe-hit count, cross-scope candidate count, latency, and fallback success. Overall hit rate is an operational outcome, not the safety objective. A lower hit rate can be the correct result after adding missing policy dimensions.

Attack Invalidation and Failure Handling

Fault-inject the path between source update and cache invalidation. Drop an event, deliver it twice, reverse event order, pause one cache shard, and change an ACL while a lookup is in flight. Assert that version verification rejects the old artifact and that the fresh path receives the current trusted scope.

Test corrupted embeddings, unavailable vector search, timeout during authorization, and malformed cache entries. A cache outage should normally degrade to uncached execution; an authorization outage should follow the product's fail-closed policy. Keep these outcomes distinct in telemetry.

Watch for cache poisoning. Inputs used to create shared entries need stronger provenance and moderation than private entries. Prevent untrusted users from writing into a namespace later consumed by higher-privilege users, even when response text appears harmless.

Release Checklist

• Define every trusted scope and response-contract field on an entry.
• Separate tenant, principal-bound, and explicitly public namespaces.
• Build paraphrase positives plus negation, date, unit, and scope hard negatives.
• Record evidence IDs, revisions, policy versions, expiry, and source lineage.
• Invalidate on content, ACL, prompt, tool, schema, and safety-policy changes.
• Verify current versions again before serving a candidate.
• Reapply required output checks and approval obligations on hits.
• Fault-inject delayed, dropped, duplicate, and reordered invalidation events.
• Report unsafe and stale hits separately from ordinary similarity errors.
• Preserve an immediate cache bypass and namespace purge control.

Action Plan: Prove Safety Before Chasing Hit Rate

Start with one low-risk cache namespace and write its complete eligibility contract. Build tenant twins and dangerous near-miss pairs before selecting a similarity threshold. Add evidence versioning, invalidation events, and read-time verification, then run the benchmark with cache bypass as the control.

Shadow lookups without serving cached responses. Review every candidate that failed a hard check and every labeled false hit. Canary serving only after the scope, freshness, and policy vetoes are observable, and keep a one-switch bypass for incident response. Optimize hit rate later, within the boundary established by those tests.