Permission-Aware RAG Tests for Cross-Tenant Retrieval Leakage

Permission-aware RAG has a binary security invariant before it has a relevance objective: evidence outside the requesting principal's authorization set must not cross the retrieval trust boundary. Filtering a final answer is too late if an unauthorized chunk already reached a reranker, model prompt, cache, trace, or analyst dashboard.

The test strategy must exercise identity, policy, indexing, ranking, generation, and observability as one path. Synthetic tenant canaries make leakage detectable without putting real customer data at risk, while authorized-recall and latency measurements ensure the secure path remains usable.

Define the security invariant and trust boundary

Write the invariant in terms of observable artifacts: for principal P, corpus snapshot C, policy version V, and request time T, every document or derived chunk entering ranking, reranking, generation, citation, caching, or logging must belong to the eligibility set authorized for that tuple. Unknown eligibility is denied, not guessed.

Define where authorization is trusted. Identity verification, group expansion, ACL resolution, and policy evaluation should happen in controlled components, not inside a prompt. The model may explain an access denial, but it must not decide that a persuasive query overrides a document policy.

The MCP security best-practices guide is an official threat-model reference for authorization boundaries and delegated access. Apply the same discipline to RAG connectors: bind access to the actual user context, avoid confused-deputy behavior, and restrict credentials and scopes to the operation being performed.

Build a principal and policy matrix

Test more than tenant ID. Include users with no groups, one group, overlapping groups, tenant administrator roles, read-only roles, suspended users, service identities, and principals whose membership changes during a session. Add document states such as tenant-private, group-shared, user-private, globally public, revoked, and time-limited.

Every case stores identity issuer, subject, tenant, roles, groups, token audience, authentication time, policy version, request time, and expected eligible document IDs. Keep authentication and authorization expectations separate: a valid token can still lack permission for a source.

Include negative combinations that look plausible. Reuse the same user ID string in two synthetic tenants, use group names shared across tenants, and create resources whose external IDs collide while canonical tenant-scoped IDs differ. These cases catch joins or cache keys that omit tenant scope.

Create synthetic canary corpora

Build at least two synthetic tenants with parallel document structures. Give each a unique, harmless marker and several documents with nearly identical wording. If retrieval separates tenants only because topics differ, the test will not exercise authorization. Matched documents force the ACL boundary, not semantics, to decide eligibility.

Create positive canaries the principal may retrieve and negative canaries it must never observe. Include markers in body text, titles, metadata, chunk boundaries, and derived summaries. Never use real customer secrets, production credentials, or personal data. The fixture should remain safe even if every artifact is exposed to a test reviewer.

Version ACL changes. A document can move from private to shared, a group membership can be revoked, or a tenant can be disabled. Preserve event order and use a controlled clock so cache invalidation and session refresh behavior can be reproduced.

{
  "caseId": "acl-tenant-cross-014",
  "principal": {
    "subject": "user-7",
    "tenant": "tenant-a",
    "groups": ["support"],
    "policyVersion": "policy-12"
  },
  "query": "Find the launch marker for the support handbook",
  "mustAllow": ["tenant-a:handbook:v3:chunk-2"],
  "mustDeny": ["tenant-b:handbook:v3:chunk-2"],
  "deniedMarkers": ["CANARY_TENANT_B_9Q7"],
  "requestTime": "2026-06-30T10:00:00Z"
}

This record is illustrative. Store generated fixture provenance and expected policy reasoning outside the model-visible payload.

Enforce authorization before ranking

The LangChain retrieval documentation describes search across indexed data as a distinct runtime step. The authorization filter belongs in the trusted path that determines which indexed data is eligible for that search. Validate that every chunk carries a resolvable parent policy and that derived representations inherit the intended scope.

Post-filtering a global top list creates two problems. First, unauthorized candidates may cross internal boundaries before removal. Second, authorized recall can collapse when denied documents consume the top ranks. Pre-ranking eligibility or a tenant-isolated index avoids those specific failure modes, though its implementation and operational tradeoffs still require testing.

If a backend cannot enforce the full policy, place a trusted retrieval service in front of it and ensure raw results never escape that service. Oversampling followed by filtering may help authorized recall, but it does not excuse logging or transmitting unauthorized candidates. High-consequence designs may require physical or cryptographic isolation rather than metadata filtering alone.

Assert every observable stage

Do not assert only the final answer. Inspect first-stage candidate IDs, metadata, similarity scores, reranker input, selected context, prompt payload, citations, model output, caches, traces, metrics labels, error messages, and review queues. The first unauthorized artifact marks the boundary failure.

Use stable canary detection on exact synthetic markers and canonical IDs. Also inspect summaries and paraphrases because generation can reveal content without repeating the marker. For deterministic fixtures, expected facts can be enumerated and prohibited at the claim level.

An authorization assertion should run before relevance scoring. A semantically perfect answer based on another tenant's document is a security failure, not a high-quality retrieval. Stop the case, preserve a restricted incident trace, and avoid sending leaked content into ordinary evaluation tooling.

Probe adversarial and confused-deputy paths

Query exact denied titles, unique markers, known document IDs, shared employee names, and phrases copied from the negative canary. Ask the assistant to compare tenants, quote an earlier user's context, ignore filters, or use an administrator connector. These prompts should not change the principal's eligibility set.

Exercise connectors with separate credentials. A service account that can read all tenants must apply user authorization before data is retrieved, not merely after a privileged search. Verify resource indicators, tenant binding, token audience, delegated scopes, and policy version at each call boundary supported by the architecture.

Test indirect paths: query rewriting, synonym expansion, hybrid branches, reranking, fallback search, document previews, autocomplete, citation resolution, and tool retries. A primary vector path may filter correctly while a lexical fallback or citation fetch uses a global index.

Test revocation, caches, and failures

Warm every cache with an authorized request, then repeat the same query as an unauthorized principal. Cache keys and stored values must include the identity and policy dimensions needed to prevent reuse. Repeat with users in the same tenant but different groups, and with the same subject string in another tenant.

Apply a revocation event and test before and after the declared propagation boundary. Old chunks, generated answers, search results, and source previews must stop appearing as required by policy. Long-lived sessions should refresh authorization according to the product contract rather than retaining access indefinitely.

Simulate identity-provider, policy-service, index, and connector failures. Protected retrieval should fail closed when eligibility cannot be established. Verify that retries do not drop filters, fallbacks do not broaden scope, and error messages do not reveal that a denied document exists.

Scope labels and corpus splits correctly

Relevance judgments are principal-specific. A document can be the best semantic answer and still have zero eligibility for the requester. Store relevant and authorized as separate fields keyed by policy version and time. Authorized recall is calculated only over eligible relevant evidence; unauthorized exposure is a separate security metric.

Keep fixture tenants, document families, query templates, and canary markers out of model training and prompt examples. Split source and generated variants before tuning retrieval. A held-out security set should include unseen marker formats and policy combinations, while a stable sentinel set protects known failure paths.

Assessors should receive only data they are permitted to view. Cross-tenant pool construction can itself leak content into an annotation platform. Use synthetic fixtures for broad attack testing and tenant-scoped review environments for any real-world relevance analysis.

Measure security, quality, and latency together

The primary security count is unauthorized artifacts observed at any declared boundary, broken down by stage and policy case. The release requirement for a controlled test suite should be zero observed leakage, while acknowledging that passing tests does not prove the absence of all vulnerabilities. Expand threat coverage continuously.

Alongside security, report authorized recall at k, first eligible rank, empty-result rate, abstention behavior, and answer groundedness. Overly broad denial can make the system secure but unusable. Never combine leakage and relevance into one average where a quality gain can offset an authorization failure.

Measure policy lookup, filtered retrieval, reranking, cache, and end-to-end latency under realistic concurrency and ACL cardinality. Compare tenant-isolated and filtered designs under the same workload if architecture is still being selected. Thresholds belong to local service objectives; the non-negotiable condition is that timeout or overload must not disable authorization.

Review uncertainty and ablations safely

Authorization outcomes should be deterministic for a frozen identity, policy, resource, and time. If policy data is incomplete or conflicting, classify the result as denied or error according to the fail-closed contract. Do not let a model judge resolve security ambiguity.

Relevance and answer judgments can still be uncertain. Preserve assessor disagreement and unknown labels without weakening the deny assertion. A candidate may require more relevance evidence while already failing security on one definite unauthorized ID.

Run unsafe ablations only against synthetic corpora in an isolated environment. Comparing no-filter, post-filter, pre-filter, and tenant-isolated paths can locate the control that prevents leakage and quantify authorized-recall effects. Never disable authorization against real multi-tenant data for convenience.

Execute a permission-aware release plan

1. Freeze the identity tuple, policy engine version, trust boundary, corpus snapshot, and fail-closed behavior.
2. Build matched synthetic tenant corpora with allowed and denied canaries, collisions, ACL changes, and controlled clocks.
3. Enforce eligibility inside a trusted retrieval boundary and validate document-to-chunk policy inheritance.
4. Assert candidates, reranker inputs, prompts, citations, caches, traces, and outputs before calculating relevance.
5. Probe exact markers, shared topics, connector delegation, query rewrites, fallbacks, revocations, and dependency failures.
6. Report unauthorized exposure separately from authorized recall, groundedness, empty results, latency, and uncertainty.
7. Block release on any definite leak or fail-open path; repair the earliest boundary and rerun the complete policy matrix.

Permission-aware retrieval is secure only when authorization survives every transformation and failure path. Synthetic canaries, complete trace assertions, and principal-scoped relevance labels turn cross-tenant isolation from an assumption into a repeatable release test.