Rag Data Exposure Audit

RAG data exposure audit

Measure whether retrieval and generation preserve document- and tenant-level authorization. Use controlled fixtures and restricted evidence handling so the audit does not amplify any sensitive material it discovers.

When to Use

• Testing document authorization and tenant isolation in an authorized RAG or enterprise-search deployment.
• Reviewing raw retrieval fields, citations, exports, caches, logs, or fallback paths for unintended disclosure.
• Reproducing a suspected source-policy mismatch with controlled documents.

When NOT to Use

• Do not enumerate real sensitive documents or another tenant's corpus without explicit written authority.
• Do not place real credentials, personal data, or production secrets into test documents.
• Do not quote discovered sensitive content when a redacted evidence reference and digest are sufficient.

Exposure states

Keep these states separate: metadata-visible, candidate-returned, chunk-retrieved, context-present, response-disclosed, citation-exposed, exported, and downstream-propagated. One state does not prove the next.

Workflow

1. Bind the audit to exact tenants, identities, roles, corpora, connectors, routes, allowed data classes, query budget, monitoring, and cleanup duties.
2. Build an authorization matrix from authoritative source policy: identity × document or collection × operation. Preserve unknown policy as unknown.
3. Establish controlled fixtures with unique inert markers across allowed, denied, tenant-specific, revoked, deleted, and stale states where authorized.
4. Record source ACL, ingestion identity, index namespace, metadata filters, retrieval identity, model identity, cache keying, and response channel for each fixture.
5. Establish same-identity baselines for ordinary allowed and denied requests.
6. Exercise only approved identity and query pairs. Capture target-origin query, retrieval, context, citation, response, export, cache, and policy receipts.
7. Test exposure surfaces independently:
   • source-to-index ACL propagation;
   • tenant and namespace filters;
   • raw chunk, metadata, debug, and citation fields;
   • query-time versus ingestion-time authorization;
   • revoked, deleted, stale, and cached content;
   • summaries, follow-up questions, exports, logs, and fallback sources;
   • downstream tools or agents that consume retrieved content.
8. Compare observed exposure with the authoritative matrix. Distinguish policy mismatch from stale policy, cache, route, index, or identity ambiguity.
9. Grade each pair enforced, exposed, inconclusive, blocked, or not-tested. Missing telemetry cannot produce enforced.
10. Remove fixtures, verify source and derived-store cleanup, and record residual retrieval or cached disclosure.

Deliverable

Return the scope, authorization matrix, controlled-fixture register, tested identity pairs, exposure-state receipts, policy mismatches, alternative explanations, restricted evidence references, cleanup status, and owned fixes.

Rationalizations to reject

• “The model refused to quote it, so access control worked.” Retrieval, context, citations, or debug fields may already have exposed the content.
• “The source ACL is correct.” Verify propagation through connectors, indexes, filters, caches, and response paths.
• “A title is harmless metadata.” Titles and metadata can disclose sensitive existence, ownership, or project context.
• “The document was deleted.” Derived stores and caches require separate cleanup evidence.
• “An admin account can see everything, so the result is expected.” Compare the tested identity with explicit authorized purpose and tenant scope.
• “One denied query proves isolation.” Coverage requires the relevant identities, routes, fields, and lifecycle states.