Testing Long-Term Agent Memory Write, Recall, and Deletion Policies

Long-term agent memory testing begins before recall. First prove that the system writes only eligible facts into the correct identity and purpose scope. Then prove selective retrieval, correction, expiry, and deletion across every derived copy. A helpful remembered preference does not compensate for one leaked secret or one record that returns after deletion.

The LangGraph persistence overview distinguishes checkpointers, which hold thread-scoped graph state, from stores, which hold application-defined data across threads. That distinction is essential: clearing a chat window, deleting a checkpoint, and deleting a durable user memory are different operations with different evidence.

Inventory every memory surface

Draw the data path from input to model and back. Include raw session turns, compacted summaries, graph checkpoints, long-term records, vector or search indexes, retrieval caches, prompt caches under your control, analytics exports, moderation artifacts, replication, queues, and backups. Mark which system is canonical and which copies can regenerate another.

The OpenAI Agents SDK sessions guide describes session memory as conversation history retrieved before a run and appended after it, with operations such as getting items and clearing a session. Those mechanics should not be mistaken for a complete long-term-memory privacy policy. A session clear can succeed while a separately extracted preference remains in a durable store.

For each surface, record owner, tenant key, user key, conversation key, purpose, encryption boundary, retention trigger, deletion method, indexing lag, and audit source. Unknown copies are a test blocker because absence cannot be verified against an incomplete inventory.

Turn memory eligibility into a policy table

Define categories with explicit actions. A stable language preference may be eligible after notice or user choice. A one-time delivery address, authentication code, medical detail, unverified accusation, or secret pasted for immediate troubleshooting should have a different disposition. Do not rely on the model to infer institutional policy from tone.

Every stored item needs at least a subject, tenant, namespace, purpose, source, observation time, policy revision, lifecycle status, and expiry or retention class. If the system extracts an inference rather than a direct fact, label it as inferred and retain enough provenance to challenge or refresh it.

Represent policy outcomes beyond allow and deny: store exact, store normalized, request confirmation, redact sensitive fields, keep session-only, or discard. Test each branch. A denial should also prove that no embedding, queue message, trace payload, or retry job contains the rejected candidate.

Test writes at the commit boundary

Capture the memory candidate before policy, the policy decision, and the committed canonical record. Use canary strings that identify field and tenant without resembling real personal data. Assert normalization rules for whitespace, locale, units, and casing without losing semantic distinctions.

Inject a crash before policy, after policy but before commit, after canonical commit but before index update, and after index update but before acknowledgement. Retry the write with a stable operation ID. The final state should follow the product's deduplication rule and must not create competing active memories from one request.

Concurrency matters when two sessions update the same preference. Exercise last-write policy only if time ordering is trustworthy; otherwise use versions, compare-and-set, or explicit conflict state. Never let a delayed extraction from an older conversation overwrite a newer user correction silently.

Evaluate cross-session recall selectively

Create a write session and a separate recall session that shares the authorized user identity but not the original transcript. Ask one question that requires the memory, one where the memory is helpful but optional, one unrelated question, and one prompt designed to retrieve another tenant's canary. Inspect retrieved records before scoring prose.

Measure retrieval eligibility, relevance, freshness, and attribution separately. Correct final text can hide a failed retrieval if the model guessed; incorrect final text can occur despite correct retrieval because the model ignored or misused it. Require citation to an internal memory ID in evaluation traces, then remove that ID from user-facing output unless product design calls for visible provenance.

Test conflict resolution: current user statement versus stored preference, verified account record versus old inferred memory, and two active memories with incompatible values. Define precedence and whether correction is immediate, confirmation-gated, or scoped to the current session.

Prevent identity and purpose leakage

Use a matrix of tenant, user, workspace, role, and agent purpose. Identical natural-language prompts should retrieve different canaries for different users and nothing for an unauthorized combination. Include changed account membership, impersonation support flows, shared devices, merged identities, and deleted workspaces.

Namespace filters must be enforced by trusted application code or storage authorization, not generated by the model as free text. Treat a missing tenant filter as a security failure even if semantic search happens to return only the expected record in a small fixture set.

Purpose limitation is narrower than identity. A memory approved to personalize documentation answers may not be eligible for advertising, employee evaluation, or a different agent. Encode purpose in the query and in the stored record, then test that a broad search endpoint cannot bypass it.

Exercise correction, expiry, and supersession

A correction should not leave two equally retrievable truths. Mark the old record superseded, update indexes, invalidate caches, and ensure ranking cannot revive it. Keep non-sensitive lineage if the product needs an audit, but do not place old secret values into change logs.

Use a controllable clock to test expiry immediately before, at, and after the policy boundary. Expired records must disappear from retrieval even if asynchronous physical cleanup runs later. A read path should enforce status and expiry rather than trusting that a scheduled purge has already completed.

Test summaries derived from several memories. Removing one source may require regenerating or deleting the summary because the removed fact can remain paraphrased. Likewise, an embedding can encode the deleted text and must be treated as a derived personal-data artifact under the product policy, not as harmless numeric metadata.

Verify deletion as a distributed workflow

Deletion starts by authenticating the requester and resolving scope: one fact, one purpose namespace, one session, or all memory for a subject. Mark the canonical key with a tombstone or deletion generation before asynchronous purge. That barrier prevents a stale indexer, retry queue, or replica reconciliation job from recreating an older value.

This TypeScript interface makes evidence explicit without retaining content:

type DeletionReceipt = {
  requestId: string
  subjectKey: string
  scope: 'memory-item' | 'purpose' | 'all-long-term-memory'
  deletionGeneration: number
  canonicalStatus: 'deleted' | 'not-found'
  derivedStores: Record<string, 'deleted' | 'pending' | 'not-applicable'>
  backupDisposition: 'expires-under-retention-policy' | 'not-present'
  verifiedAt: string
}

Poll each derived store to its documented terminal state, then run exact-key, metadata-filtered, semantic, and prompt-driven recall probes. Inspect queues for older generations and replay them in a safe fixture to prove tombstones win. The receipt should contain IDs and outcomes, not the removed plaintext or vector.

Backups require honest language. An online deletion may not rewrite immutable backup media immediately. Record the approved retention window, restore controls, and a process that reapplies tombstones before restored data serves traffic. Do not report "fully deleted" while recoverable copies remain outside the stated policy.

Build a memory adapter that is testable

Keep policy and storage behind a narrow contract. The following JSON is an illustrative fixture schema, not a framework requirement:

{
  "fixtureLabel": "illustrative memory candidate",
  "subjectKey": "tenant-a:user-17",
  "namespace": "assistant-preferences",
  "purpose": "response-personalization",
  "fact": {"preferredLanguage": "Hindi"},
  "source": {"type": "user-stated", "sessionId": "session-write-4"},
  "policyRevision": "memory-policy-test-r3",
  "expiresAt": "2030-01-01T00:00:00Z",
  "status": "active"
}

Adapter tests should cover evaluate_candidate, put_if_current, search_authorized, supersede, expire, and delete_scope behaviors even if production names differ. Integration tests then verify the real database, index, cache, and queue preserve the same contract.

Use a failure model with visible evidence

Add storage outage, partial index failure, duplicate deletion, deletion of an absent item, deletion concurrent with recall, and account closure during extraction. Fail closed for new writes when policy cannot be evaluated. For reads, return a classified degraded state rather than substituting memory from an unauthorized cache.

Execute a decisive memory plan

1. Inventory session, checkpoint, long-term, derived, cached, queued, replicated, exported, and backup surfaces.
2. Approve a versioned eligibility matrix covering purpose, sensitivity, provenance, scope, confirmation, and expiry.
3. Fault-inject writes around policy and commit boundaries with concurrent corrections and stable operation IDs.
4. Test required, optional, irrelevant, stale, conflicting, and cross-tenant recall in new sessions while inspecting retrieval evidence.
5. Exercise supersession and a controllable expiry clock across canonical records, indexes, caches, and summaries.
6. Delete by verified scope, establish a tombstone generation, purge every derived store, and prove stale work cannot resurrect data.
7. Block release on unauthorized recall, secret storage, policy bypass, resurrection, unverifiable backup handling, or audit payloads that retain deleted content.

Long-term memory is a governed data product, not a larger prompt. The release is ready only when useful recall and reliable forgetting are both observable, scoped, and enforceable.