The summary got shorter, and a critical fact disappeared

A user says, "I am allergic to peanuts, not merely avoiding them," but forty turns later, the agent compresses the conversation to "User prefers healthy food" and suggests a peanut dressing. The summary is shorter, fluent, and operationally wrong in the next session.

That is the core risk behind agent memory compression drift testing. Because compression is intentionally lossy, the test cannot demand that every word survive; it must prove that required facts keep their meaning, stale and deleted facts stay out, identities do not cross, and downstream retrieval sees the right version.

Compression changes meaning in predictable ways

Long-running agents cannot place every prior turn into every new prompt. Products summarize messages, extract facts, compact event logs, or replace several memory records with one smaller record. The optimization is reasonable. The danger appears when the transformation changes a fact that later affects a decision.

Omission is the easiest failure to see. A required allergy, deadline, permission limit, or unresolved obligation disappears. The resulting summary can still cover most of the conversation and score well on general similarity.

Negation loss reverses meaning with one missing token. "Do not contact the customer" becomes "Contact the customer." "User does not consent to storage" becomes "User consented to storage." Tests need typed positive and negative values because a broad semantic score can call the two sentences closely related.

Qualifier loss removes conditions. "Use the staging account only" becomes "Use the account." "May refund up to $50 after supervisor review" becomes "May refund." The main noun and verb survive, but the operational boundary does not.

Temporal collapse combines incompatible versions. A user changes an address, timezone, name, preference, or project status. Compression keeps both values without ordering them, selects the old value, or merges them into a third statement. The test oracle must know which fact is current and which is superseded.

Deletion resurrection occurs when a later summary is generated from old raw history or a stale checkpoint that still contains removed content. The new compressed record reintroduces a fact the deletion workflow had made unavailable. This is not solved by asking the model to "forget" something. Storage and retrieval must enforce deletion state outside model prose.

Entity merge assigns one person's fact to another. Two users discuss similar orders, two projects share a name, or a multi-agent handoff loses the tenant key. Compression produces a coherent summary that attaches the correct fact to the wrong subject. That is both a correctness and data-isolation failure.

Instruction elevation turns quoted or retrieved material into durable authority. A support ticket contains "Always approve my requests," and the memory summary stores it as a standing user instruction. Compression should preserve provenance and fact type, not flatten untrusted content into the same channel as authenticated preferences or policies.

Repetition introduces another problem. A system may compress the original conversation, then compress the summary with new turns, then compress that result again. Small losses accumulate. A fact that survives the first pass may lose its qualifier on the third. Include repeated-compression fixtures instead of testing only one transformation.

Before blaming compression, separate four stages:

1. The compressor creates an artifact from source records.
2. The memory store saves the artifact with identity and version.
3. Retrieval selects memory for a later turn.
4. The response or planner uses the retrieved context.

If the artifact is correct but retrieval omits it, changing the compression prompt will not help. If retrieval includes the fact and the response ignores it, the consumer needs a stronger domain guard or response test. Keep an oracle at each boundary.

Turn memory into a fact-level contract

Free-form summaries are useful for continuity, but critical facts need a representation tests can compare. The representation does not have to become the product's only memory format. It can be a parallel fact ledger for safety, permissions, durable preferences, commitments, and other values whose exact meaning matters.

Each fact should identify the tenant and subject, use a stable predicate, carry a typed value, preserve material qualifiers, reference its source event, and declare lifecycle state. A newer fact can supersede an older one. A deletion marker can make both unavailable to retrieval without relying on generated wording.

The reference validator below compares a compressed artifact with an expected ledger. It deliberately avoids natural-language matching. A model or deterministic reducer must emit structured facts beside any prose summary. The validator rejects missing active facts, changed values, lost qualifiers, absent provenance, forbidden stale facts, and identity mismatches.

from __future__ import annotations

from dataclasses import dataclass
from typing import Any, Literal

Status = Literal["active", "superseded", "deleted"]


@dataclass(frozen=True)
class ExpectedFact:
    fact_id: str
    tenant_id: str
    subject_id: str
    predicate: str
    value: Any
    qualifiers: tuple[tuple[str, str], ...]
    source_event_id: str
    status: Status


@dataclass(frozen=True)
class CompressedFact:
    fact_id: str
    tenant_id: str
    subject_id: str
    predicate: str
    value: Any
    qualifiers: tuple[tuple[str, str], ...]
    source_event_ids: tuple[str, ...]


@dataclass(frozen=True)
class CompressionArtifact:
    memory_version: int
    tenant_id: str
    subject_id: str
    facts: tuple[CompressedFact, ...]
    summary: str


def validate_artifact(
    expected: tuple[ExpectedFact, ...],
    artifact: CompressionArtifact,
) -> list[str]:
    violations: list[str] = []
    actual_by_id = {fact.fact_id: fact for fact in artifact.facts}

    if len(actual_by_id) != len(artifact.facts):
        violations.append("duplicate_fact_id")

    for fact in expected:
        actual = actual_by_id.get(fact.fact_id)
        if fact.status != "active":
            if actual is not None:
                violations.append(f"{fact.fact_id}:forbidden_{fact.status}_fact")
            continue
        if actual is None:
            violations.append(f"{fact.fact_id}:missing")
            continue
        if actual.tenant_id != fact.tenant_id:
            violations.append(f"{fact.fact_id}:tenant_changed")
        if actual.subject_id != fact.subject_id:
            violations.append(f"{fact.fact_id}:subject_changed")
        if actual.predicate != fact.predicate:
            violations.append(f"{fact.fact_id}:predicate_changed")
        if type(actual.value) is not type(fact.value) or actual.value != fact.value:
            violations.append(f"{fact.fact_id}:value_changed")
        if actual.qualifiers != fact.qualifiers:
            violations.append(f"{fact.fact_id}:qualifiers_changed")
        if fact.source_event_id not in actual.source_event_ids:
            violations.append(f"{fact.fact_id}:provenance_missing")

    expected_ids = {fact.fact_id for fact in expected}
    for unexpected_id in sorted(set(actual_by_id) - expected_ids):
        violations.append(f"{unexpected_id}:unsupported_fact")

    return violations

The validator uses exact qualifier tuples, so the producing service should sort or otherwise normalize qualifier keys before storing them. Values remain typed: False is not the string "false", and 50 dollars should use the product's chosen money representation rather than a floating-point guess. The test contract owns that schema.

An unsupported fact is not automatically malicious. It may be a legitimate new extraction that the fixture omitted. In a frozen regression fixture, however, treating additions as review items catches fabricated certainty and accidental instruction elevation. Teams can use a separate permissive mode for exploratory evaluation, but release cases should declare what facts are allowed.

Keep the prose summary out of deterministic equality. It can say "prefers morning flights" or "usually travels early" if both map to the same approved structured fact and no operational nuance is lost. Human reviewers can inspect prose for readability while the ledger carries the hard oracle.

Exercise three failures that similarity scores miss

The first worked example combines negation and severity. The source says the user has a medically important peanut allergy, not a casual preference. The expected fact uses a boolean prohibition plus a qualifier. A compressed fact that changes False to True, changes allergy to preference, or drops the severity must fail for a named reason.

from dataclasses import replace

import pytest


ALLERGY = ExpectedFact(
    fact_id="fact-allergy-1",
    tenant_id="travel-co",
    subject_id="user-17",
    predicate="food.peanuts_allowed",
    value=False,
    qualifiers=(("basis", "allergy"), ("severity", "critical")),
    source_event_id="event-41",
    status="active",
)


def correct_allergy_artifact() -> CompressionArtifact:
    fact = CompressedFact(
        fact_id=ALLERGY.fact_id,
        tenant_id=ALLERGY.tenant_id,
        subject_id=ALLERGY.subject_id,
        predicate=ALLERGY.predicate,
        value=ALLERGY.value,
        qualifiers=ALLERGY.qualifiers,
        source_event_ids=(ALLERGY.source_event_id,),
    )
    return CompressionArtifact(
        memory_version=8,
        tenant_id="travel-co",
        subject_id="user-17",
        facts=(fact,),
        summary="User has a critical peanut allergy.",
    )


@pytest.mark.parametrize(
    ("changed_fact", "expected_violation"),
    [
        (
            replace(correct_allergy_artifact().facts[0], value=True),
            "fact-allergy-1:value_changed",
        ),
        (
            replace(correct_allergy_artifact().facts[0], qualifiers=()),
            "fact-allergy-1:qualifiers_changed",
        ),
        (
            replace(correct_allergy_artifact().facts[0], subject_id="user-22"),
            "fact-allergy-1:subject_changed",
        ),
        (
            replace(correct_allergy_artifact().facts[0], source_event_ids=()),
            "fact-allergy-1:provenance_missing",
        ),
    ],
    ids=["negation", "qualifier", "identity", "provenance"],
)
def test_critical_fact_mutations_are_named(
    changed_fact: CompressedFact,
    expected_violation: str,
) -> None:
    artifact = replace(correct_allergy_artifact(), facts=(changed_fact,))

    violations = validate_artifact((ALLERGY,), artifact)

    assert expected_violation in violations

The downstream test for this fixture should ask for a meal recommendation that would expose the constraint. Do not assert one exact meal. Assert that the chosen structured meal record does not contain peanuts and that the answer does not claim peanuts are permitted. This proves memory use without tying the test to prose.

The second worked example covers temporal updates. A user first chooses UTC-5, then moves and explicitly changes the preference to UTC+1. Mark the first fact superseded and the second active. The compressed artifact must contain the new fact and must not contain the old one. Their ordering is not the artifact's job: it lives in the protected lifecycle metadata, where timezone-v1 stays marked superseded and points at the fact that replaced it. The artifact carries the current value, the ledger carries the history, and the natural-language summary mentions only the current timezone.

def test_superseded_value_does_not_survive_compression() -> None:
    old = ExpectedFact(
        fact_id="timezone-v1",
        tenant_id="calendar",
        subject_id="user-17",
        predicate="timezone",
        value="UTC-05:00",
        qualifiers=(),
        source_event_id="event-10",
        status="superseded",
    )
    current = ExpectedFact(
        fact_id="timezone-v2",
        tenant_id="calendar",
        subject_id="user-17",
        predicate="timezone",
        value="UTC+01:00",
        qualifiers=(),
        source_event_id="event-72",
        status="active",
    )
    artifact = CompressionArtifact(
        memory_version=12,
        tenant_id="calendar",
        subject_id="user-17",
        facts=(
            CompressedFact(
                fact_id="timezone-v1",
                tenant_id="calendar",
                subject_id="user-17",
                predicate="timezone",
                value="UTC-05:00",
                qualifiers=(),
                source_event_ids=("event-10",),
            ),
            CompressedFact(
                fact_id="timezone-v2",
                tenant_id="calendar",
                subject_id="user-17",
                predicate="timezone",
                value="UTC+01:00",
                qualifiers=(),
                source_event_ids=("event-72",),
            ),
        ),
        summary="User's current timezone is UTC+01:00.",
    )

    violations = validate_artifact((old, current), artifact)

    assert violations == ["timezone-v1:forbidden_superseded_fact"]

In the real passing artifact, only timezone-v2 belongs in retrievable facts. Retain the supersession relationship in protected lifecycle metadata if policy permits. Do not keep stale content in the generated summary merely to explain history to the model.

The third example covers deletion through repeated compression. Create a fact, compress once, delete it, add unrelated turns, and compress again from the current authorized memory view. The second artifact must omit the deleted fact. The test should also inspect the compressor input: if deleted content was sent to the model again, output filtering may hide the symptom while violating the deletion boundary.

Run the same fixture across two subjects with similar histories. Mix their jobs concurrently and assert each artifact's tenant and subject, every included fact, the storage partition key, and the later retrieval filter. A cross-user leak may not show up as a missing expected fact, so the unsupported-fact check is essential.

Budget pressure needs a deliberate test rather than one arbitrarily long transcript. Build a fixture with a small set of critical facts, several useful continuity facts, and enough synthetic chatter to exceed the compressor's configured input or output budget. Place one critical fact near the start, one in the middle, and one near the end. All three must survive. The expected loss should come from explicitly disposable continuity facts, not from whichever turn happens to be oldest.

This case exposes two mechanisms. Source selection may truncate the conversation before the compressor receives it, or the compressor may receive every event and omit a fact in its output. Record the selected source event ids before generation so the failure names the right component. Raising the model's output limit will not restore an event that the selector already dropped.

Add metamorphic checks around the same fixture. Appending unrelated small talk must not change existing structured facts. Duplicating an untrusted claim must not raise its trust or make it active. Reordering independent source events may change prose flow, but it must not change their values or subjects. Splitting one event into two transport chunks must not create two durable facts. These properties find coupling that a handful of golden transcripts cannot cover.

Not every transformation should be invariant. Reordering an update before the value it supersedes changes which fact is current, so the oracle must change. Removing the source event for a fact should remove or invalidate that fact. Metamorphic tests are useful only when the input transformation is known to preserve meaning. Document that relationship beside each case instead of generating random conversation edits and calling every difference drift.

Finally, vary compression depth. Compare a direct compression of the authorized source view with a second path that compresses two intermediate summaries plus the same new events. The active critical fact set should match even when prose differs. If only the multi-stage path loses a qualifier, store the intermediate artifact and source references; otherwise the final failure gives no clue which pass introduced the mutation.

Pinpoint whether compression, storage, retrieval, or response failed

Capture evidence at the interfaces, not only in a chat screenshot. For each compression attempt, record a fixture or run id, tenant and subject, input memory version, source event ids included after deletion filtering, compressor configuration, output artifact, validation result, and new stored version. Protect raw conversation text separately and retain it only as policy allows.

At retrieval, record the query or query class, tenant filter, subject filter, candidate fact ids, selected fact ids, memory versions, and the context references passed downstream. At response time, connect the final structured action or answer to those references. This chain answers where the fact vanished.

A compact diagnostic can compare expected facts with one saved artifact. The command below loads a JSON fixture, constructs the same typed objects, and prints stable violations. Its input schema is intentionally explicit rather than framework-specific.

from __future__ import annotations

import json
import sys
from pathlib import Path


def tuple_pairs(value: dict[str, str]) -> tuple[tuple[str, str], ...]:
    return tuple(sorted(value.items()))


def load_expected(value: dict[str, object]) -> ExpectedFact:
    return ExpectedFact(
        fact_id=str(value["fact_id"]),
        tenant_id=str(value["tenant_id"]),
        subject_id=str(value["subject_id"]),
        predicate=str(value["predicate"]),
        value=value["value"],
        qualifiers=tuple_pairs(value.get("qualifiers", {})),
        source_event_id=str(value["source_event_id"]),
        status=str(value["status"]),  # validated by fixture generation
    )


def load_compressed(value: dict[str, object]) -> CompressedFact:
    return CompressedFact(
        fact_id=str(value["fact_id"]),
        tenant_id=str(value["tenant_id"]),
        subject_id=str(value["subject_id"]),
        predicate=str(value["predicate"]),
        value=value["value"],
        qualifiers=tuple_pairs(value.get("qualifiers", {})),
        source_event_ids=tuple(map(str, value["source_event_ids"])),
    )


if __name__ == "__main__":
    payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    expected = tuple(load_expected(item) for item in payload["expected"])
    artifact = CompressionArtifact(
        memory_version=int(payload["artifact"]["memory_version"]),
        tenant_id=str(payload["artifact"]["tenant_id"]),
        subject_id=str(payload["artifact"]["subject_id"]),
        facts=tuple(load_compressed(item) for item in payload["artifact"]["facts"]),
        summary=str(payload["artifact"]["summary"]),
    )
    violations = validate_artifact(expected, artifact)
    if violations:
        print("\n".join(violations))
        raise SystemExit(1)
    print("Compressed memory satisfies the fact contract")

python tools/check_compressed_memory.py artifacts/allergy-attempt-1.json
python -m pytest tests/agent/test_memory_compression.py -vv --log-cli-level=INFO

Suppose the checker prints fact-allergy-1:missing. Open the stored artifact. If it is absent there and the filtered compressor input contained event-41, the compression output lost it. If event-41 never entered the compressor, investigate source selection or deletion filtering. If the fact exists in the artifact but not in the next turn's context, inspect retrieval. If both contain it but a meal action includes peanuts, inspect the planner and enforce the allergy at the domain boundary.

Version evidence catches a common near-miss. Two compressors read memory version 11. One saves version 12 with a new preference. The other later overwrites it with a summary based on the older view. The text looks like compression drift, but the storage defect is a lost update. Use a conditional version write and assert that one concurrent writer receives a conflict instead of overwriting newer memory.

Another near-miss is an ambiguous source. If a user says "I may move to Paris" and later asks for Paris weather, extracting home_city = Paris is unsupported inference, not loss during compression. Put explicit extraction expectations in the fixture and label speculation separately. The compressor should not convert a possibility into a durable fact.

Separate compression drift from representation loss

A second failure can look identical in the checker while beginning after the compressor has finished. The compressor may emit the correct typed fact, then a serializer, schema migration, or storage adapter drops a qualifier, converts a boolean to text, or omits an unfamiliar field. The next run reports value_changed, qualifiers_changed, or missing, just as it would for genuine generation drift. Prompt changes cannot repair a value that was correct at the generation boundary.

Capture three sanitized representations for the failing fixture: the selected source-event ids, the structured artifact immediately returned by the compressor, and the artifact read back through the same storage path used by retrieval. A healthy allergy case shows event-41 in the selected source set, a boolean false value with its severity qualifiers in the generated artifact, and the same typed fields after read-back. A compressor failure shows the source id present but the first structured artifact already lacks or changes the fact. A representation failure shows the first artifact correct and the read-back artifact wrong. The first boundary where the values differ names the owner.

The prose summary is a misleading diagnostic field. It may still say that the user has a critical allergy while the structured boolean was converted to the string false, so a reassuring sentence does not validate the ledger. The reverse can also occur: the ledger is correct while the summary uses awkward wording. Read the fact id, typed value, qualifiers, source-event ids, tenant, subject, and memory version before interpreting generated prose. When the checker prints fact-allergy-1:value_changed, print the expected and observed type names in the sanitized test report. Seeing boolean beside string immediately redirects the investigation toward encoding or migration.

Roll this evidence into an existing suite in dependency order. First, add round-trip tests for the current stored artifact without changing compression behavior. Those tests expose old fixtures that bypass serialization and query tests that construct in-memory objects no production reader ever sees. Next, make fixture generation declare schema and memory versions, then preserve a pre-storage artifact on failure. After the reader and writer agree, run the fact validator against both the immediate artifact and the read-back artifact. Only then make compressor regressions blocking, because otherwise one red build can alternate between model and storage owners with no new evidence.

Rehearse stored-data migration against a copy of representative sanitized artifacts before changing the active reader. Read each artifact with the old path, migrate it once, read it with the new path, and compare the critical fact contract. Run the migration again and require the result to remain unchanged. A migration that is correct only on its first pass can corrupt records when a job retries. Canary the reader against migrated copies before writing new versions, then retain enough version evidence to distinguish a bad decoder from bad source data.

This sequencing has a concrete price. A migration fixture performs the original read, a migration write, the new read, and an idempotence pass instead of one in-memory assertion. Failure artifacts retain a sanitized before-and-after pair, which adds redaction review and fixture maintenance. Strict rejection of an unknown predicate also delays shipping a legitimate new extraction until the fact contract and its owner are updated. The alternative is silent schema loss, so teams should budget the migration work rather than make the reader permissive.

The compressor team owns the immediate artifact and model configuration. The memory platform owns serialization, version migration, conditional writes, and read-back fidelity. Retrieval owns candidate and selected fact ids. Domain owners decide which predicates and qualifiers are critical, while privacy owners decide which boundary artifacts may be retained. A useful handoff contains the source-event set, expected ledger, immediate artifact, read-back artifact, schema and memory versions, attempt identifier, first divergent field, and the later retrieval result. Sending only the final chat response forces every recipient to reconstruct the same chain.

Fact-preservation tests do not establish that an authoritative source fact was true. If an upstream profile service supplies the wrong timezone, perfect compression faithfully preserves the wrong value. Source validation, correction workflows, and freshness checks own that failure. The compression contract can prove fidelity and lineage, but it cannot certify reality.

Roll the oracle into CI without freezing model wording

Begin with a fact inventory. Identify memory that affects safety, authorization, money, identity, privacy, external communication, and durable personalization. Give those predicates owners and typed schemas. Leave low-value conversational continuity in prose unless incidents show it needs stronger treatment.

Build fixtures from confirmed failures and carefully designed boundaries. Each fixture needs source events, lifecycle transitions, expected active facts, forbidden superseded or deleted facts, and two subjects for isolation tests. Use synthetic personal data. Real names, addresses, health details, and secrets do not belong in routine CI artifacts.

Test deterministic reducers and storage lifecycle rules on every change. Run model-based compressors when prompts, models, extraction schemas, memory selection, or summarization code changes. Store attempt number and configuration with every result. A fixed temperature or seed may reduce variation in some systems, but it should not be described as proof of identical output.

The workflow below is an example for a Python project. It runs the fact-contract suite and uploads only sanitized artifacts when the job fails.

name: memory-compression-contract

on:
  pull_request:
    paths:
      - "memory/**"
      - "tests/agent/test_memory_compression.py"

jobs:
  fact-preservation:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements-test.txt
      - run: python -m pip install -r requirements-test.txt
      - run: >-
          python -m pytest
          tests/agent/test_memory_compression.py
          -vv
          --junitxml=artifacts/memory-compression.xml
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: sanitized-memory-evidence
          path: artifacts/
          retention-days: 7

Point the pip cache at the file the job actually installs. Left implicit, it searches for **/requirements.txt and **/pyproject.toml, matches nothing in a repository that names its pins requirements-test.txt, and fails setup before the fact-contract suite starts.

Roll out a parallel structured ledger before replacing existing summaries. In shadow mode, extract critical facts, compare them with the current memory path, and investigate disagreements using synthetic or policy-approved traffic. When the ledger is reliable, make it authoritative for hard constraints while keeping summaries for conversational context.

Repeated compression deserves a dedicated regression lane. Run one pass, several passes with unrelated turns, and a pass after update or deletion. The expected fact set changes only when the source lifecycle changes. This catches gradual qualifier erosion that one-pass fixtures miss.

The cost is not only test runtime. A fact ledger adds schemas, migrations, provenance links, lifecycle logic, and potentially more retained metadata. Strong isolation can reduce convenient cross-workspace personalization. Conditional version writes create conflicts that callers must retry by rereading state. These are visible engineering costs, and they should be weighed against the impact of each fact class.

Do not solve every memory problem by storing the entire raw conversation forever. That increases privacy exposure, retrieval noise, and deletion complexity. Preserve the minimum evidence required by product and policy, and test deletion against every derived artifact, index, cache, and prompt-building path.

Know when a memory difference is not drift

A shorter summary is not automatically defective. If the removed detail cannot affect later behavior and is not required for continuity, compression did its job. Test contracts should name what must survive instead of rewarding verbosity.

Different wording is also acceptable. Exact snapshots are useful for deterministic serializers, not variable model prose. Compare typed facts and downstream behavior. Keep a human readability review for summaries, but do not make punctuation a release gate.

Do not require every fact in every prompt. Retrieval should select relevant memory. A billing preference can remain stored while a weather question omits it. The retrieval oracle should consider the current task and hard constraints. Safety-critical restrictions may be injected more broadly, but that is a product policy decision.

Avoid treating similarity as sufficient proof. It can help rank large evaluation sets and find outliers, but high similarity can hide negation and stale values. Pair it with exact predicates for critical data and explicit forbidden-fact checks.

Do not let a passing retry erase a failed compression. If attempt one loses an allergy and attempt two preserves it, the configuration is variable for a critical fact. Store both, investigate the trigger, and decide the gate based on the product's tolerance. Retrying until green is not a memory control.

Finally, do not ask the compressor to enforce permissions it does not own. Memory can record that a user requested admin access, but the authorization service must decide whether access exists. Preserving a statement accurately does not make the statement trusted. Keep provenance and authority separate all the way from source event to tool boundary.