The chatbot remembered it, until the next release

A customer corrects the delivery city from Mumbai to Pune, confirms the order, and sees Mumbai in the final summary. The model sounds confident because the application supplied a coherent history, just the wrong version of it. A one-turn chat test will never expose that defect.

The phrase “memory failure” hides several different bugs. The write may have failed. The next request may have loaded another conversation. A trimming rule may have removed the correction. A summarizer may have kept the original value. The model may also receive the correct fact and still answer incorrectly. A useful retention test identifies which boundary lost the fact before it scores the prose.

Trace where a fact can disappear

Text generation requests do not acquire your application’s earlier turns by magic. A multi-turn product must provide conversation state using whatever mechanism its integration supports. OpenAI's conversation state guide, for example, documents manual history and provider-managed state as explicit integration choices. Some applications retrieve durable memories or send a compressed summary plus recent turns. The important test fact is the same: conversation-specific state must reach the current request before the model can use it reliably.

Follow one business fact through six observable stages. First, capture what the user said, including the turn number and conversation identity. Second, inspect what storage committed. Third, inspect what the next request loaded. Fourth, record the selection or compaction decision. Fifth, capture the model-facing payload or a privacy-safe structural representation of it. Sixth, evaluate the answer. If you retain only the final answer, five possible causes collapse into one vague failure.

Give test facts stable identities that are separate from their wording. shipping_city is an identity. “Please send it to Pune instead” is one expression of its value. That distinction matters when a correction replaces an earlier value. It also helps when the assistant paraphrases “Tuesday after 3 PM” as “Tuesday afternoon.” Your ledger can assert which fact version was active without forcing every answer to repeat the fixture sentence.

Start with facts that have an objective product meaning. Delivery city, selected subscription, preferred language, allergy warning, approval status, and a user’s explicit “do not email me” instruction all work. Avoid beginning with taste questions such as whether the assistant preserved the “spirit” of a story. Those need a judge and a rubric, which adds uncertainty before the plumbing is proven.

Not every earlier statement should survive. A deleted address must disappear. An expired quote should not be treated as current. A fact from one account must never enter another account’s chat. Define retention together with revision, deletion, scope, and expiry. Otherwise a suite can reward the product for remembering information it was required to forget.

Distance is another part of the contract. “Remember this later” is not testable until later has a boundary. Useful boundaries come from the product: the last-message window, a summary refresh interval, a session handoff, a browser reload, a worker restart, or a migration between storage versions. Put cases immediately before and after each boundary. Twenty random filler turns do not tell you much if the trimming rule activates at a token budget rather than a turn count.

Use semantically varied distractors. A shipping test should include another city in a different role, such as the warehouse location, because naive retrieval may choose the most recent city token. A scheduling test should include several dates while keeping only one active appointment. A privacy test should include a fact in conversation A and ask for it from conversation B. These cases exercise selection, not mere keyword survival.

The first diagnostic question is not “Did the model remember?” Ask, “At which stage did the active fact stop matching the ledger?” A database row with Pune followed by a request payload containing Mumbai is a context-builder defect. A payload containing Pune followed by an answer claiming Mumbai is a model-level miss or an instruction conflict. Those failures can look identical in a screenshot, but they belong to different owners and need different fixes.

Build an oracle that a regression can actually fail

A retention oracle needs a falsifying change. If deleting the persistence call, reversing correction order, or reusing the wrong conversation ID would still pass, the test is decorative. Split the oracle into deterministic checks for state transport and a narrower answer check for the model’s behavior. Pytest's assertion guidance shows how plain assertions retain the compared values in a failure report.

The following Python module reduces a sequence of fact events to the active memory state. It handles replacement and deletion rather than assuming every introduced value remains valid forever. The tests fail if the implementation changes to first-write-wins, ignores a deletion, or mixes conversations.

from dataclasses import dataclass


@dataclass(frozen=True)
class FactEvent:
    conversation_id: str
    sequence: int
    fact_id: str
    value: str | None


def active_facts(events: list[FactEvent], conversation_id: str) -> dict[str, str]:
    active: dict[str, str] = {}
    relevant = sorted(
        (event for event in events if event.conversation_id == conversation_id),
        key=lambda event: event.sequence,
    )
    for event in relevant:
        if event.value is None:
            active.pop(event.fact_id, None)
        else:
            active[event.fact_id] = event.value
    return active


def test_latest_correction_replaces_the_old_value() -> None:
    events = [
        FactEvent("order-41", 4, "shipping_city", "Pune"),
        FactEvent("order-41", 1, "shipping_city", "Mumbai"),
    ]
    assert active_facts(events, "order-41") == {"shipping_city": "Pune"}


def test_deleted_fact_is_not_returned() -> None:
    events = [
        FactEvent("chat-a", 1, "phone", "+91-555-0101"),
        FactEvent("chat-a", 2, "phone", None),
    ]
    assert active_facts(events, "chat-a") == {}


def test_other_conversation_cannot_replace_the_active_value() -> None:
    events = [
        FactEvent("chat-a", 1, "account_tier", "starter"),
        FactEvent("chat-b", 2, "account_tier", "enterprise"),
    ]
    assert active_facts(events, "chat-a") == {"account_tier": "starter"}

In a real suite, call the production reducer or repository instead of copying its logic into the test. The compact implementation above makes the contract executable, but duplicating production code inside an oracle can reproduce the same bug twice. Prefer expected values written from business rules and fixtures reviewed by someone other than the implementation author.

Next, assert the selected context before invoking a live model. A model call is slow, variable, and expensive compared with a pure context test. If the request builder already dropped shipping_city, another model run adds no diagnostic value. Record enough of the model-facing request to prove inclusion, order, scope, and version. Do not dump raw customer conversations into CI artifacts. Stable fact IDs, hashes, turn numbers, and redacted values are usually enough for failure analysis.

Parameterized cases earn their keep here because each row can represent a different rule, not a paraphrase of the same scenario. Pytest runs each parameter set as a separate case, so one row can verify a correction, another deletion, a third tenant isolation, and a fourth the compaction boundary. Give each case a readable ID so the failure report names the broken promise.

from dataclasses import dataclass
import pytest


@dataclass(frozen=True)
class ContextItem:
    conversation_id: str
    fact_id: str
    value: str


def select_context(items: list[ContextItem], conversation_id: str) -> list[ContextItem]:
    return [item for item in items if item.conversation_id == conversation_id]


@pytest.mark.parametrize(
    ("case_id", "items", "conversation_id", "expected"),
    [
        (
            "keeps-active-preference",
            [ContextItem("c1", "language", "Tamil")],
            "c1",
            [("language", "Tamil")],
        ),
        (
            "does-not-cross-conversations",
            [
                ContextItem("c2", "account_tier", "enterprise"),
                ContextItem("c1", "account_tier", "starter"),
            ],
            "c1",
            [("account_tier", "starter")],
        ),
        (
            "keeps-all-items-for-the-conversation",
            [
                ContextItem("c1", "language", "Tamil"),
                ContextItem("c2", "language", "English"),
                ContextItem("c1", "account_tier", "starter"),
            ],
            "c1",
            [("language", "Tamil"), ("account_tier", "starter")],
        ),
    ],
    ids=lambda value: value if isinstance(value, str) else None,
)
def test_context_selection(
    case_id: str,
    items: list[ContextItem],
    conversation_id: str,
    expected: list[tuple[str, str]],
) -> None:
    selected = select_context(items, conversation_id)
    observed = [(item.fact_id, item.value) for item in selected]
    assert observed == expected, f"{case_id}: selected {observed}"

Do not turn every fact into a substring assertion against natural prose. Exact matching is appropriate when the value must be copied exactly, such as an order number or a consent state. It is weak for a response that may legitimately paraphrase. For those answers, ask the application to expose structured claims in test mode, or run a separate extractor whose errors you have measured on labeled examples. Keep the transport assertion independent so a grader cannot turn a missing payload into a pass.

One practical answer contract is a list of fact IDs and normalized values used in the final response. This metadata may come from a deterministic rendering layer, a structured model response, or a reviewed extractor. The test below catches an obsolete value, a missing active constraint, and an unexpected claim. It does not pass merely because the output is fluent.

def assert_claims(
    actual: list[dict[str, str]], expected: dict[str, str]
) -> None:
    by_id: dict[str, str] = {}
    for claim in actual:
        fact_id = claim["fact_id"]
        assert fact_id not in by_id, f"duplicate claim for {fact_id}"
        by_id[fact_id] = claim["value"]
    assert by_id == expected


def test_confirmation_uses_current_order_facts() -> None:
    response_claims = [
        {"fact_id": "shipping_city", "value": "Pune"},
        {"fact_id": "delivery_window", "value": "Tuesday 15:00-18:00"},
    ]
    assert_claims(
        response_claims,
        {
            "shipping_city": "Pune",
            "delivery_window": "Tuesday 15:00-18:00",
        },
    )


def test_confirmation_rejects_old_and_current_values_for_one_fact() -> None:
    response_claims = [
        {"fact_id": "shipping_city", "value": "Mumbai"},
        {"fact_id": "shipping_city", "value": "Pune"},
        {"fact_id": "delivery_window", "value": "Tuesday 15:00-18:00"},
    ]
    try:
        assert_claims(
            response_claims,
            {
                "shipping_city": "Pune",
                "delivery_window": "Tuesday 15:00-18:00",
            },
        )
    except AssertionError as error:
        assert str(error).startswith("duplicate claim for shipping_city")
    else:
        raise AssertionError("duplicate stale and current claims were accepted")


def test_confirmation_rejects_stale_missing_and_unexpected_claims() -> None:
    expected = {
        "shipping_city": "Pune",
        "delivery_window": "Tuesday 15:00-18:00",
    }
    invalid_cases = {
        "stale": [
            {"fact_id": "shipping_city", "value": "Mumbai"},
            {"fact_id": "delivery_window", "value": "Tuesday 15:00-18:00"},
        ],
        "missing": [
            {"fact_id": "shipping_city", "value": "Pune"},
        ],
        "unexpected": [
            {"fact_id": "shipping_city", "value": "Pune"},
            {"fact_id": "delivery_window", "value": "Tuesday 15:00-18:00"},
            {"fact_id": "discount_code", "value": "SAVE20"},
        ],
    }

    for case_id, response_claims in invalid_cases.items():
        try:
            assert_claims(response_claims, expected)
        except AssertionError:
            continue
        raise AssertionError(f"{case_id} claims were accepted")

These functions test the oracle, not the chatbot. The live conversation test must pass the claims returned by the application into assert_claims; replacing that call with the sample list would create a test that cannot observe product behavior. The duplicate case proves that an old and current value cannot collapse into one dictionary entry and pass by insertion order.

The startswith in the duplicate case is not laziness, and swapping it for == breaks the test. Pytest rewrites the bytecode of every module it collects so that a bare assert reports the values that made it fail, and it appends that introspection to the message of the AssertionError it raises. Because assert_claims lives in a collected test module, its rewritten guard raises duplicate claim for shipping_city followed by a newline and assert 'shipping_city' not in {'shipping_city': 'Mumbai'}, so an exact-equality check on str(error) fails on healthy code. A prefix check, or pytest.raises(..., match=...) which searches rather than compares, survives the rewriting. The alternative is to move the guard into a plain helper module that pytest does not collect and raise a purpose-built exception type from it, which is the better shape once several tests depend on the message. Either way, do not settle the question by weakening the assertion until it passes: run the test once with a duplicate claim and once without, and confirm it is red exactly when the guard is gone.

Equality is deliberately strict because the final confirmation is a transaction boundary. For a brainstorming assistant, requiring every available fact would be wrong. Match the oracle to what the response is required to claim, not everything the system happens to know.

Work through three failures that look like forgetting

The corrected-city example is a versioning test. The fixture starts with Mumbai, inserts unrelated turns, records a correction to Pune, and asks for a final order confirmation. Preserve the event sequence, the reduced active state, the selected context, and the response claims. If storage contains both events and the reducer selects Mumbai, inspect ordering types first. Lexicographic ordering of sequence strings can place 10 before 2. Timestamp ties can also make a “latest” query unstable when no secondary order exists.

The strongest negative control changes the correction while keeping the rest of the transcript identical. Replace Pune with Nashik and expect Nashik. A test that still passes probably checks only that some city appeared. Another useful mutation removes the correction event and expects Mumbai. Together, those variants prove the oracle responds to the source fact rather than a hard-coded word in the final prompt.

The second example is an expiry failure that resembles lost memory. A travel assistant stores a passport-expiry reminder with a product rule that it remains active until the trip ends. The user asks about documents after a session reload, and the reminder is absent. Storage and conversation identity are correct. The retrieval query filtered it out because the service compared a date-only expiry in the user’s time zone with a UTC instant. The answer looks like ordinary forgetting, but the evidence shows a policy boundary calculation.

Test the date immediately before expiry, exactly at the documented boundary, and immediately after it. Freeze the clock through your application’s clock interface rather than changing the machine clock. State the time zone in the fixture. A “remember for seven days” case without a reference zone or inclusivity rule is not a testable requirement. The fix may be to store an absolute instant, or it may be to preserve a local date because the business promise is calendar-based. QA should force that decision before encoding an oracle.

The third example is cross-conversation contamination, the dangerous inverse of forgetting. A support agent opens two browser tabs for two customers. One tab asks about an enterprise renewal. The other is a starter account. If a cache key uses only the agent’s user ID, the second chat can receive facts selected for the first. A conventional retention score may call this excellent recall because the model repeats a real fact. The scope oracle must reject it because the fact belongs to another conversation and customer.

Use two concurrent conversations with deliberately conflicting values. Give both the same human operator and different tenant IDs, conversation IDs, and account tiers. Interleave writes, then load each context several times. Assert both inclusion of its own active facts and exclusion of the other conversation’s facts. Run this at the repository and request-builder layers. A final-response-only test may miss the leak if the model declines to mention the contaminated value.

Concurrency evidence differs from ordinary retrieval evidence. Look for cache keys, transaction boundaries, trace correlation IDs, and the order of writes. If rerunning each conversation alone always passes but the interleaved case fails, model nondeterminism is a poor first hypothesis. The schedule is the important variable. Preserve it in the failure artifact.

A fourth near-miss deserves attention: the assistant has the fact but follows a newer instruction that legitimately overrides it. A user first says, “Always answer in French,” then later says, “Use English for this message.” An English reply is not forgetting. Your ledger needs precedence and scope, not only chronology. Mark the first preference as persistent and the second as turn-scoped, or record the opposite product rule. Without that metadata, two reviewers can label the same response differently and both appear reasonable.

These examples also show why filler turns must have a purpose. Distractors should challenge entity resolution, ordering, budget selection, or instruction precedence. Ten greetings add length but little coverage. A smaller conversation that crosses a real summary or storage boundary is more diagnostic.

Tell a context defect from a model miss

Start from the earliest reliable artifact. Confirm the test posted the intended turn to the intended conversation. Then query the state through the same public repository or service boundary the product uses. Direct database inspection is useful as a second view, but it can mislead when the application applies event reduction, soft deletion, tenant filters, or encryption after the raw row is read.

Compare four values for the failed fact: expected active version, stored event, selected context item, and model-facing representation. Include sequence, scope, and expiry metadata. A compact diagnostic might say:

python -m pytest tests/evals/test_conversation_retention.py -q -k corrected_city
# Illustrative output from the application's assertion helper:
# F
# E  AssertionError: shipping_city diverged at context_selection
# E  expected active value: Pune (event sequence 4)
# E  stored latest value:  Pune (event sequence 4)
# E  selected value:       Mumbai (event sequence 1)
# E  request fact ids:     shipping_city@1, delivery_window@3

That output is illustrative, not a claim about pytest’s exact formatting for your custom helper. The field values show the useful shape: earliest divergence, expected and observed versions, and identifiers that let an engineer find the trace. Your helper should produce these lines from actual captured values, not from constants copied into the assertion message.

If the selected context is correct, inspect serialization. Unicode normalization, accidental truncation, role conversion, and summary rendering can alter a value between selection and transport. Capture a hash of each model-facing content block and a redacted preview. For a fact that may contain personal data, log the fact ID, version, length, and keyed digest. Do not print the address merely because the test failed.

If the exact request is correct and the response is wrong, freeze everything you can legitimately freeze: model identifier, system instructions, tools, response schema, and conversation payload. Run the same case more than once and retain every outcome. A single failure among repeated identical requests is a model-level reliability signal. A failure that follows one prompt version is more likely an instruction or prompt-regression issue. Do not call it storage loss when the fact appears in the request.

Compaction creates another diagnostic branch. Compare the pre-compaction transcript, the generated summary, and the post-compaction request. If the summary omits the fact, the model answering from that summary never had a chance. If the summary contains the obsolete version, test the reducer that feeds the summarizer. If the summary is correct but context selection chooses an older summary record, inspect versioning and cache invalidation.

Tool-mediated facts need their own provenance. Suppose a booking tool returns seat 14C and the assistant later claims 14A. Record the tool call ID, tool result, any normalized memory event, selected fact, and final claim. An assertion only against the user’s earlier text would miss the authoritative tool update. Define which source wins when user text and tool state disagree.

HTTP timeouts and cancelled streams can produce an incomplete transcript that looks like memory loss on the next turn. Check whether the assistant response or tool result was committed before the client retried. Two user-visible turns may map to three server attempts. Conversation state should be tied to committed events, not what the browser happened to render. This is also why trace IDs and idempotency identifiers belong in diagnostics.

Treat absence differently from contradiction. An answer that avoids mentioning the city may be acceptable unless the task required confirmation. An answer that states Mumbai when Pune is active is a clear contradiction. A response that asks the user to reconfirm may be a safe fallback. Give those outcomes different labels. Collapsing them into a binary keyword match hides both cautious behavior and harmful stale claims.

Roll the checks into an existing suite

Begin with capture, not a release gate. Instrument the fact path for a small set of non-sensitive test conversations. Record stable IDs at write, load, selection, request, and response boundaries. Run the cases against the current release to learn which product behaviors are intentional. This baseline is not proof that existing behavior is correct. It is a map of what a new gate would change.

Next, add deterministic repository and context-builder tests. These should run on every change because they are fast and provide precise failures. Include corrections, deletion, expiry, scope isolation, and one boundary around compaction. Keep live model calls out of that job. A red build caused by a database selector should not spend time sampling model responses.

Add a smaller integration set after the transport layer is stable. Use conversations that represent supported promises, not a scrape of production chats. Mark each required claim and allowed outcome before running the model. OpenAI's evals guide is one official example of defining test data and grading criteria before interpreting results. Store model and prompt versions with results. Review new failures during the first rollout phase instead of blocking immediately, because the team still needs to separate mislabeled expectations from regressions.

Promote cases to blocking status one contract at a time. Consent withdrawal, tenant isolation, confirmed transaction facts, and safety constraints deserve stricter gates than casual personalization. Require a reproducible deterministic failure or a model-level failure rate policy backed by enough reviewed runs. Do not turn one disputed semantic grade into an automatic rollback.

Wire the layers as separate CI jobs so ownership stays visible. The following configuration is a complete GitHub Actions job shape, assuming the repository already lists test dependencies in requirements-test.txt and registers the live_llm marker. Secrets are only needed for the explicitly triggered integration job.

name: conversation-retention

on:
  pull_request:
  workflow_dispatch:

jobs:
  deterministic-contracts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-test.txt
      - run: python -m pytest tests/context tests/evals/test_fact_reducer.py -q

  live-retention-sample:
    if: >-
      github.event_name == 'workflow_dispatch' &&
      github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-test.txt
      - run: python -m pytest tests/evals/test_conversation_retention.py -m live_llm -q
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

The manual live job is restricted to the repository's default branch because the dispatch UI can otherwise target another ref. Put its credentials in a protected environment as well when your repository uses environment approvals. The cost is more than API spend. Fact-level instrumentation adds schema and privacy work. Golden conversations need owners when product rules change. Repeated live calls add latency to the release process. Structured claim metadata may require an application change. Name those costs in the rollout proposal so the suite does not quietly decay after launch.

Keep failure artifacts short-lived and access-controlled. Synthetic conversations are preferable, but even synthetic fixtures can contain realistic phone numbers or addresses that monitoring systems mistake for production data. Use unmistakably reserved values and redact at the logger. Test the redaction itself with a canary secret, then assert the canary never appears in captured logs.

When a prompt, model, compaction strategy, or persistence schema changes, run the diagnostic layers in order. Do not regenerate expected answers from the new system and call the result a baseline. Review changed outcomes against the product contract. Otherwise the migration can erase the evidence of exactly the regression the suite was built to catch.

Know when retention is the wrong requirement

Do not preserve a fact merely because it appeared earlier. Privacy deletion, account closure, consent withdrawal, and retention limits can require the opposite. The correct test may assert that the fact is absent from storage, retrieval, and the next model request. A system that “remembers everything” is not automatically higher quality.

Avoid durable memory for transient task state when the workflow already has an authoritative source. An order service should supply the current shipping address at confirmation time. Copying that address into an LLM memory store creates another stale replica. Test that the agent consults the authoritative tool and treats its result as current instead of rewarding recall from an old chat.

Do not use a semantic judge to diagnose a deterministic pipeline. If a fact ID never appears in selected context, a grader’s opinion about the final prose cannot add useful evidence. Fix or reject the transport defect first. Judges are appropriate when several phrasings satisfy the answer contract and a deterministic claim representation is unavailable.

Skip exact answer snapshots for open-ended conversations. They punish harmless wording changes and encourage teams to freeze prompts for the wrong reason. Assert required claims, forbidden contradictions, scope, and provenance. Then sample style or helpfulness separately. A test should fail because the product broke a promise, not because punctuation moved.

Long-conversation tests are also a poor substitute for load tests. A 200-turn transcript may cross a context boundary, but it does not reproduce concurrent writes, datastore latency, or cache eviction under traffic. Use targeted retention cases for state correctness and a separate performance workload for capacity. Mixing them makes failures expensive and ambiguous.

Finally, do not block a release on an uncalibrated aggregate “memory score.” Averages hide the difference between forgetting a preferred greeting and reviving withdrawn consent. Keep case-level severity and evidence. The useful outcome is not a single impressive percentage. It is a failure report that says which active fact disappeared, at which boundary, under which version, and what user promise that loss violated.