When a chatbot forgets what the user is still asking

A customer asks for a refund, corrects the order number eight turns later, and receives instructions for the original order. The last reply sounds helpful when read alone, yet it is irrelevant to the conversation the customer actually had. A single-turn check will pass it. A useful evaluation has to reconstruct what remained active when the assistant answered.

Why the last-turn score misses the failure

Relevance changes as a conversation moves. A request can be introduced, narrowed, corrected, paused, completed, or replaced. The words remain in the transcript after their status changes, so simple history matching treats stale context as if it were current context. That is how an assistant can quote an earlier order number accurately and still be wrong.

The first question for a QA engineer is not, “Does the response resemble something in the transcript?” It is, “Which goals, entities, and constraints were active at this turn?” The difference matters in support, booking, healthcare administration, coding assistants, and any workflow where a user revises a decision. Old text is evidence of what happened, not proof of what the assistant should do now.

Consider a travel conversation. The user first asks for the cheapest flight, later rules out overnight travel, and finally says, “Show me the best option.” A last-message evaluator sees an underspecified request and may accept the cheapest itinerary. A conversation-aware oracle carries forward the no-overnight constraint. The same response moves from plausible to failing when the active state is included.

There are three useful levels of evidence:

1. Turn evidence records exactly what the user and assistant said, with stable turn IDs.
2. State evidence records which intents, entities, and constraints became active or inactive after each turn.
3. Decision evidence explains which active item the final response served, missed, or contradicted.

Do not collapse those levels into one score. If the stored transcript is incomplete, the test case is invalid. If the state reducer kept an obsolete entity active, the oracle is wrong. If both are correct and the response still addresses the obsolete entity, the product failed. One decimal value cannot preserve that distinction.

This also explains why a high aggregate result can hide an expensive defect. Most turns in a long chat are often straightforward acknowledgements or information gathering. One late reply can undo the whole interaction by acting on a superseded address, date, account, or approval. Averaging every turn gives the harmless turns more influence than the decision turn. Score the decision-bearing turns separately and keep their failure categories visible.

A semantic grader may help when relevance depends on paraphrase or domain judgment, but it should not be the first oracle for facts the suite already knows. Exact order IDs, dates, selected plans, explicit exclusions, and completion states are deterministic. Check those directly. Reserve semantic review for questions such as whether an explanation actually answers a broad troubleshooting goal.

The practical unit of a long conversation relevancy evaluation is therefore not “one transcript, one score.” It is a versioned case with a state transition, a response under test, and an expected relationship between them. That unit can tell you what broke without pretending that all relevance is reducible to keyword overlap.

Build an oracle from conversation state

Start by annotating transitions, not by writing a giant expected answer. A transition says that order A-119 was replaced by B-204, that “no overnight flights” remains active, or that the VPN issue was resolved before a password-rotation question began. These labels survive harmless wording changes in the assistant response.

For mature suites, keep the annotation beside the transcript rather than deriving it from the model output. If the output decides which intent was active and the evaluator then grades the output against that decision, the test is circular. A reviewer, product rule, or deterministic fixture should establish the expected state first.

The following diagnostic is deliberately narrow. It checks exact evidence that a fixture declares required or forbidden. It does not claim that substring matching understands language. That limitation is useful: a failure means a known identifier or phrase crossed a boundary, and a pass means only that these explicit checks passed.

# conversation_relevance.py
from __future__ import annotations

from dataclasses import asdict, dataclass
import json
from pathlib import Path
import sys
from typing import Iterable


@dataclass(frozen=True)
class RelevanceCase:
    case_id: str
    active_intents: tuple[str, ...]
    response: str
    required_evidence: tuple[str, ...]
    forbidden_evidence: tuple[str, ...]


@dataclass(frozen=True)
class Issue:
    code: str
    evidence: str


def present(markers: Iterable[str], response: str) -> list[str]:
    folded = response.casefold()
    return [marker for marker in markers if marker.casefold() in folded]


def inspect(case: RelevanceCase) -> list[Issue]:
    found_required = set(present(case.required_evidence, case.response))
    found_forbidden = present(case.forbidden_evidence, case.response)
    issues = [
        Issue("missing_required_evidence", marker)
        for marker in case.required_evidence
        if marker not in found_required
    ]
    issues.extend(
        Issue("revived_stale_evidence", marker)
        for marker in found_forbidden
    )
    return issues


def load_case(path: Path) -> RelevanceCase:
    payload = json.loads(path.read_text(encoding="utf-8"))
    return RelevanceCase(
        case_id=payload["case_id"],
        active_intents=tuple(payload["active_intents"]),
        response=payload["response"],
        required_evidence=tuple(payload["required_evidence"]),
        forbidden_evidence=tuple(payload["forbidden_evidence"]),
    )


def report(case: RelevanceCase) -> dict[str, object]:
    issues = inspect(case)
    return {
        "case_id": case.case_id,
        "status": "fail" if issues else "pass",
        "active_intents": list(case.active_intents),
        "issues": [asdict(issue) for issue in issues],
    }


DEMO = RelevanceCase(
    case_id="refund-order-correction",
    active_intents=("refund B-204",),
    response="I can help exchange order A-119.",
    required_evidence=("refund", "B-204"),
    forbidden_evidence=("exchange", "A-119"),
)


if __name__ == "__main__":
    cases = [load_case(Path(arg)) for arg in sys.argv[1:]] or [DEMO]
    for item in cases:
        print(json.dumps(report(item), sort_keys=True))

Running the file with no arguments evaluates its deliberate negative control. It prints this one-line record:

The record contains two missing items and two stale items. Those are fixture facts produced by the code, not measurements of a deployed assistant. More importantly, the result does not say “low relevance.” It says that B-204 and the refund intent disappeared while A-119 and the exchange intent returned.

Exact markers work well for account IDs, product SKUs, dates in a controlled format, selected option names, and phrases that policy requires. They are weaker for synonyms and natural explanations. A test that requires the literal word “refund” would reject “send the payment back,” even if that wording is acceptable. Add accepted variants only when the product contract permits them. Do not create an unreviewed thesaurus until every answer can pass somehow.

State transitions need their own deterministic test. The reducer below models four events: start an intent, replace it, add a constraint, and resolve it. It keeps the mechanism independent from any model vendor or prompt format.

# conversation_state.py
from __future__ import annotations

from dataclasses import dataclass, field
from typing import Literal


EventKind = Literal["start", "replace", "constrain", "resolve"]


@dataclass(frozen=True)
class Event:
    turn_id: str
    kind: EventKind
    intent_id: str
    value: str


@dataclass
class ConversationState:
    active: dict[str, str] = field(default_factory=dict)
    constraints: dict[str, list[str]] = field(default_factory=dict)
    resolved: set[str] = field(default_factory=set)


def reduce_state(events: list[Event]) -> ConversationState:
    state = ConversationState()
    for event in events:
        if event.kind == "start":
            state.active[event.intent_id] = event.value
            state.resolved.discard(event.intent_id)
        elif event.kind == "replace":
            if event.intent_id not in state.active:
                raise ValueError(
                    f"{event.turn_id}: cannot replace inactive {event.intent_id}"
                )
            state.active[event.intent_id] = event.value
        elif event.kind == "constrain":
            if event.intent_id not in state.active:
                raise ValueError(
                    f"{event.turn_id}: cannot constrain inactive {event.intent_id}"
                )
            state.constraints.setdefault(event.intent_id, []).append(event.value)
        elif event.kind == "resolve":
            if event.intent_id not in state.active:
                raise ValueError(
                    f"{event.turn_id}: cannot resolve inactive {event.intent_id}"
                )
            del state.active[event.intent_id]
            state.resolved.add(event.intent_id)
    return state


if __name__ == "__main__":
    state = reduce_state(
        [
            Event("t01", "start", "order-help", "A-119 exchange"),
            Event("t08", "replace", "order-help", "B-204 refund"),
            Event("t09", "constrain", "order-help", "return to original card"),
        ]
    )
    assert state.active == {"order-help": "B-204 refund"}
    assert state.constraints == {
        "order-help": ["return to original card"]
    }
    print(state.active["order-help"])

This reducer costs annotation effort. Someone has to decide that turn t08 replaces the earlier order request rather than creating a second request. That judgment should be visible in review. Hidden heuristics inside an evaluator are cheaper at authoring time, but much harder to debug when a release fails.

Protect the oracle with negative controls. A detector that passes approved replies is not enough because it may pass everything. One test should prove that a stale entity is caught, another should prove that an active constraint is caught, and a third should prove that a resolved topic is no longer active.

# test_conversation_relevance.py
import pytest

from conversation_relevance import Issue, RelevanceCase, inspect


@pytest.mark.parametrize(
    "case",
    [
        RelevanceCase(
            case_id="refund-current-order",
            active_intents=("refund B-204",),
            response="I will outline the refund steps for B-204.",
            required_evidence=("refund", "B-204"),
            forbidden_evidence=("A-119", "exchange"),
        ),
        RelevanceCase(
            case_id="daytime-flight",
            active_intents=("choose a daytime flight",),
            response="Option C departs at 14:10 and has no overnight segment.",
            required_evidence=("14:10", "no overnight"),
            forbidden_evidence=("01:20",),
        ),
    ],
    ids=lambda case: case.case_id,
)
def test_approved_transcripts_keep_active_context(
    case: RelevanceCase,
) -> None:
    assert inspect(case) == []


def test_negative_control_revives_replaced_order() -> None:
    case = RelevanceCase(
        case_id="stale-order-negative-control",
        active_intents=("refund B-204",),
        response="Here are the exchange steps for A-119.",
        required_evidence=("refund", "B-204"),
        forbidden_evidence=("exchange", "A-119"),
    )

    assert inspect(case) == [
        Issue("missing_required_evidence", "refund"),
        Issue("missing_required_evidence", "B-204"),
        Issue("revived_stale_evidence", "exchange"),
        Issue("revived_stale_evidence", "A-119"),
    ]

The approved examples test the expected contract. The negative control tests the evaluator itself. In a live integration suite, replace the recorded response string with the reply captured by your existing adapter. Keep that adapter outside the oracle module so transport errors, authentication failures, and relevance failures retain different owners.

Work through three failures that look alike

The order IDs and transcript excerpts below are illustrative fixtures. They show different failure shapes and do not represent production measurements.

A corrected entity comes back from the dead. A support chat begins with “I want to exchange order A-119.” After discussing eligibility, the user says, “Sorry, wrong order. Refund B-204 instead.” Several turns cover the payment method. The assistant finally provides exchange instructions for A-119.

The decisive evidence is the replacement event. The fixture should show A-119 becoming inactive and B-204 becoming active at the correction turn. In the failing response, both the action and entity are stale. If the saved state at generation time still contains only A-119, investigate context assembly. If the saved state contains B-204 and the response uses A-119, investigate response generation or a later transformation. The product symptom is identical, but the first incorrect artifact is different.

The fix is not to repeat the entire transcript in every assertion. Assert the active entity and action at the decision turn, plus explicit exclusions for the replaced values. Preserve the correction turn in the failure report. A diff that shows “expected B-204, observed A-119, replaced at t08” gives an engineer a place to start.

This approach costs maintenance when identifiers or accepted wording change. It is still cheaper than investigating a generic relevance score with no failing field. Keep identifiers in fixture data, not duplicated across test code, so one reviewed fixture update changes the case coherently.

An early constraint disappears while the topic stays correct. A traveler asks for flights from Bengaluru to Singapore. Midway through the chat, they add, “No overnight flights because I am travelling with a child.” Later they ask for the cheapest remaining choice. The assistant recommends an overnight itinerary that is cheaper than the daytime options.

A topic classifier may call this relevant because every turn concerns the same route. Entity checks may also pass because the cities and dates are correct. The failure is a dropped constraint. Record constraints separately from the intent label so the oracle can say that route selection stayed active while the no-overnight rule was violated.

Look at the response and the structured option it references. If the itinerary metadata identifies an overnight segment but the prose calls it daytime, the response rendering or upstream data mapping may be wrong. If the selected option is genuinely overnight and the active state includes the constraint, selection logic ignored a valid condition. If the constraint never reached the state snapshot, the conversation-memory path lost it.

The direct fix is to assert on stable itinerary attributes rather than infer time-of-day from prose. A fixture can require a chosen option ID whose reviewed attributes satisfy the constraint. That makes the test less tolerant of catalog changes, so version the option data with the transcript. When dynamic inventory makes a fixed option impossible, assert the constraint on the returned structured data and review the explanation separately.

A resolved problem steals the next answer. A user spends several turns fixing a VPN connection. They confirm, “That worked, the VPN is connected now,” then ask how to rotate their account password. The assistant responds with another VPN diagnostic step.

This is not a missing fact inside one active intent. The old intent should have moved to resolved, and a new intent should have started. A fixture that merely lists both topics cannot express the error because both VPN and password rotation really do appear in the transcript. The state needs status and turn order.

Evidence begins at the acknowledgement. Confirm that the system captured the user’s resolution statement, then inspect the state immediately before the password reply. If VPN remains active after a clear resolution event, the transition logic failed. If VPN is resolved and password rotation is active, but the answer still returns to VPN, the generation path selected stale context. If the password request never appears in the captured transcript, the problem is ingestion, not relevance.

The safest fix is to make completion events first-class in important flows. That adds state complexity, especially when a user reopens a problem. A reopened intent should create an explicit transition rather than silently changing a resolved flag. Tests then cover resolution and reopening as different paths.

These examples should not share one catch-all threshold. The refund case needs entity and action checks. The flight case needs a persistent constraint. The VPN case needs lifecycle state. A single score may help sort review work, but it cannot replace the failure-specific oracle.

Separate relevance loss from nearby defects

Wrong answers in long chats often look the same in a report: the response refers to old or unsuitable information. Before filing “context loss,” locate the earliest artifact that contradicts the fixture.

Retrieval returned stale facts. Suppose the active intent is correct and the assembled request asks about the current refund policy, but the retrieved document is an obsolete policy revision. The assistant may answer that document faithfully. The conversation relevance check should pass the active-intent relationship and a freshness or grounding check should fail the source. Calling this context loss sends the issue to the wrong team.

Save source identifiers, revision fields, or retrieval timestamps when the application exposes them. Compare those values with the fixture’s approved source set. Do not infer a retrieval failure just because the final answer is wrong. Without source evidence, label the cause unknown and reproduce at the integration boundary.

A tool or service failed. A travel assistant may preserve the no-overnight constraint but receive no usable itinerary because a dependency timed out. A fallback such as “Please try again later” is unhelpful, yet it has not revived an old intent. Test tool completion and error handling separately. If the fallback suddenly answers an earlier topic, then relevance also failed, and the report should carry both findings.

The transcript was truncated before evaluation. An evaluator cannot judge a constraint it never receives. Compare the case version and turn IDs entering the application with those entering the evaluator. Missing t05 is an evaluation-pipeline defect even if t05 contains the constraint that would have made the response fail. Do not mark the assistant as passing or failing from incomplete evidence.

The response is relevant but factually wrong. An assistant can address the refund for B-204 and invent an unsupported processing rule. Relevance passes because the response serves the active request. Factuality, grounding, or policy compliance fails. Keeping these dimensions separate prevents teams from “fixing relevance” by adding more topical words to a false answer.

The response is correct but delayed by one turn. Streaming, retries, or asynchronous updates can attach the right reply to the wrong turn ID in a test harness. Check correlation IDs and recorded ordering before diagnosing model memory. If the product itself shows the answer under the wrong user message, that is still a user-visible defect, but its mechanism is sequencing.

A useful failure bundle contains:

• the redacted transcript with stable turn IDs;
• the expected state after each decision-bearing turn;
• the actual state snapshot supplied to the response path, when available;
• the final response before any UI formatting;
• retrieved source or tool evidence, if relevant;
• the exact deterministic rule that failed;
• the evaluator version and case version.

Inspect those artifacts in that order. The first divergence usually gives you the responsible boundary. A screenshot alone proves the symptom. It rarely proves whether capture, state reduction, retrieval, generation, or rendering introduced it.

Pytest’s assertion report is useful here because the failing parameter ID and compared values remain visible. Give cases readable IDs such as refund-order-correction rather than generated indexes. When many cases fail, run one node ID directly and preserve its logs before changing retries or thresholds.

For the negative control in the earlier test module, an unexpected detector result produces a comparison showing the missing or extra Issue objects. For a release transcript, format the assertion message with the case ID, active intents, and issue list. The report field should contain a code such as revived_stale_evidence, not a prose guess such as “the model forgot.”

Wire evidence into CI without hiding the cause

CI should answer two questions quickly: did the evaluator still catch its negative controls, and did a reviewed product transcript violate a stable contract? Run those lanes separately. If the oracle lane fails, product results from the same run are not trustworthy.

The workflow below assumes the two Python files shown earlier live at the repository root and the tests live under tests/conversation_relevance. It uses documented pytest invocation and JUnit XML output. The application-specific step that captures new responses should run before these checks and write fixtures in the reviewed schema.

name: conversation relevance checks

on:
  pull_request:
    paths:
      - "conversation_relevance.py"
      - "conversation_state.py"
      - "tests/conversation_relevance/**"
      - "evals/conversation-cases/**"
  workflow_dispatch:

permissions:
  contents: read

jobs:
  oracle:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install pytest
      - run: >
          python -m pytest -q
          tests/conversation_relevance/test_oracle.py
          --junitxml=oracle-results.xml

  reviewed-cases:
    needs: oracle
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install pytest
      - run: >
          python -m pytest -q
          tests/conversation_relevance/test_release_transcripts.py
          --junitxml=conversation-results.xml

Installing pytest without a project lock is concise for the example but weak for a real release gate. Pin it through the repository’s normal test dependency mechanism. The pin adds update work, while leaving it floating lets dependency changes alter the gate without a product change.

Use the command line to reproduce one case before rerunning the whole suite:

set -o pipefail

python conversation_relevance.py \
  evals/conversation-cases/refund-order-correction.json \
  | tee refund-order-correction.result.json

python -m pytest -vv \
  tests/conversation_relevance/test_release_transcripts.py \
  -k "refund_order_correction" \
  --log-cli-level=INFO

The first command saves the structured finding. The second selects the named test and shows live log records at INFO level. If the case is parametrized, its pytest node ID is even more precise than a keyword expression. Use the node ID printed during collection or in the prior failure report rather than guessing it.

Keep raw model prompts and transcripts out of general CI logs when they can contain customer data. Redact at collection time, not after a failure has already been uploaded. Stable synthetic transcripts are preferable for the release gate. Production samples belong in a controlled review path with retention and access rules set by the organization.

Do not retry a relevance failure automatically. A retry can produce a different answer and turn a reproducible regression into a green build. Retries are appropriate only when the failure category proves that setup or transport was transient. Record the first response even if the surrounding integration job later retries.

Release reporting should count invalid cases, deterministic product failures, semantic-review cases, and infrastructure failures separately. Those counts describe the run. They do not establish a universal model quality rate, especially when the dataset is curated around known risks.

Roll out the gate without freezing product changes

Dropping hundreds of old transcripts into a blocking job usually creates noise. Some fixtures encode obsolete product behavior. Others lack the turn where a constraint changed. A few pass only because their expected answer was copied from one historical response. Treat migration as test development, not bulk score generation.

Begin with incident-backed cases. Choose conversations where the active intent and the incorrect response are not disputed. Add one repaired response and one negative control for each mechanism. This gives the suite immediate value and exposes whether the evaluator can fail for the intended reason.

Next, run new cases in observation mode. Review every finding against the transcript and state labels. Fix invalid fixtures before changing a threshold. If reviewers disagree about whether an intent was replaced or added, record that ambiguity and keep the case out of the release block. Disagreement is dataset evidence, not an automatic product failure.

When the stable set is clean, gate only new deterministic findings. A baseline file can make that rollout mechanical, but it carries a clear cost: existing defects remain allowed until someone burns down the baseline. Store issue identity, not an aggregate score, so repairing one case removes one explicit exception.

# gate_new_relevance_findings.py
from __future__ import annotations

import json
from pathlib import Path
import sys


def finding_keys(path: Path) -> set[tuple[str, str, str]]:
    records = json.loads(path.read_text(encoding="utf-8"))
    return {
        (record["case_id"], issue["code"], issue["evidence"])
        for record in records
        for issue in record["issues"]
    }


if len(sys.argv) != 3:
    raise SystemExit(
        "usage: gate_new_relevance_findings.py BASELINE CURRENT"
    )

known = finding_keys(Path(sys.argv[1]))
current = finding_keys(Path(sys.argv[2]))
introduced = sorted(current - known)

for case_id, code, evidence in introduced:
    print(f"NEW {case_id}: {code} ({evidence})")

raise SystemExit(1 if introduced else 0)

Baseline entries should have owners and removal work. Otherwise the file becomes a permanent pardon for known defects. Do not baseline an oracle failure, missing transcript, schema error, or ambiguous case as if it were a product exception. Those conditions make the result unfit for a release decision.

Expand coverage by transition type rather than transcript length. Add corrections, persistent constraints, resolved intents, reopened intents, parallel goals, and explicit topic switches. A twelve-turn chat with two state changes can be more valuable than a hundred-turn chat that repeats one question. Length is a stress dimension, not the oracle.

Vary where transitions occur. Systems often behave differently when a correction appears near the beginning, immediately before a decision, or after unrelated small talk. Keep the expected state identical while moving the transition in separate fixtures. That isolates position sensitivity without changing the business rule.

The costs are real:

• More state labels improve diagnosis but increase authoring and review time.
• Exact evidence checks are stable but miss acceptable paraphrases.
• Semantic review covers paraphrase but introduces calibration work and uncertain outcomes.
• Full transcript replay raises latency and may consume paid model capacity.
• Storing detailed evidence speeds debugging but increases privacy and retention obligations.
• A strict release gate catches regressions early but can slow harmless prompt experimentation.

Use separate lanes to manage those costs. Run deterministic incident cases on every relevant change. Run larger semantic sets on a schedule or before a release. Send disputed cases to review. That structure gives fast feedback without pretending that the cheapest lane covers every conversational behavior.

Remove a case only when its user path no longer exists, and document the replacement when behavior changes. If a prompt update intentionally changes how parallel requests are handled, update the state contract in review before accepting new responses. Editing expected answers until a build passes destroys the history that made the case useful.

Know when not to use this gate

Skip conversation-state scoring for a truly stateless, single-turn feature. If every request is contractually independent, a multi-turn oracle adds annotation without testing a supported behavior. Test the request and response contract directly.

Do not use relevance as a substitute for safety, authorization, factuality, or tool correctness. A response can be perfectly relevant while exposing private data, acting without permission, citing a stale policy, or sending the wrong tool arguments. Those risks need their own assertions and release rules.

Avoid exact phrase markers when acceptable answers span many languages or paraphrases and no stable entity is available. Literal checks will measure wording instead of intent. Use reviewed semantic labels, domain-specific structured fields, or human assessment for that slice, and report its uncertainty openly.

Do not block a release on a fresh semantic rubric that reviewers have not applied consistently. First run it on clear positive cases, clear negative cases, and boundary cases. If the reasons do not tell reviewers why the response served or abandoned the active goal, improve the rubric before raising its authority.

Leave exploratory conversations out of the blocking lane when product behavior is intentionally unsettled. They can still run as research cases. Label them non-blocking, retain their evidence, and promote them only after the team agrees on the supported state transitions.

Finally, do not turn this suite into a leaderboard for broad claims about one assistant being better than another. The cases reflect chosen workflows, annotations, and risks. Use them to protect those workflows. If the team needs a comparative study, design sampling, review, and reporting for that question instead of recycling a regression gate.