PRACTICAL GUIDE / agent context eviction priority testing

When an agent forgets the one instruction that mattered

Learn how to prove an agent keeps safety and user constraints under context pressure, diagnose wrong evictions, and add reliable checks to CI.

By The Testing AcademyUpdated August 4, 202622 min read
All field guides
In this guide6 sections
  1. Why the wrong memory survives
  2. Build an oracle before testing the model
  3. Work through three different failures
  4. Prove whether eviction is really the culprit
  5. Move the policy into CI without breaking the suite
  6. Accept the cost, and know when not to evict

What you will learn

  • Why the wrong memory survives
  • Build an oracle before testing the model
  • Work through three different failures
  • Prove whether eviction is really the culprit

The agent follows a refund limit for most of a conversation, then ignores it as soon as a long tool result arrives. Nothing crashed. The instruction was removed from the prompt while a stale observation survived.

That failure is easy to mislabel as model inconsistency. It is often a plain ordering bug in the code that chooses what fits into the next prompt. The useful test is not whether the final answer sounds reasonable. It is whether the memory policy removed the entries its own contract said were disposable.

Why the wrong memory survives

Most agent stacks have several decisions between stored conversation data and the prompt sent for the next turn. A retriever chooses possible entries. An eligibility filter removes expired, private, or irrelevant records. An eviction policy handles the remaining set when its combined cost is too large. A prompt assembler places retained entries into a final structure. Testing only the final answer collapses those boundaries and makes every defect look like a model defect.

Start by naming the contract your application owns. In the examples below, every context entry has five policy fields:

  • id identifies the exact entry across logs and fixtures.
  • cost_units is an integer supplied by the production accounting adapter.
  • priority ranks removable entries, with a lower value removed first.
  • pinned is a hard protection flag, not another point on the ranking scale.
  • last_used_order breaks priority ties, with an older value removed first.

Those meanings are choices made for this test policy. They are not universal agent framework behavior. Another product can legitimately use the opposite priority direction, an expiry timestamp, or a semantic-value score. The test must encode the product's documented rule instead of assuming that the word priority has a standard interpretation.

The separation between priority and pinning matters. If a safety instruction merely receives priority 100, a future feature can introduce priority 200 and push the supposedly protected instruction down the list. A pin is a different rule: the entry is ineligible for eviction. Code that turns user text, retrieved web content, or model output directly into a pinned entry creates a privilege problem, so only a trusted part of the application should set that field. The unit test can prove how the flag is handled; a separate authorization test must prove who can set it.

Cost needs the same precision. This article uses cost_units because the policy does not know how a production prompt is measured. A fixture value of 8 means eight units in this defined test, not eight tokens reported by a provider. The adapter that calculates production cost deserves its own contract tests. Conflating estimated text length with the policy decision can produce a green eviction suite and an oversized final prompt.

A reliable oracle checks more than the first ID in an eviction list. For every successful plan, the retained and evicted ID sets must be disjoint. Their union must equal the input IDs. No pinned ID may be evicted. Retained cost must be at or below the budget. Repeating the same logical input in another list order must produce the same eviction decision when input order is not part of the policy. Duplicate IDs, negative budgets, and invalid costs must be rejected before ranking begins.

One more result is necessary: blocked. If pinned entries alone exceed the available budget, no valid eviction plan exists under this contract. Returning the remaining over-budget entries with a success flag hides the conflict. Dropping a pin violates the contract. An explicit blocked result forces the caller to choose a visible fallback.

This layer should be deterministic. It does not need a language model, network call, or semantic grader. That is an advantage, not a simplification that weakens the test. The policy can be exercised across every boundary value without waiting for an external model, and a failure points to code your team controls. Model-level scenarios still belong above it, where they verify that a retained instruction is actually used.

Build an oracle before testing the model

A pure function makes the decision inspectable. It receives complete entries and a budget, validates the fixture, ranks only eligible entries, and returns a plan without editing the input collection. Save this listing as eviction_policy.py if you want to run the later examples unchanged.

Python
from dataclasses import dataclass
from typing import Literal, Sequence


@dataclass(frozen=True)
class ContextEntry:
    id: str
    priority: int
    pinned: bool
    cost_units: int
    last_used_order: int


@dataclass(frozen=True)
class EvictionPlan:
    status: Literal["fits", "blocked"]
    retained: tuple[ContextEntry, ...]
    evicted_ids: tuple[str, ...]
    used_cost_units: int
    reason: str | None = None


def plan_eviction(
    entries: Sequence[ContextEntry], budget: int
) -> EvictionPlan:
    if isinstance(budget, bool) or not isinstance(budget, int) or budget < 0:
        raise ValueError("budget must be a non-negative integer")

    ids = [entry.id for entry in entries]
    if any(not entry_id for entry_id in ids):
        raise ValueError("every context entry needs a non-empty id")
    if len(ids) != len(set(ids)):
        raise ValueError("context entry ids must be unique")
    if any(
        isinstance(entry.cost_units, bool)
        or not isinstance(entry.cost_units, int)
        or entry.cost_units <= 0
        for entry in entries
    ):
        raise ValueError("cost_units must be positive integers")

    original = tuple(entries)
    original_cost = sum(entry.cost_units for entry in original)
    if original_cost <= budget:
        return EvictionPlan("fits", original, (), original_cost)

    candidates = sorted(
        (entry for entry in original if not entry.pinned),
        key=lambda entry: (
            entry.priority,
            entry.last_used_order,
            entry.id,
        ),
    )

    selected_ids: list[str] = []
    remaining_cost = original_cost
    for candidate in candidates:
        if remaining_cost <= budget:
            break
        selected_ids.append(candidate.id)
        remaining_cost -= candidate.cost_units

    if remaining_cost > budget:
        return EvictionPlan(
            status="blocked",
            retained=original,
            evicted_ids=(),
            used_cost_units=original_cost,
            reason="pinned context exceeds the available budget",
        )

    selected = set(selected_ids)
    retained = tuple(entry for entry in original if entry.id not in selected)
    return EvictionPlan(
        status="fits",
        retained=retained,
        evicted_ids=tuple(selected_ids),
        used_cost_units=remaining_cost,
    )

Notice the atomic behavior on overflow. The function may examine removable candidates while planning, but a blocked result returns the original entries and an empty eviction list. A caller cannot accidentally apply a partial decision and then continue with an invalid prompt. That choice costs some convenience because the caller has to handle another status, but it keeps the protection rule honest.

The sort key is deliberately complete. Priority resolves the policy's main preference. last_used_order resolves equal priorities. The ID resolves a remaining tie. Depending on the incoming list position would make database order, concurrent retrieval completion, or fixture construction part of the result even though none is in the stated policy.

Diagnostics need the same inputs the oracle used. Logging only evicted_ids tells you the outcome but not why it happened. Save the following executable script as tools/diagnose_context_eviction.py. It records candidate order, cost before and after planning, retained IDs, and the blocked reason. It contains IDs and policy metadata, not context text, which reduces the chance of copying conversation secrets into a CI artifact.

Python
import json

from eviction_policy import ContextEntry, plan_eviction


def diagnostic_record(case_id: str, entries: list[ContextEntry], budget: int) -> dict:
    plan = plan_eviction(entries, budget)
    candidates = sorted(
        (entry for entry in entries if not entry.pinned),
        key=lambda entry: (entry.priority, entry.last_used_order, entry.id),
    )
    return {
        "case_id": case_id,
        "budget": budget,
        "input_cost_units": sum(entry.cost_units for entry in entries),
        "candidate_order": [entry.id for entry in candidates],
        "evicted_ids": list(plan.evicted_ids),
        "retained_ids": [entry.id for entry in plan.retained],
        "retained_cost_units": plan.used_cost_units,
        "status": plan.status,
        "reason": plan.reason,
    }


if __name__ == "__main__":
    fixture = [
        ContextEntry("safety-policy", 1, True, 4, 1),
        ContextEntry("refund-limit", 90, True, 3, 2),
        ContextEntry("tool-result-17", 60, False, 8, 3),
        ContextEntry("greeting", 5, False, 2, 0),
    ]
    print(
        json.dumps(
            diagnostic_record("refund-after-tool-result", fixture, budget=9),
            indent=2,
            sort_keys=True,
        )
    )

Running that fixture reports an input cost of 17 and a retained cost of 7. Candidate order is greeting, then tool-result-17; both are evicted. The two pinned entries remain. These are calculated fixture results, not measurements from a deployed agent.

The choice also exposes a genuine trade-off in this policy. It removes the low-priority greeting before the larger tool result, even though removing the tool result alone would satisfy the budget. The objective is strict priority order, not the fewest removals. If your product values conversational continuity more than strict rank order, that is a different optimization problem and needs a different oracle.

Pytest's assertion documentation explains why ordinary Python assert statements produce useful value comparisons in discovered tests. That is enough for this policy. A custom semantic grader would add uncertainty without answering a semantic question.

Work through three different failures

The first failure is a protected instruction represented only by a large number. Imagine a policy migration that changes the range from 0-100 to 0-1000. The refund limit remains at 90, while a fresh tool result receives 500. With a budget that admits the tool result but not both entries, a priority-only implementation evicts the limit first. The numbers are valid, the sort succeeds, and the final answer may still look plausible.

The fix is not to pick an even larger magic number. Mark the refund limit as pinned and prove that pinning overrides rank. Give it a deliberately low priority in the fixture so the test fails if an engineer later removes the pin check. This is fault injection with a precise purpose: it makes the hard constraint and soft preference disagree.

Useful failure evidence names the contract, not just the response defect. A good pytest comparison for this case shows safety-policy or refund-limit under extra items in the evicted set and tool-result-17 under missing items. Seeing the protected ID in the policy output localizes the issue before anyone spends time tuning prompts.

The second failure appears only when entries share a priority. A common implementation sorts on priority and then trusts whatever order it received. The test passes while a fixture loader returns a fixed list. It fails in a full run when retrieval finishes in another order. Retrying often makes it green, which is exactly why retrying is the wrong response.

Use explicit, persistent tie-break data. In this contract, the least recently used order comes next, followed by ID. The ID is not meant to represent semantic value; it merely prevents an unresolved tie. If recency is generated from wall-clock timestamps during the test, equal or near-equal records can still move around. Fixed integer sequence values make the case replayable.

The third failure is an impossible budget. Two pinned entries cost more than the allowed total before optional memory is considered. A naive loop evicts every optional record, sees there are no candidates left, and returns the oversized remainder as though planning succeeded. Another bad implementation removes the lowest-ranked pinned record and quietly weakens a rule the rest of the system believes is mandatory.

Treat that condition as a first-class outcome. The application may choose to reject the turn, request a larger budget, shorten a trusted instruction through a separately reviewed process, or omit a feature-specific block. The eviction function should not make that product decision. Its job is to state that the constraints cannot all be satisfied.

A fourth failure lives one layer up, at the boundary where stored records become ContextEntry values. plan_eviction trusts the pinned flag completely, so whoever sets that flag decides what the budget can never remove. If a tool result or a retrieved document can carry pinned: true straight into the builder, untrusted content has bought itself permanent residence in the prompt and can even push the whole turn into a blocked plan. Keep that translation in one small function so it can be tested on its own. Save it as context_assembly.py.

Python
from typing import Any, Literal

from eviction_policy import ContextEntry

RecordSource = Literal[
    "policy_store", "operator_console", "retrieval", "tool_output", "user"
]

TRUSTED_SOURCES: frozenset[str] = frozenset({"policy_store", "operator_console"})


def build_context_entry(
    record: dict[str, Any], source: RecordSource
) -> ContextEntry:
    return ContextEntry(
        id=record["id"],
        priority=int(record["priority"]),
        pinned=bool(record.get("pinned", False)) and source in TRUSTED_SOURCES,
        cost_units=int(record["cost_units"]),
        last_used_order=int(record["last_used_order"]),
    )

The builder honors a requested pin only when the record arrived from a trusted source. That single and is the entire privilege boundary, which is why it deserves a case that runs the built entries back through plan_eviction rather than an assertion about a hand-written fixture.

These four cases fit into one compact test module. Save it as tests/test_eviction_policy.py. The permutation test evaluates every ordering of its three-entry fixture. It is an exhaustive check for that fixture, not a claim about all possible context sets.

Python
from itertools import permutations

from context_assembly import build_context_entry
from eviction_policy import ContextEntry, plan_eviction


def test_pin_overrides_a_low_priority() -> None:
    entries = [
        ContextEntry("safety-policy", 0, True, 4, 1),
        ContextEntry("tool-result-17", 900, False, 8, 2),
        ContextEntry("greeting", 1, False, 2, 0),
    ]

    plan = plan_eviction(entries, budget=6)

    assert plan.status == "fits"
    assert plan.evicted_ids == ("greeting", "tool-result-17")
    assert {entry.id for entry in plan.retained} == {"safety-policy"}


def test_untrusted_records_cannot_request_protection() -> None:
    trusted = build_context_entry(
        {
            "id": "safety-policy",
            "priority": 0,
            "pinned": True,
            "cost_units": 4,
            "last_used_order": 1,
        },
        source="policy_store",
    )
    injected = build_context_entry(
        {
            "id": "tool-result-17",
            "priority": 900,
            "pinned": True,
            "cost_units": 8,
            "last_used_order": 2,
        },
        source="tool_output",
    )

    assert trusted.pinned is True
    assert injected.pinned is False

    plan = plan_eviction([trusted, injected], budget=6)

    assert plan.status == "fits"
    assert plan.evicted_ids == ("tool-result-17",)
    assert {entry.id for entry in plan.retained} == {"safety-policy"}


def test_equal_priority_decision_ignores_input_order() -> None:
    tied = [
        ContextEntry("memory-c", 10, False, 3, 8),
        ContextEntry("memory-a", 10, False, 3, 8),
        ContextEntry("memory-b", 10, False, 3, 8),
    ]

    decisions = {
        plan_eviction(list(order), budget=3).evicted_ids
        for order in permutations(tied)
    }

    assert decisions == {("memory-a", "memory-b")}


def test_oversized_pinned_set_blocks_atomically() -> None:
    entries = [
        ContextEntry("safety-policy", 100, True, 6, 1),
        ContextEntry("user-constraint", 100, True, 6, 2),
        ContextEntry("stale-note", 0, False, 1, 0),
    ]

    plan = plan_eviction(entries, budget=10)

    assert plan.status == "blocked"
    assert plan.evicted_ids == ()
    assert plan.retained == tuple(entries)
    assert plan.used_cost_units == 13
    assert plan.reason == "pinned context exceeds the available budget"


def test_successful_plan_preserves_partition_invariants() -> None:
    entries = [
        ContextEntry("pinned", 0, True, 2, 0),
        ContextEntry("old", 1, False, 4, 1),
        ContextEntry("new", 2, False, 4, 2),
    ]

    plan = plan_eviction(entries, budget=6)
    input_ids = {entry.id for entry in entries}
    retained_ids = {entry.id for entry in plan.retained}
    evicted_ids = set(plan.evicted_ids)

    assert plan.status == "fits"
    assert retained_ids.isdisjoint(evicted_ids)
    assert retained_ids | evicted_ids == input_ids
    assert plan.used_cost_units <= 6

The second test deserves attention because it is the only one that does not start from a hand-written fixture. Asserting that a literal list of entries has the pin states you just typed proves nothing about the code under test. Building the entries through build_context_entry and then planning against them does: a builder that copies the requested pinned flag from tool output makes both entries protected, their combined cost of 12 exceeds the budget of 6, and the plan comes back blocked with an empty eviction list instead of the expected ("tool-result-17",). Trusted policy data becomes pinned, untrusted content cannot request the same status, and the eviction policy is exercised on entries the production path actually produces.

Boundary coverage should also include a total exactly equal to the budget, a zero budget with only removable entries, one entry whose cost is larger than the budget, duplicate IDs, and an empty collection. Those cases exercise distinct branches. Repeating the same four-record happy path with different names does not.

Prove whether eviction is really the culprit

A missing instruction is not automatically an eviction defect. The same final prompt can result from retrieval, eligibility filtering, policy ranking, prompt assembly, or a later transformation. Capture ID membership at each boundary and identify the first stage where the expected entry disappears.

Suppose refund-limit is absent from the next prompt. If it never appears in retrieved_ids, the retriever did not supply it. Changing eviction priority cannot fix that. If it appears in retrieval but not eligible_ids, inspect expiry, tenant, privacy, and scope filters. If it is eligible and then listed in evicted_ids, the ranking policy owns the result. If the plan retains it but rendered_ids omits it, prompt assembly is the broken boundary.

The final possibility is subtler: the ID appears in the rendered prompt and the response still violates it. At that point, the eviction test has passed for this entry. Investigate prompt structure and answer behavior separately. Do not change a correct memory policy merely because an end-to-end assertion failed above it.

This small classifier turns the boundary evidence into a consistent first diagnosis. The stage names belong to the example's own telemetry schema, so there is no dependency on a vendor trace API.

Python
from dataclasses import dataclass


@dataclass(frozen=True)
class PipelineEvidence:
    expected_id: str
    retrieved_ids: tuple[str, ...]
    eligible_ids: tuple[str, ...]
    retained_ids: tuple[str, ...]
    evicted_ids: tuple[str, ...]
    rendered_ids: tuple[str, ...]


def locate_first_loss(evidence: PipelineEvidence) -> str:
    expected = evidence.expected_id
    retrieved = set(evidence.retrieved_ids)
    eligible = set(evidence.eligible_ids)
    retained = set(evidence.retained_ids)
    evicted = set(evidence.evicted_ids)
    rendered = set(evidence.rendered_ids)

    collections = (
        evidence.retrieved_ids,
        evidence.eligible_ids,
        evidence.retained_ids,
        evidence.evicted_ids,
        evidence.rendered_ids,
    )
    if any(len(values) != len(set(values)) for values in collections):
        return "duplicate-id-evidence"
    if not eligible.issubset(retrieved):
        return "inconsistent-eligibility-evidence"
    if retained & evicted or retained | evicted != eligible:
        return "inconsistent-plan-evidence"
    if not rendered.issubset(retained):
        return "inconsistent-render-evidence"

    if expected not in evidence.retrieved_ids:
        return "retrieval"
    if expected not in evidence.eligible_ids:
        return "eligibility-filter"
    if expected in evidence.evicted_ids:
        return "eviction-policy"
    if expected not in evidence.rendered_ids:
        return "prompt-assembly"
    return "present-in-rendered-prompt"


if __name__ == "__main__":
    assembly_loss = PipelineEvidence(
        expected_id="refund-limit",
        retrieved_ids=("refund-limit", "tool-result-17"),
        eligible_ids=("refund-limit", "tool-result-17"),
        retained_ids=("refund-limit",),
        evicted_ids=("tool-result-17",),
        rendered_ids=(),
    )
    retrieval_loss = PipelineEvidence(
        expected_id="refund-limit",
        retrieved_ids=("tool-result-17",),
        eligible_ids=("tool-result-17",),
        retained_ids=("tool-result-17",),
        evicted_ids=(),
        rendered_ids=("tool-result-17",),
    )

    assert locate_first_loss(assembly_loss) == "prompt-assembly"
    assert locate_first_loss(retrieval_loss) == "retrieval"
    print(locate_first_loss(assembly_loss))

Run the diagnostic against the same case ID in an isolated test and in the full suite. Compare candidate IDs and all three sort fields. If they differ before planning, the eviction function received different inputs. If they match but evicted_ids differ, the function has hidden state or the running code version differs. If the plan matches and rendered membership differs, move the investigation downstream.

Cost-accounting drift is another near-miss. The policy can report 90 units retained while the prompt builder reports 104 under its own exact accounting. Those values are illustrative, not production measurements. The important evidence is that the two components did not measure the same artifact with the same unit. Record the accounting method and input version beside each value. Do not loosen eviction expectations until the discrepancy is explained.

Logs should preserve identifiers, policy version, budget, rank fields, status, and decision. Raw context text is rarely required for this diagnosis and may contain personal data, credentials, or tool output. If content inspection is necessary, keep it in an access-controlled debugging path rather than a routine CI artifact. Redaction must happen before durable storage, not after the log has already reached a shared system.

An error message such as expected refund-limit in retained_ids, found it in evicted_ids is actionable. Agent gave a bad answer is not. The first points to one boundary and one record. The second sends engineers across retrieval, memory, prompting, and model behavior with no starting evidence.

Move the policy into CI without breaking the suite

An existing agent suite usually has end-to-end cases but no seam around prompt selection. Introduce that seam in stages. First, add read-only diagnostics to the current planner: case ID, policy version, input IDs, ranking fields, budget, retained IDs, and evicted IDs. Compare those records with incident traces before changing behavior. This reveals undocumented conventions such as higher numbers meaning lower value or a database query providing an accidental tie-break.

Next, extract the current behavior into a pure function and characterize it with fixtures. Characterization is not approval. Mark each fixture as an accepted contract, a known defect, or an unresolved product decision. Locking every historical quirk into a permanent test suite makes the later correction harder.

Build the first accepted fixtures from three sources: a minimal normal conversation, a boundary case at the budget, and a real incident with content replaced by safe labels. Add the protected-entry conflict and tie permutation even if neither has appeared in production. They attack properties of the algorithm rather than wording of one transcript.

Run the new planner in shadow mode beside the old path. It must not alter the prompt during this phase. Compare decisions by ID and explain each difference. Some mismatches will be test data problems, especially missing pin metadata or costs produced by different adapters. Others will expose the old bug you intended to fix. Keep those categories separate in the rollout report.

After the policy owners approve the contract, switch deterministic unit cases to blocking. Leave model-response comparisons outside this gate unless they have their own stable oracle. A policy regression should fail because an exact entry was wrongly removed, not because a generated answer used different wording.

Version any change to priority direction, tie-break order, protection rules, or cost units. Store the policy version with diagnostics. When an older fixture changes expectation, the review should say whether the product contract changed or a bug was fixed. Silently editing the expected eviction list destroys the evidence that makes regression tests useful.

The runnable CI entry point can stay small. This script assumes the policy, diagnostic script, and test module are checked into the paths shown in the article. It writes one metadata-only artifact for triage and relies on pytest's nonzero failure exit to stop the shell.

Shell
#!/usr/bin/env bash
set -euo pipefail

artifact_dir="artifacts"
mkdir -p "$artifact_dir"

PYTHONPATH=. python -m pytest -q tests/test_eviction_policy.py

PYTHONPATH=. python tools/diagnose_context_eviction.py \
  | tee "$artifact_dir/context-eviction-diagnostic.json"

Pytest documents direct module invocation and path selection, while its parametrization guide supports expanding boundary rows without copying test bodies. Keep case IDs visible in parametrized output. A compact failure named pinned-overflow or equal-priority-permutation is easier to route than an index with no scenario meaning.

Do not gate on a percentage assembled from unrelated cases. A single protected-entry eviction is a different release decision from a duplicate-ID fixture error. Report at least four buckets: contract violations, invalid fixtures, blocked budgets, and diagnostic inconsistencies. The counts describe this test run only. They are not a claim about universal agent reliability.

During the switch, retain an emergency rollback for the new planner, but do not let rollback discard the new diagnostics. If the old behavior returns, you still need to know which entries it removed. Once the corrected path has survived the team's normal release observation period, remove dual execution to recover its extra compute and logging cost.

Accept the cost, and know when not to evict

Deterministic ranking is not free. The example sorts removable entries, which takes time that grows with the number of candidates. It also creates tuples, a candidate list, and a set rather than mutating the caller's data. That extra allocation buys isolation and readable evidence. In a high-volume service with large context stores, measure the implementation under representative workloads before putting full diagnostics on every request.

Pinning consumes budget. A team that labels every instruction important will eventually create an impossible set. Require an owner and reason for protected categories, then test the overflow path. The cost of an explicit rejection is visible user friction. The cost of silently dropping a protection can be a policy violation no dashboard attributes to memory.

Strict priority order has its own coverage cost. The sample planner can evict several small, low-ranked entries before one large, higher-ranked entry. That may remove more conversational detail than a packing algorithm would. A packing approach, however, needs a clearly stated objective such as minimizing removed value while meeting the budget. Without that objective, a clever optimizer is impossible to test because nobody can say which valid subset is correct.

Detailed traces add storage, privacy risk, and triage work. Metadata-only decision records are enough for most ranking failures. Sample richer evidence only where access controls and retention rules are defined. Remember that eviction from a prompt is not deletion from the memory store. If a privacy requirement says data must be erased, test the storage lifecycle separately.

There are several situations where adding eviction logic is the wrong fix:

  • The candidate set already fits the budget. Removing context early reduces coverage without solving pressure.
  • The expected entry never reaches the planner. Retrieval or eligibility owns that defect.
  • The plan retains the entry but the assembler omits it. Fix prompt construction and keep the eviction oracle unchanged.
  • The final prompt contains the instruction, yet the response violates it. That needs an answer-behavior test, not a new priority value.
  • An external component manages context opaquely and exposes neither inputs nor decisions. An exact white-box eviction assertion would pretend to observe data you do not have. Test the visible contract or move selection into code you control.
  • The content must remain available in full for audit or legal reasons. Prompt trimming does not satisfy retention, and storage deletion does not follow from prompt eviction.
  • Summarization is the approved product behavior. Test summary provenance, protected facts, and loss boundaries instead of asserting that individual source entries remain verbatim.

One tempting anti-pattern is to raise the budget until the test passes. A larger budget can be a valid product decision, but it does not repair nondeterministic ranking, weak pin authorization, or a broken assembler. Keep the original failing fixture. If the new budget is intentional, update the versioned contract and add a case that still forces eviction below that new boundary.

Another is to assert only that the safety text appears somewhere in a serialized prompt. Duplicate, stale, or contradictory copies can satisfy that string check. Verify the retained entry ID and the final rendered membership, then give the higher layer a scenario that proves the operative instruction is followed. Those two tests answer different questions and fail for different owners.

Finally, avoid using generated summaries as silent replacements for pinned instructions. Summarization changes content, so it is not eviction under this contract. If shortening a protected entry is necessary, make it a separate, reviewed transformation with its own input, output, and equivalence checks. A blocked budget is useful precisely because it exposes that product choice instead of burying it in a sorting loop.

// FIELD DISPATCH

Get the QA Field Notes

Weekly QA battles, AI testing guides, and interview drills. Free on Substack.

// LIVE COURSE / THE TESTING ACADEMY

AI Tester Blueprint

Master GenAI, AI Agents, MCP, RAG, CrewAI. Build 23+ real AI projects.

From the instructor behind this guide.

AI testing roles are up 180% and pay 12-22 LPA. 12+ weeks / 65+ live hrs / Sat-Sun 8:30 AM IST.

Code PROMODE / 10% offJoin the batch

The Testing Academy editorial desk

Practical QA guidance built around test evidence, production tradeoffs, and interview-ready explanations.

Published July 25, 2026 / Reviewed August 4, 2026

PRIMARY REFERENCES

Verify the details at the source

QABattle guides are practical explanations. Product behavior, standards, and APIs can change, so use these primary references for the canonical details.

  1. 01
    Official docs.pytest.org reference

    docs.pytest.org

    Primary documentation selected and verified for the claims in this guide.

  2. 02
    Official docs.pytest.org reference

    docs.pytest.org

    Primary documentation selected and verified for the claims in this guide.

  3. 03
    Official docs.pytest.org reference

    docs.pytest.org

    Primary documentation selected and verified for the claims in this guide.

  4. 04
    Evaluate complex agents

    LangSmith

    Official guidance for final-response, trajectory, and single-step agent evaluation.

FAQ / QUICK ANSWERS

Questions testers ask

How do I test which agent memory gets dropped first?

Give the eviction function a fixed set of entries whose costs, priorities, pin states, and tie-break fields are explicit. Assert the exact retained and evicted IDs, then repeat the case with the input order changed so an accidental list-order dependency cannot hide.

Should safety instructions rely on a high priority score?

No. A ranking only decides the order among entries that are eligible for removal. Mark non-negotiable instructions with a separate protected state, validate who is allowed to set it, and fail explicitly if protected content alone exceeds the budget.

Why does my eviction test pass alone and fail in the full suite?

Hidden ordering or shared session state is the first place to look. Capture the candidate IDs and every ranking field immediately before the decision, then compare isolated and suite runs rather than retrying a changed fixture.

What should happen when pinned context is larger than the budget?

Return an explicit overflow result without pretending the plan fits. The caller can reject the request, reduce optional context, or choose a reviewed fallback, but the eviction layer must not quietly remove a protected entry.

Can an end-to-end answer prove the eviction policy works?

Usually not, because a good answer can be produced even when the wrong entry was removed. Inspect the policy input, candidate ranking, decision, and final prompt membership first; use answer-level checks as a separate layer.