Your agent benchmark looks too good. Check what leaked.

Your agent passes a private task it has supposedly never seen. The trace looks clean until a reviewer finds the grader's example answer inside a retrieval index mounted for debugging. The score is real, but it measures access to the answer as well as the skill the team meant to test.

Why a benchmark can improve while the agent does not

A benchmark result is only useful when the task tests the capability named in the report. If the agent can read an answer key, recognize a task copied from a worked example, or receive a grader hint between attempts, the run tests a different system. It tests the model plus an unintended information channel.

That distinction matters because "contamination" covers several mechanisms that need different fixes. Direct leakage is the easy case. The exact task, expected answer, or grading rubric is present in the prompt, mounted files, retrieval corpus, tool response, memory, or cache. Derived leakage is subtler. A development example supplies the same answer-specific reasoning while names and formatting change. Operational leakage happens when a grader result, previous trace, or human review note returns to a later attempt. Historical exposure is broader still: a public benchmark may have appeared in training or fine-tuning material that the evaluation team cannot inspect.

A local audit can make strong claims about the first three categories when the evidence exists. It can show which files were mounted, which retrieval records were returned, what state entered the run, and when evaluator output was written. It usually cannot prove the absence of historical model exposure. The honest report says, "No leak was found in the controlled evaluation path," not, "The model has never seen this task."

Scores do not diagnose the mechanism. An unexpectedly high result is a reason to inspect lineage and access. It is not proof by itself. A low result is not proof of cleanliness either. An agent can see a leaked hint and still fail because a tool is broken or the grader expects the wrong final state.

Treat each benchmark result as a relationship among versioned artifacts:

• The task version defines the start state, goal, constraints, and permitted information.
• The execution version identifies the model configuration, prompt, tools, memory policy, and runtime code.
• The reference version contains the facts or outcomes used for grading.
• The evaluator version turns a completed run into a judgment.
• The run record ties those versions to a trace, final state, and outcome.

Change any member and you have changed the evaluated system. A dashboard that stores only "model name, task name, score" cannot tell whether a surprising jump came from the model, the task, the tool bundle, or the grader.

The W3C PROV data model offers a useful vocabulary for this evidence. It describes provenance in terms of entities, activities, and agents, plus relationships such as generation, use, and derivation. You do not need to implement PROV-DM to audit a benchmark. The practical lesson is to record what artifact existed, what process used or created it, and which identity was responsible. A hash without that lineage proves only that two captured byte sequences match or differ.

I treat agent benchmark contamination auditing as a validity investigation, not a plagiarism detector. The central question is whether information outside the task contract could influence the result. Exact overlap checks, access tests, and run-ledger rules each answer one part of that question. None deserves to stand in for all the others.

Build evidence before you run the agent

Start with an inventory that a reviewer can follow without opening the private prompt. Every task needs an immutable ID and a version. Record its split, owner, creation time, source classification, prompt fingerprint, reference ID, evaluator ID, and required tool policy. Keep the reference content in a separate store. The manifest may point to its identifier, but the agent-facing task bundle should not contain its path or body.

Use splits for operational control, not decoration. A development task is available while prompts and tools change. A validation task supports limited tuning decisions. A private holdout is withheld from the normal development path and used under a controlled release policy. Moving a task from private to development is a one-way disclosure event for the people and systems that receive it. Renaming the split later does not make it private again.

Content fingerprints catch exact reuse after a declared normalization rule. The rule matters. Lowercasing, Unicode normalization, and whitespace folding can find copies that differ only cosmetically. Removing all punctuation, numbers, or domain terms can collapse genuinely different tasks and create noise. Store both the raw file hash and the normalized-content hash, along with the normalization algorithm version, so a future reviewer knows what "same" meant.

The following scanner is intentionally narrow. It validates IDs and splits, reads each prompt in a trusted audit job, and reports duplicate normalized content only when matching tasks span different splits, without printing task text. It uses only the Python standard library. The agent process should never receive the manifest access held by this job.

#!/usr/bin/env python3
from __future__ import annotations

import hashlib
import json
import sys
import unicodedata
from collections import defaultdict
from pathlib import Path
from typing import Any

ALLOWED_SPLITS = {"development", "validation", "private"}


def canonicalize(text: str) -> str:
    normalized = unicodedata.normalize("NFKC", text).casefold()
    return " ".join(normalized.split())


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def load_rows(manifest_path: Path) -> list[dict[str, Any]]:
    value = json.loads(manifest_path.read_text(encoding="utf-8"))
    if not isinstance(value, list):
        raise ValueError("manifest root must be a JSON array")
    return value


def audit(manifest_path: Path) -> list[str]:
    errors: list[str] = []
    seen_ids: set[str] = set()
    by_content: dict[str, list[tuple[str, str]]] = defaultdict(list)

    for index, row in enumerate(load_rows(manifest_path)):
        task_id = row.get("task_id")
        split = row.get("split")
        prompt_file = row.get("prompt_file")

        if not isinstance(task_id, str) or not task_id:
            errors.append(f"row {index}: missing task_id")
            continue
        if task_id in seen_ids:
            errors.append(f"task {task_id}: duplicate task_id")
        seen_ids.add(task_id)

        if split not in ALLOWED_SPLITS:
            errors.append(f"task {task_id}: invalid split")
            continue
        if not isinstance(prompt_file, str) or not prompt_file:
            errors.append(f"task {task_id}: missing prompt_file")
            continue

        path = (manifest_path.parent / prompt_file).resolve()
        text = path.read_text(encoding="utf-8")
        content_hash = sha256_text(canonicalize(text))
        by_content[content_hash].append((task_id, split))

    for members in by_content.values():
        splits = {split for _, split in members}
        if len(members) > 1 and len(splits) > 1:
            task_ids = ",".join(sorted(task_id for task_id, _ in members))
            split_names = ",".join(sorted(splits))
            errors.append(
                "duplicate normalized task content "
                f"tasks={task_ids} splits={split_names}"
            )

    return sorted(errors)


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: audit_manifest.py MANIFEST.json", file=sys.stderr)
        return 2

    errors = audit(Path(sys.argv[1]))
    for error in errors:
        print(f"ERROR {error}", file=sys.stderr)
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

The loader assumes a trusted manifest; because prompt_file is resolved without a containment check, reject paths outside the approved benchmark root before reading manifests from less trusted authors.

For a deliberately bad fixture that places the same normalized prompt in development and private splits, the useful diagnostic is the identity pair, not the secret text:

$ python tools/audit_manifest.py benchmark/manifest.json
ERROR duplicate normalized task content tasks=dev-password-reset,holdout-042 splits=development,private

Duplicates confined to one split are not reported by this scanner. Add a separate within-split check if accidental copies or repeated task weighting could distort the benchmark.

An exact match is a hard failure when your policy says split content must be unique. A near match is different. Token overlap or an embedding score can nominate two tasks for review, but a threshold cannot decide whether they test the same reasoning. Boilerplate such as "Use the available tools" may be harmless. A paraphrased sequence containing the distinctive answer and its required order may be disqualifying. Record the reviewer, decision, rationale, and scanner version instead of silently clearing the alert.

Now draw the access boundary. List every channel visible to the execution identity: system instructions, task payload, environment variables, mounted directories, retrieval collections, browser state, memory, tool credentials, cached responses, and prior messages. List grader-only artifacts separately. For each channel, name the service or process that enforces the rule. "The prompt tells the agent not to read answers" is not access control.

Use fake canaries to test those boundaries. Put a harmless, unique marker in a grader-only fixture. Attempt to retrieve it through every agent-facing tool. A denied call proves the tested route enforced its rule for that run. Finding the marker in agent-visible state proves a leak. Not finding it does not prove that every unknown route is clean, so preserve the tested tool list and policy version.

Never use a production password, customer record, or real secret as a canary. A canary exists to be safe if a test exposes it. Give it an opaque task association so the marker itself does not reveal an answer.

Work through failures that need different fixes

A copied holdout survives because its ID changed

A team promotes a support workflow from development into a private set. Someone assigns a new ID, changes the fictional customer name, and believes the holdout is fresh. The agent now performs unusually well on the private task.

The first scanner finds no duplicate ID. A raw file hash also differs because the name changed. That does not settle the question. Reviewers compare the two tasks and find the same unusual account state, the same three decisions, and the same exception at the end. The private item tests recall of a development path more than transfer to a fresh case.

The fix is to retire that holdout from claims about unseen performance. Keep it for regression if it still has value, but label its exposure. Write a new task from the capability contract, not by paraphrasing the leaked item. Change the facts that drive decisions, the required evidence, and the correct end state. Then have a reviewer who did not author either task compare difficulty and answer cues.

This costs task-authoring time and consumes scarce subject-matter review. It also breaks historical comparability because the replacement is not the same test. Preserve the old series under its old benchmark version rather than splicing the new score into the same chart.

The evidence that separates this case from direct answer leakage is where the overlap appears. No forbidden file read or canary exists in the runtime trace. Instead, provenance shows that the private task derives from a development artifact. That is enough to reject the "unseen holdout" claim even though it says nothing about the model provider's training data.

A file tool can walk from the agent bundle into grader storage

Another suite creates two directories on the same worker: one for the agent and one for the grader. The task loader passes only the agent directory, but the file tool accepts arbitrary paths. A relative path reaches the sibling grader folder.

Fix the production boundary in the file service or infrastructure policy. Then add a regression test at the adapter layer. The example below shows the behavior of a small local reader, including resolution before the root check. It is runnable as a pytest test, but it is not a substitute for operating-system or service-level authorization in a real deployment.

import json
from pathlib import Path

import pytest


class VisibleFileReader:
    def __init__(self, allowed_roots: list[Path]) -> None:
        self.allowed_roots = tuple(
            root.resolve(strict=True) for root in allowed_roots
        )

    def read_text(self, requested: Path) -> str:
        target = requested.resolve(strict=True)
        allowed = any(
            target == root or root in target.parents
            for root in self.allowed_roots
        )
        if not allowed:
            raise PermissionError("path is outside agent roots")
        return target.read_text(encoding="utf-8")


def test_grader_reference_is_not_agent_visible(tmp_path: Path) -> None:
    agent_root = tmp_path / "agent"
    grader_root = tmp_path / "grader"
    agent_root.mkdir()
    grader_root.mkdir()

    canary = "CANARY_FOR_HOLDOUT_042"
    (agent_root / "task.txt").write_text(
        "Resolve the account state.", encoding="utf-8"
    )
    reference = grader_root / "reference.txt"
    reference.write_text(canary, encoding="utf-8")

    reader = VisibleFileReader([agent_root])
    trace: list[dict[str, str]] = []

    with pytest.raises(
        PermissionError, match="path is outside agent roots"
    ):
        reader.read_text(reference)

    trace.append(
        {"event": "tool_error", "message": "path is outside agent roots"}
    )
    assert canary not in json.dumps(trace)

The official pytest documentation says the tmp_path fixture supplies a temporary pathlib.Path directory unique to each test function. That makes it useful for a regression fixture because the allowed and forbidden trees do not depend on a developer's machine. The pass signal should remain boring:

$ python -m pytest tests/test_benchmark_access.py -q
.                                                                        [100%]
1 passed

A green unit test proves only this reader behaved as expected. Run a second test against the deployed tool adapter with the real execution identity. Check the authorization event and the agent-visible trace. Do not include the canary value in an assertion message, because failure output may be retained more broadly than the protected artifact.

Pytest captures stdout and stderr during tests by default and usually displays captured output for a failure. That is convenient for ordinary debugging, but it can publish private prompts if test code prints them. Assert on opaque IDs and safe event codes. Configure protected artifact handling separately when a reviewer truly needs the body.

The dashboard keeps the best retry and hides the first failure

A third team sees the same suspicious shape, a sudden jump in the reported score, but finds no leaked task or answer. The run table reveals that the reporting job replaced an earlier failure with a later pass under the same task ID. The benchmark did not become contaminated. Its aggregation rule changed.

Retries are samples when the agent actually starts the task. Preserve them. A runner crash before task delivery may be invalid if a predeclared policy says so. An agent that loops until the time limit is a valid failure when termination is part of the evaluated behavior. Decide that distinction before reading results.

This ledger code refuses to accept an invalid attempt without a reason and reports all valid outcomes. The fixture values are illustrative test data, not measurements from an experiment.

from dataclasses import dataclass
from typing import Literal

Status = Literal["pass", "fail", "invalid"]


@dataclass(frozen=True)
class Attempt:
    run_id: str
    task_id: str
    status: Status
    invalid_reason: str | None = None


def summarize(attempts: list[Attempt]) -> dict[str, int]:
    for attempt in attempts:
        if attempt.status == "invalid" and not attempt.invalid_reason:
            raise ValueError(
                f"run {attempt.run_id}: invalid result needs a reason"
            )
        if attempt.status != "invalid" and attempt.invalid_reason is not None:
            raise ValueError(
                f"run {attempt.run_id}: valid result cannot have "
                "an invalid reason"
            )

    valid = [item for item in attempts if item.status != "invalid"]
    return {
        "total_attempts": len(attempts),
        "valid_attempts": len(valid),
        "passes": sum(item.status == "pass" for item in valid),
        "failures": sum(item.status == "fail" for item in valid),
        "invalid_attempts": sum(
            item.status == "invalid" for item in attempts
        ),
    }


def test_summary_keeps_failed_retries() -> None:
    attempts = [
        Attempt("run-101", "holdout-042", "fail"),
        Attempt("run-102", "holdout-042", "pass"),
        Attempt(
            "run-103",
            "holdout-042",
            "invalid",
            invalid_reason="task was not delivered",
        ),
    ]

    assert summarize(attempts) == {
        "total_attempts": 3,
        "valid_attempts": 2,
        "passes": 1,
        "failures": 1,
        "invalid_attempts": 1,
    }

The cost is a less flattering dashboard and a migration problem. Historical rows that kept only the best attempt cannot be reconstructed unless raw attempts still exist. Mark that period as non-comparable. Do not manufacture missing failures from an assumed retry rate.

This near-miss matters because its visible symptom matches leakage: performance rises without a model change. The decisive evidence lives in the run ledger, not the task corpus. A contamination-only investigation would waste time scanning prompts while the reporting query continues to discard failures.

Tell contamination from failures that imitate it

Start from one run ID, not the aggregate chart. Reconstruct the ordered inputs before the agent's first action. Include the resolved system prompt, task version, tool descriptors, mounted resource identifiers, restored memory, retrieval results, and cached messages. Then mark the point where execution ends and grading begins. A reference seen after that boundary may be acceptable for an isolated grader. The same reference present before the boundary invalidates the run.

Trace order answers questions that content search alone cannot. Suppose a canary appears in an evaluator event. If the event was created after the agent state was sealed and the evaluator had no route back into execution, the canary did not help that attempt. If the next attempt restores the whole previous conversation, including that evaluator event, the leak affects the retry. The fix is state separation between attempts, not a new task fingerprint.

Compare version manifests before comparing scores. A changed evaluator can turn identical traces into different outcomes. Re-grade preserved traces with both evaluator versions when policy and data handling allow it. If the classifications change while agent actions remain fixed, you have grader drift. That finding does not excuse a bad evaluator, but it is not benchmark exposure.

Task difficulty can also imitate contamination. A public set may require unfamiliar tools while a private set contains shorter or more forgiving cases. A public-versus-private gap means little until reviewers compare skills, constraints, available evidence, and grading strictness. Paired tasks can help, but a "twin" made by swapping names preserves answer cues. A useful pair requires the same capability through different facts and decisions.

Infrastructure creates another false lead. If private runs use a faster service, a larger context budget, or a tool credential with broader data, they test a different setup. That broader credential may itself be a leak if it exposes answers. It may instead provide legitimate domain data that makes the task easier. The task contract must say what the agent is allowed to know.

Use these evidence patterns to sort the cases:

• Exact private content appears in a development artifact created earlier. The holdout lineage is compromised, even if the runtime access log is clean.
• A grader-only canary enters agent-visible state before completion. The configured execution path has a confirmed leak.
• The canary appears only in isolated post-run grader state. Check attempt reset and write boundaries before calling it exposure.
• Identical traces receive different labels from two evaluator versions. Investigate grader drift.
• Every raw attempt is unchanged, but the dashboard result changes. Investigate aggregation and exclusion rules.
• No local channel shows overlap, yet a public task scores well. Report the local audit as clean and historical exposure as unknown.

Semantic similarity deserves special restraint. A scanner should show reviewers why it paired two records, such as shared distinctive phrases or matching required decisions. Do not claim that a decimal threshold proves contamination. Thresholds depend on the corpus, representation, and review goal. Tune them on labeled examples from your own benchmark, publish the rule version, and keep human decisions available for later audit.

Failure output needs the same discipline. Pytest supports parametrizing one test over multiple input rows, which is useful for running an integrity rule across task records. Give each case an opaque task ID so the failure identifies the record without printing its prompt. Pytest's values are passed to parametrized tests as supplied, so avoid mutating shared dictionaries in one case and accidentally changing the next case.

When evidence remains ambiguous, use "needs review" as a real state. Do not force it into pass to make a release report green. Do not force it into contamination to make the audit sound decisive. Assign an owner, preserve the artifacts, and state which missing fact would resolve it.

Roll the controls into CI without publishing the holdout

Introduce the controls in layers. A big-bang gate tends to reveal years of messy lineage at once, then gets disabled because nobody can ship. Begin with an inventory-only run. Backfill stable task IDs, split ownership, content fingerprints, and evaluator versions. Report missing fields without blocking while owners fix them.

Next, block deterministic integrity failures on benchmark changes. Duplicate IDs, exact cross-split content, absent task versions, references packaged in the agent bundle, and a successful forbidden canary read are good gate candidates. Their remedies are clear. Put fuzzy overlap, unexpected score movement, and alternate valid trajectories into a review queue because they need judgment.

Keep the trusted scanner separate from the agent runner. The scanner may need read access to every split to compare fingerprints. The agent runner should receive only the selected task payload and permitted resources. The grader should receive the completed trace and reference after execution. Separate service identities are stronger than relying on directory naming inside one broadly privileged process.

A small shell entry point gives developers and CI the same deterministic gate:

#!/usr/bin/env bash
set -euo pipefail

python tools/audit_manifest.py benchmark/manifest.json

python -m pytest   tests/test_benchmark_access.py   tests/test_run_ledger.py   -q   --show-capture=no

The --show-capture option accepts no, stdout, stderr, log, or all and controls which captured stdout, stderr, and log sections appear in pytest's failure report. It does not sanitize tracebacks, assertion values, external job logs, or artifacts. The safer design is still to keep private bodies out of test parameters and messages. The flag is a final reduction in accidental output, not a secrecy boundary.

Do not run the expensive agent suite when the manifest gate has already failed. Once deterministic checks pass, trigger repeated execution only for changes that can alter behavior: model configuration, system prompt, tool implementation, memory policy, task version, evaluator, or orchestration. A spelling change in an unrelated page does not consume a private holdout.

Private runs need a narrow output contract. Ordinary logs can contain run ID, task ID, component versions, timestamps, status, safe rule codes, and protected artifact locations. Keep prompts, reference answers, raw tool payloads, and canaries in access-controlled evidence storage. Set retention according to the sensitivity and investigation needs of the benchmark. Logging everything forever is not the same as auditability.

Rollout against an existing suite works best in four passes:

1. Freeze the current benchmark version and export the raw attempt ledger before changing aggregation.
2. Add deterministic manifest checks in report-only mode, then assign every failure to an owner.
3. Fix access boundaries and run canary tests against the deployed adapters, not only local doubles.
4. Establish a new comparable baseline after task, grader, and reporting migrations are complete.

The first clean result after migration starts a new series. Keep the old series visible with its limitations. If earlier tasks were exposed in logs, do not call them private in later reports. They can remain valuable regression tasks, just not evidence of performance on unseen work.

A release report should show more than one percentage. Include the benchmark version, valid and invalid attempt counts, the exclusion rule, outcome distribution, unresolved overlap reviews, integrity gate status, and known exposure limits. Link to protected traces by opaque ID. A reviewer should be able to explain why the measurement is trustworthy without receiving the holdout body in a slide deck.

Parameterize deterministic tests when many records share one rule. The official pytest documentation supports parameterizing test functions and fixtures across multiple argument sets. Keep integration runs separate when they carry distinct credentials or expensive setup. Combining every security boundary into one giant parameterized test makes a failure harder to isolate and can widen the credentials available to the process.

Know the cost, and when to stop auditing

Strong isolation spends engineering time. Separate identities, stores, and jobs add configuration and operational support. A team must rotate credentials, review permissions, and debug failures across boundaries. If the grader and runner once shared a filesystem, separating them may expose hidden coupling that takes several releases to remove.

Private tasks are also consumable assets. Every time a prompt is pasted into a ticket, chat, CI log, or model-assisted debugging session, its exposure grows. Restricting access slows investigation because fewer people can see the evidence. The practical compromise is an opaque failure report, a protected review path, and an explicit process for retiring disclosed tasks.

Repeated runs cost runtime and money. They also lengthen release feedback. Use deterministic integrity checks on every relevant change, then size repeated execution around the decision and observed variability. There is no universal repetition count that makes every agent benchmark reliable. Declare the policy before the run and show all valid outcomes.

Similarity review carries a different cost. A sensitive threshold finds more candidates and consumes reviewer time. A strict threshold misses indirect copies. Automation can rank work, but it cannot decide whether two tasks require the same answer-specific insight. Track reviewer agreement and overturned decisions before investing in a more elaborate detector.

Outcome grading may accept several legitimate paths, while safety rules can require exact ordering. Do not use a golden trajectory as a contamination control. An agent can take a new valid route on a clean task. Conversely, a familiar route can still violate approval or authorization. Grade the final outcome and contractual safety events, then retain trajectory differences as diagnostic evidence.

Skip the full private benchmark when the change cannot affect the evaluated system. Documentation-only edits, formatting, or an unrelated service migration usually need the fast integrity gate at most. Save the holdout budget for changes to models, prompts, tools, memory, tasks, graders, or execution logic.

Do not use this audit to make claims about a provider's private training corpus unless you have evidence from that corpus and authority to assess it. Runtime traces and repository history support claims about your controlled path. They do not reveal every source used before the model reached you.

Avoid canary tests when the only available marker would be sensitive data. Create a synthetic marker instead. Do not deliberately leak a private task to see whether the agent repeats it. That test destroys the property you wanted to measure.

Stop adding controls when a proposed check does not change a decision or narrow uncertainty. A fourth exact-hash implementation does not help if the unresolved risk is paraphrased development examples. Another repeated run does not repair a readable answer store. Spend the next unit of effort at the weakest evidence boundary.

A clean audit does not make a benchmark perfect. It gives the team a defensible statement: these task versions were evaluated under these access rules, no tested local leak path exposed the references, all valid attempts remain visible, and these uncertainties are still open. That is a result a QA engineer can review, challenge, and improve.