Stop private data from leaking into agent trace graders

A grader marks an agent run safe, then someone opens the stored trace and finds a bearer token inside the tool arguments. The dashboard had replaced it with a black box, but the exported JSON still held the original credential. You now have two bugs: private data crossed a storage boundary, and the evidence used to approve the run cannot be trusted.

Trace grading makes this failure easy to miss. A useful grade often needs the sequence of model messages, tool names, arguments, results, errors, and policy decisions. That same detail can contain account identifiers, email addresses, session cookies, access tokens, support messages, document text, and values copied from one tool into the next. The better the trace is for debugging, the more carefully its collection path must be designed.

The right goal is not to turn every value into [REDACTED]. A grader that sees no arguments cannot tell whether the agent selected the correct customer, honored a tool restriction, or recovered from a bad response. The goal is a trace that preserves the facts needed by the rubric while removing data the grader, artifact store, and CI log have no reason to receive.

Find the copy that escaped your trust boundary

An agent usually needs the real value while it performs the live action. A mail tool cannot deliver a message without an address, and an authenticated HTTP client cannot make its request without credentials. Redaction therefore belongs on the observation branch, not in front of execution. Let the authorized tool receive its required input. Give the tracer a separately constructed safe representation.

Think of the path as a fan-out. The agent creates a tool call. One branch executes it. Other branches may write a local log, enqueue telemetry, persist a span, build a grader request, render a viewer, attach an artifact to CI, or export a dataset. Every branch after capture is a new copy. Sanitizing only the final HTML viewer leaves the queue message, database row, grader prompt, and downloadable file untouched.

This distinction explains a common incident report: "The value is redacted in the UI." That observation proves only what the UI rendered. It says nothing about the payload received by the browser, the response cached by an intermediary, or the record stored behind the page. Inspect the serialized value at the earliest outbound boundary and at each durable sink.

There are four useful classes of trace data:

1. Secrets grant authority or unlock protected material. Examples include an Authorization value, cookie, password, API key, refresh token, and private key. MDN describes the HTTP Authorization header as carrying credentials for access to a protected resource. A grader should not need the credential itself. It may need a derived fact such as authorization_present: true or credential_scope: "read-only", produced by trusted application code.
2. Direct identifiers point to a person or account. Email addresses, phone numbers, customer IDs, and ticket submitter names belong here. Delete them when correlation is unnecessary. Use a scoped pseudonym when the rubric must verify that two steps referred to the same subject.
3. Sensitive free text has no reliable field name. A model can repeat a token in its explanation, a tool can embed an email in an error, and a support note can contain health or financial details. A key denylist will not catch these values.
4. Operational structure makes grading possible. Tool names, call order, response class, approved argument keys, retry relationships, and terminal state often survive without exposing private values. Preserve these deliberately rather than retaining the whole payload by default.

Distributed tracing standards make the propagation risk concrete. The W3C Trace Context specification says personally identifiable or otherwise sensitive information does not belong in traceparent or tracestate. Those fields exist for correlation, not as convenient storage for a customer email. The W3C Baggage specification is even more relevant to application metadata: baggage values are application-defined and can be passed to downstream systems, so private entries must be removed when they should not propagate. An agent framework's trace format may differ, but the trust-boundary problem is the same.

Capture timing matters. If a tracing hook receives the live object before your exporter sanitizes it, another hook may log or retain that object. If an exception handler interpolates the raw arguments into a message, scrubbing the structured arguments field will not touch the copied text. If a grader request is built from the execution object rather than the sanitized record, the stored trace can be clean while the external request still leaks. Draw the actual data flow. Do not infer it from component names.

The privacy test should identify the first unsafe copy. A raw value in the in-memory execution request is expected when the tool needs it. The same value in a telemetry event before persistence is a failure. Finding that boundary tells the owner where to fix the pipeline without corrupting the real operation.

Redact once, before traces fan out

A practical redactor needs more than one rule. Exact field handling catches predictable secrets. Free-text rules catch a small set of recognizable copies. A keyed pseudonym preserves equality where the rubric needs it. An allowlist controls high-risk objects whose contents are too varied to sanitize safely.

The following Python module accepts JSON-compatible values and returns a new object. It never edits the live tool arguments. Secret-bearing fields become a marker, selected identifiers become keyed HMAC values, URL fields retain only a validated HTTP or HTTPS origin, body fields become a marker, and obvious credentials or email addresses copied into strings are removed. The full HMAC digest is retained in this example because arbitrary truncation adds a collision decision that the application may not have evaluated.

from __future__ import annotations

import hashlib
import hmac
import json
import os
import re
from typing import Any
from urllib.parse import urlsplit


REDACTED = "[REDACTED]"

URL_KEYS = frozenset({"url", "requesturl", "responseurl"})
BODY_KEYS = frozenset({"body", "responsebody"})

SECRET_KEYS = frozenset(
    {
        "authorization",
        "proxyauthorization",
        "cookie",
        "setcookie",
        "password",
        "passwd",
        "apikey",
        "accesstoken",
        "refreshtoken",
        "clientsecret",
        "privatekey",
    }
)

PSEUDONYM_KEYS = frozenset(
    {
        "userid",
        "accountid",
        "customerid",
        "email",
    }
)

EMAIL_IN_TEXT = re.compile(
    r"(?i)(?<![\w.+-])[\w.+-]+@[\w-]+(?:\.[\w-]+)+"
)
AUTH_IN_TEXT = re.compile(
    r"(?i)\b(?:bearer|basic)\s+[A-Za-z0-9._~+/=-]+"
)
NAMED_SECRET_IN_TEXT = re.compile(
    r"(?i)\b(api[_ -]?key|access[_ -]?token|refresh[_ -]?token|password)"
    r"\s*[:=]\s*[^\s,;]+"
)


def canonical_key(name: str) -> str:
    return re.sub(r"[^a-z0-9]", "", name.lower())


def pseudonymize(value: Any, hmac_key: bytes) -> str:
    digest = hmac.new(
        hmac_key,
        str(value).encode("utf-8"),
        hashlib.sha256,
    ).hexdigest()
    return f"subject:hmac-sha256:{digest}"


def origin_only(value: Any) -> str:
    if not isinstance(value, str):
        return REDACTED

    parsed = urlsplit(value)
    if parsed.scheme.lower() not in {"http", "https"} or parsed.hostname is None:
        return REDACTED

    try:
        port = parsed.port
    except ValueError:
        return REDACTED

    host = parsed.hostname
    if ":" in host:
        host = f"[{host}]"
    if port is not None:
        host = f"{host}:{port}"
    return f"{parsed.scheme.lower()}://{host.lower()}"


def scrub_text(value: str) -> str:
    value = AUTH_IN_TEXT.sub("[AUTHORIZATION]", value)
    value = EMAIL_IN_TEXT.sub("[EMAIL]", value)
    return NAMED_SECRET_IN_TEXT.sub(
        lambda match: f"{match.group(1)}=[REDACTED]",
        value,
    )


def redact(
    value: Any,
    *,
    hmac_key: bytes,
    field_name: str | None = None,
) -> Any:
    key = canonical_key(field_name) if field_name is not None else None

    if key in SECRET_KEYS:
        return REDACTED

    if key in BODY_KEYS:
        return REDACTED

    if key in URL_KEYS:
        return origin_only(value)

    if key in PSEUDONYM_KEYS and value is not None:
        return pseudonymize(value, hmac_key)

    if isinstance(value, dict):
        return {
            str(child_key): redact(
                child_value,
                hmac_key=hmac_key,
                field_name=str(child_key),
            )
            for child_key, child_value in value.items()
        }

    if isinstance(value, list):
        return [redact(item, hmac_key=hmac_key) for item in value]

    if isinstance(value, str):
        return scrub_text(value)

    if value is None or isinstance(value, (bool, int, float)):
        return value

    raise TypeError(f"trace contains unsupported value: {type(value).__name__}")


def safe_trace_json(raw_trace: dict[str, Any]) -> str:
    hmac_key = os.environ["TRACE_REDACTION_HMAC_KEY"].encode("utf-8")
    safe_trace = redact(raw_trace, hmac_key=hmac_key)
    return json.dumps(safe_trace, separators=(",", ":"), sort_keys=True)

The module intentionally fails on an unsupported Python object instead of calling str() on it. An object's string representation can include every private field it owns, and that fallback silently creates a new leak. Convert framework-specific objects into a reviewed schema before calling the redactor.

Worked example: credentials inside nested arguments. Suppose an agent calls a customer API with a bearer token and an email. The live request still needs both. The trace adapter can separately record the route template, method, response status, whether authentication was present, and a pseudonym for the customer. It must not record the token, raw email, full request URL, or unfiltered response body.

import json
import os

from trace_redaction import redact


raw_call = {
    "tool": "fetch_customer_orders",
    "arguments": {
        "method": "GET",
        "url": "https://api.example.test/customers/customer-482/orders?include=items",
        "headers": {
            "Authorization": "Bearer qa-canary-token-47",
            "Cookie": "session=qa-canary-cookie-18",
        },
        "email": "privacy-canary@example.test",
    },
    "result": {
        "status": 200,
        "body": {"order_count": 2},
    },
    "grading_evidence": {
        "authorization_present": True,
        "route_template": "/customers/{customer_id}/orders",
    },
}

hmac_key = os.environ["TRACE_REDACTION_HMAC_KEY"].encode("utf-8")
safe_call = redact(raw_call, hmac_key=hmac_key)
serialized = json.dumps(safe_call, sort_keys=True)

assert safe_call["arguments"]["headers"]["Authorization"] == "[REDACTED]"
assert safe_call["arguments"]["headers"]["Cookie"] == "[REDACTED]"
assert safe_call["arguments"]["email"].startswith("subject:hmac-sha256:")
assert safe_call["arguments"]["url"] == "https://api.example.test"
assert safe_call["result"]["body"] == "[REDACTED]"
assert safe_call["grading_evidence"]["authorization_present"] is True
assert "qa-canary-token-47" not in serialized
assert "privacy-canary@example.test" not in serialized
assert "/customers/customer-482/orders" not in serialized
assert "include=items" not in serialized
assert "order_count" not in serialized

The authorization_present field is not produced by parsing the redacted marker. Trusted code derives it from the live request before the trace record is built. This matters because [REDACTED] can mean several things: a secret existed, policy removed an optional value, or a fixture supplied the marker itself. Derived evidence should say exactly which fact the rubric may use.

This policy has a maintenance cost. New clients invent field names such as credential, auth, or sessionToken. A denylist will miss them until someone adds a rule. For high-risk tool families, build an allowlisted trace adapter that constructs a new record from approved fields. Use generic recursive redaction as a second layer, not as permission to serialize an unknown object.

Exercise the failures that a key denylist misses

Most privacy suites test password and declare victory. Production leaks tend to arrive through copying and shape changes. The same token appears inside an exception. A URL gains a signed query parameter. A model repeats an email in its final explanation. A tool result returns an entire customer record when the grader needs only a status. Each path needs a distinct fixture because one field-level assertion cannot prove the others.

Worked example: a secret copied into free text. An HTTP wrapper can produce an error message such as request failed with Authorization: Bearer .... Replacing the structured headers.Authorization value does not affect that string. The first module's text rules catch the synthetic bearer token and email pattern, but pattern matching is deliberately limited. It should not be marketed as a general personal-data detector.

Run tests against the final serialized representation, not only the Python dictionary. A custom JSON encoder, log formatter, or exporter can reintroduce raw values after the unit under test returns. These pytest cases use synthetic markers, so a failure identifies an exact leak without putting a real credential in a test repository.

import json
import logging

import pytest

from trace_redaction import redact


@pytest.fixture
def hmac_key() -> bytes:
    return b"unit-test-key-with-no-production-use"


@pytest.mark.parametrize(
    ("raw_value", "forbidden"),
    [
        (
            {"error": "request failed: Bearer qa-canary-error-91"},
            "qa-canary-error-91",
        ),
        (
            {"message": "Contact privacy-canary@example.test"},
            "privacy-canary@example.test",
        ),
        (
            {"meta": {"refresh-token": "qa-canary-refresh-32"}},
            "qa-canary-refresh-32",
        ),
    ],
)
def test_serialized_trace_does_not_contain_canary(
    raw_value: dict,
    forbidden: str,
    hmac_key: bytes,
) -> None:
    safe_value = redact(raw_value, hmac_key=hmac_key)
    assert forbidden not in json.dumps(safe_value, sort_keys=True)


def test_redaction_does_not_mutate_live_arguments(hmac_key: bytes) -> None:
    raw_value = {"headers": {"Authorization": "Bearer live-tool-token"}}
    safe_value = redact(raw_value, hmac_key=hmac_key)

    assert raw_value["headers"]["Authorization"] == "Bearer live-tool-token"
    assert safe_value["headers"]["Authorization"] == "[REDACTED]"


def test_safe_trace_is_what_the_logger_receives(
    caplog: pytest.LogCaptureFixture,
    hmac_key: bytes,
) -> None:
    logger = logging.getLogger("trace_export")
    raw_value = {"error": "email=privacy-canary@example.test"}
    safe_value = redact(raw_value, hmac_key=hmac_key)

    with caplog.at_level(logging.INFO, logger="trace_export"):
        logger.info("trace=%s", json.dumps(safe_value, sort_keys=True))

    assert "privacy-canary@example.test" not in caplog.text
    assert "[EMAIL]" in caplog.text

Pytest's caplog fixture exposes captured logging records and text to the test. That is useful here because the logger is a separate sink. Pytest also captures standard output and standard error, and captured output can be shown when a test fails. A clean JSON artifact does not compensate for a raw print(raw_trace) in the failing test path.

Worked example: stable identity without raw identity. A grader may need to confirm that the account selected in step two matches the account approved in step one. Replacing every ID with the same marker destroys that relation. A keyed HMAC preserves equality for identical inputs under the same key. It does not make the value anonymous. Anyone who can query the pseudonymizer or obtain its key can test guesses, and a stable output remains linkable across traces.

Choose the scope deliberately. A per-run key permits joins inside one trace but prevents comparison across runs. A per-dataset key supports regression analysis across that dataset but increases linkability. A long-lived organization-wide key creates the widest correlation surface. The code cannot choose that privacy decision for you. Document it in the trace policy and restrict access to the key separately from the artifacts.

Worked example: a response body that is too rich to clean. Consider a lookup_customer result containing name, address, notes, payment metadata, and an eligibility flag. Writing regexes for each possible field is the wrong abstraction when the rubric needs only eligible: true and lookup_status: "found". Build an allowlisted evidence object at the tool boundary. If the raw response is required to execute later steps, keep it in the authorized runtime state, not in the grading record.

A typed marker also carries useful meaning. [SECRET], [EMAIL], [USER_ID:HMAC], and [OMITTED_BODY] let the grader distinguish categories without seeing values. Keep the vocabulary small and stable. Do not include secret prefixes, last characters, email domains, or string lengths unless a documented rubric needs those features. Even partial values can narrow the identity or credential.

Free-text detection remains a backstop. Regular expressions produce false negatives when a secret has an unfamiliar shape and false positives when ordinary text resembles a token. A model can split a value across fields or encode it. For a tool that processes arbitrary documents, the safest grading trace may include a content classification, byte count, parser result, and approved extract rather than the document text. That costs debugging detail, but it puts the decision where the schema is understood.

Prove which sink leaked instead of blaming the grader

Start with a synthetic trace containing a different canary in every risky location. Use recognizable values that cannot occur by accident. Put markers in a nested header, email field, URL query, free-text error, tool result, model message, and logging call. Execute the normal capture and export path. Search every resulting sink separately.

The scanner below audits JSON or JSON Lines artifacts. It reports an unsafe sensitive field, an email-shaped string, an authorization-shaped string, or an exact canary supplied on the command line. Because it exits nonzero when it finds anything, the same command can run locally and in CI. Its patterns are a test oracle for controlled fixtures, not a promise to discover every secret in production data.

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path
from typing import Any, Iterator


REDACTED = "[REDACTED]"
SECRET_KEYS = frozenset(
    {
        "authorization",
        "proxyauthorization",
        "cookie",
        "setcookie",
        "password",
        "passwd",
        "apikey",
        "accesstoken",
        "refreshtoken",
        "clientsecret",
        "privatekey",
    }
)
EMAIL = re.compile(r"(?i)(?<![\w.+-])[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
AUTH = re.compile(r"(?i)\b(?:bearer|basic)\s+[A-Za-z0-9._~+/=-]+")


def canonical_key(name: str) -> str:
    return re.sub(r"[^a-z0-9]", "", name.lower())


def iter_records(path: Path) -> Iterator[tuple[int, Any]]:
    with path.open("r", encoding="utf-8") as source:
        if path.suffix == ".jsonl":
            for line_number, line in enumerate(source, start=1):
                if line.strip():
                    yield line_number, json.loads(line)
        else:
            yield 1, json.load(source)


def findings(
    value: Any,
    *,
    json_path: str = "$",
    canaries: tuple[str, ...] = (),
) -> Iterator[tuple[str, str]]:
    if isinstance(value, dict):
        for key, child in value.items():
            child_path = f"{json_path}.{key}"
            if canonical_key(str(key)) in SECRET_KEYS and child != REDACTED:
                yield child_path, "sensitive key contains an unsanitized value"
            yield from findings(
                child,
                json_path=child_path,
                canaries=canaries,
            )
        return

    if isinstance(value, list):
        for index, child in enumerate(value):
            yield from findings(
                child,
                json_path=f"{json_path}[{index}]",
                canaries=canaries,
            )
        return

    if not isinstance(value, str):
        return

    if EMAIL.search(value):
        yield json_path, "email-shaped value remains"
    if AUTH.search(value):
        yield json_path, "authorization-shaped value remains"
    for canary in canaries:
        if canary in value:
            yield json_path, f"synthetic canary remains: {canary}"


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("artifact", type=Path)
    parser.add_argument("--canary", action="append", default=[])
    args = parser.parse_args()

    failed = False
    for line_number, record in iter_records(args.artifact):
        for json_path, reason in findings(
            record,
            canaries=tuple(args.canary),
        ):
            failed = True
            print(
                f"FAIL {args.artifact}:{line_number} "
                f"{json_path} {reason}"
            )
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

With a deliberately unsafe synthetic fixture, the diagnostic identifies both location and rule. The lines beginning with # below show the expected output from this scanner; they are not measurements from a production system.

python scripts/audit_trace.py artifacts/synthetic-trace.jsonl \
  --canary qa-canary-token-47 \
  --canary qa-canary-error-91
# FAIL artifacts/synthetic-trace.jsonl:1 $.spans[1].arguments.headers.Authorization sensitive key contains an unsanitized value
# FAIL artifacts/synthetic-trace.jsonl:1 $.spans[1].arguments.headers.Authorization authorization-shaped value remains
# FAIL artifacts/synthetic-trace.jsonl:1 $.spans[1].arguments.headers.Authorization synthetic canary remains: qa-canary-token-47
# FAIL artifacts/synthetic-trace.jsonl:1 $.spans[3].error synthetic canary remains: qa-canary-error-91

Multiple messages for one path are useful during development because they prove independent rules fired. A CI reporter may deduplicate them after retaining the strongest reason. Do not print the matched production value. The scanner prints only a synthetic canary supplied by the test author; for uncontrolled data, report the rule and JSON path.

Use the evidence to separate similar failures:

The 401 row is an important near-miss. MDN documents 401 Unauthorized as the response when a request lacks valid authentication credentials. If a fixture never attached its header, the absence of a token in the trace does not prove redaction worked. Assert that the synthetic secret entered the live execution branch, then assert that it did not enter the observation branch. Both facts are required.

Another near-miss is encryption. Encrypted storage can reduce exposure to someone who lacks decryption access, but the grader or viewer may decrypt the record and receive the original value. Encryption and redaction solve different problems. Test the plaintext at the consumer boundary, not the bytes at rest.

A third near-miss comes from sampling. A clean sampled trace says nothing about an unsampled tool path that uses a different serializer. Run one synthetic privacy case for every adapter and exception path. The W3C Baggage specification explicitly calls for testing code paths that send baggage; the same discipline applies when agent tools have separate exporters.

Roll the policy out without blinding your regression suite

Changing trace shape can break graders even when the application is correct. Treat redaction as a versioned data contract. A grader that previously compared raw customer_id values needs a replacement equality signal. A rubric that read an error body may need an error category. A report that linked directly to a document may need a safe artifact reference or no link at all.

Start with an inventory, not a global regex. List every producer and sink: agent messages, tool arguments, tool results, exception objects, HTTP metadata, model responses, local logs, telemetry callbacks, queues, databases, grader inputs, review screens, dataset exports, and CI artifacts. For each field, record whether execution needs it, whether grading needs it, the safe transformation, retention owner, and deletion path. Unknown fields should not default to unrestricted storage.

Next, create a synthetic corpus. One trace should exercise nested structured secrets. Another should copy markers into free text. A third should test pseudonym equality across spans. Add cases for failure handling, retries, serialization errors, and cancelled runs because cleanup paths often bypass the normal exporter. These are different failures, not paraphrases of one happy path.

Introduce the sanitized schema beside the current grader contract, but do not create another durable store of raw production traces for comparison. You can replay reviewed synthetic traces through old and new rubric adapters. For approved non-sensitive historical fixtures, compare decisions and record why they differ. When a rubric loses necessary evidence, add a narrow derived field rather than reopening the whole payload.

Version three things together:

• The trace schema identifies which fields exist and their types.
• The redaction policy identifies which fields are removed, marked, pseudonymized, or allowlisted.
• The grader rubric identifies which safe evidence produces a score or release decision.

Store those version identifiers in the sanitized record. Never store the HMAC key, denylist source values, or matched secret beside it. A reviewer should be able to reproduce the transformation using controlled fixtures without gaining access to a customer's original data.

Wire the privacy lane before the broader eval lane. If privacy tests fail, do not send the candidate artifact to a grader to see whether it still scores well. The grader is one of the sinks being protected. This shell script assumes the project environment already contains its declared test dependencies; it does not install or download anything during the privacy job.

#!/usr/bin/env bash
set -euo pipefail

python -m pytest -q tests/privacy/test_trace_redaction.py
test -f tests/fixtures/synthetic-trace.jsonl
python scripts/audit_trace.py tests/fixtures/synthetic-trace.jsonl \
  --canary qa-canary-token-47 \
  --canary qa-canary-error-91 \
  --canary privacy-canary@example.test
python -m pytest -q tests/evals/test_sanitized_trace_grader.py

Keep the privacy test output itself clean. A failed assertion such as assert secret not in payload can cause some test frameworks or assertion helpers to print the entire payload. Prefer a scanner that reports the path and rule. Review CI settings that enable live logs or local variables in tracebacks. More diagnostic output is not automatically safer output.

Roll out adapter by adapter. Begin with a low-variance tool whose safe schema is obvious, such as a lookup that can expose status, found, and a scoped pseudonym. Then migrate write tools, document tools, browser tools, and free-form code execution paths. Each adapter should graduate only after its live branch still receives required data, every observation sink is clean, and its grader has enough safe evidence.

Existing raw artifacts need a separate response. Stop further propagation first. Identify which copies exist and who can access them. Rotate any credential that may have been exposed. Then involve the people responsible for security, privacy, retention, and customer commitments in deciding quarantine or deletion. A code rewrite cannot determine the legal or contractual handling of already stored personal data.

The useful release evidence is small: privacy fixtures executed, sinks scanned, schema and policy versions, failures by rule, and owners for unresolved adapters. Counts from a synthetic suite describe coverage of those fixtures, not the absence of every possible private value. Avoid turning a passing regex scan into a broad privacy claim.

Accept the costs and know when redaction is the wrong tool

Every privacy control spends something. The first cost is grading signal. Removing a document body prevents a grader from checking whether the response faithfully summarized that document. You may replace the body with reviewed assertions, a synthetic document, or a specialized evaluation inside a tighter boundary. None is identical to grading the original production content.

The second cost is engineering complexity. Field rules must follow schema changes. Text rules need adversarial fixtures. Tool adapters need ownership. Policy and rubric versions must stay aligned. A generic middleware layer is attractive because it is easy to install, but it sees values after domain meaning has been lost. The tool adapter usually knows whether id names a public catalog item, an employee, or a payment account.

The third cost is runtime work. Copying a large nested trace, scanning strings, and serializing it for verification add processing and memory pressure. Do not publish invented latency figures. Benchmark the actual payload distribution in your environment with synthetic or approved data, then decide whether to omit large fields earlier, stream safe events, or move expensive checks to a test lane. The production boundary still needs deterministic protection.

The fourth cost is false confidence. Regexes recognize formats, not meaning. A string can be private without resembling an email or token. A string can resemble a token while being harmless test syntax. Measure rules separately, retain false-positive fixtures, and prefer construction from approved fields where a false negative would matter.

Pseudonymization has its own price. Stable identifiers help a grader follow an entity through a trace, but they also let anyone with multiple artifacts correlate that entity. Scope the HMAC key as narrowly as the rubric permits. Do not describe hashed or pseudonymized traces as anonymous. Linkability is a feature for grading and a privacy risk at the same time.

There are also cases where redaction should not touch the payload in question:

• Do not sanitize the live tool request. Replacing a real account ID or credential before execution changes application behavior and invalidates the trace. Sanitize the separate observation object.
• Do not replace every trace identifier with one constant marker. Correctly generated trace IDs provide graph correlation, and the W3C specification reserves trace context for that job. Keep compliant identifiers free of personal data, evaluate trust boundaries, and avoid destroying uniqueness merely to make the record look private.
• Do not retain an unknown object because a regex found nothing. Absence of a pattern match is not an allowlist. Drop the object or map it to a reviewed schema.
• Do not use a general-purpose grader for a task that genuinely requires raw protected content. Run that evaluation inside an explicitly approved boundary with appropriate access and retention controls, or redesign the test around synthetic data. A redacted input cannot prove content-level correctness it no longer contains.
• Do not redact harmless synthetic values so aggressively that the test loses its oracle. Purpose-built fixtures can contain fake addresses, tokens, and documents designed for evaluation. Mark them clearly, keep them isolated from production, and use them to test the full behavior when no real person is represented.
• Do not make a UI mask your deletion mechanism. Presentation is the last layer. Protection belongs before storage, propagation, grading, and logging.

Sometimes exclusion is better than replacement. If a screenshot, document, database row, or code execution transcript has no defined grading purpose, omit it. A marker is useful when the grader needs to know that a field existed. No field is better when even its presence reveals something the consumer does not need.

The strongest design gives every retained field a reason. The tool still performs the authorized action with its real input. The trace records a purpose-built account of what happened. The grader receives enough structure to judge the behavior. CI proves synthetic secrets cannot cross the observation boundary. When those four statements are independently testable, privacy redaction becomes an engineering contract rather than a cosmetic filter.