When an agent keeps planning but never gets closer

The research agent calls the same search tool four times, reaches its step limit, and still returns no supplier list. The dashboard stays green because the process exited normally. A customer sees an empty answer, while the trace shows nearly identical searches and a forced stop. That is a planning failure, even though no tool call crashed.

Why the trace tells a different story than the final status

A runtime status answers a narrow question: did the process complete, fail, time out, or get cancelled? It does not tell you whether the agent made progress. Planning quality lives in the ordered path between the request and the answer.

That distinction changes the test oracle. Checking for an HTTP 200 response, a non-null run identifier, or a worker exit code proves transport and runtime behavior. A planning detector needs the goal, the ordered events, the tool outcomes, and the termination reason. Without those fields, it can count activity but cannot judge progress.

Three detector families are useful because they rely on evidence you can preserve.

A repeated-work rule looks for a sequence of calls to the same tool whose returned entities overlap heavily. Result identity is stronger evidence than similar wording. Two searches can use different phrases and still retrieve the same records. Conversely, repeated wording can be valid when the tool returns new evidence after an index update or a filter change. A credible alert therefore points to the event span and the result overlap, rather than declaring that two strings “look similar.”

A termination rule checks explicit state. If the runtime records a step-limit termination and the answer is empty, the user did not receive the requested result. That finding does not depend on why the budget was chosen. It also does not mean every run that uses its full allowance is broken. A completed run with a usable answer has a different termination state and should stay separate.

A tool-contract rule compares each call with a list approved for the task category. This is a deliberately modest version of goal-drift detection. It can prove that a supplier lookup used a tool outside the supplier workflow contract. It cannot prove that weather data is always irrelevant. A logistics workflow might need a weather check. The permitted-tool list belongs to the product specification, and a detector should report a contract mismatch rather than pretend it can read intent from a tool name.

Event order has to survive ingestion. Give every event a stable identifier before the trace reaches the detector. Timestamps alone are a poor substitute because several events can share a timestamp, clocks can differ across services, and buffered exporters can deliver records later. If your system propagates W3C Trace Context, its traceparent and tracestate fields help correlate work across services. The standard does not define an agent’s internal step schema or guarantee application-event order. You still need your own event identifiers and ordering rule.

A minimum trace record for these checks contains:

Do not silently replace missing values with empty collections. An absent result list means “unknown,” while an observed empty result means “the tool returned no entities.” Collapsing those states can create both false positives and false negatives. The detector should emit an evidence-gap finding or leave the trace unclassified.

Build the detector around evidence, not vibes

The following detector is intentionally small. It uses only Python’s standard library, takes immutable trace objects, and returns structured findings. The repeated-call count and overlap value are illustrative policy settings for the worked fixtures, not measurements from a production system.

Save this as planning_detector.py:

from __future__ import annotations

from dataclasses import dataclass
from typing import Literal

Termination = Literal["completed", "step_limit", "cancelled", "tool_error"]
FindingKind = Literal[
    "repeated_result_loop",
    "step_limit_without_answer",
    "tool_outside_task_contract",
    "insufficient_loop_evidence",
]


@dataclass(frozen=True)
class Step:
    event_id: str
    tool: str
    result_ids: tuple[str, ...] | None


@dataclass(frozen=True)
class Trace:
    trace_id: str
    task_category: str
    termination: Termination
    answer: str | None
    steps: tuple[Step, ...]


@dataclass(frozen=True)
class Policy:
    revision: str
    repeated_call_count: int
    minimum_result_overlap: float
    permitted_tools: frozenset[str]


@dataclass(frozen=True)
class Finding:
    kind: FindingKind
    event_ids: tuple[str, ...]
    policy_revision: str
    reason: str


def overlap_coefficient(
    left: tuple[str, ...],
    right: tuple[str, ...],
) -> float:
    left_set = set(left)
    right_set = set(right)
    if not left_set or not right_set:
        return 0.0
    return len(left_set & right_set) / min(len(left_set), len(right_set))


def detect(trace: Trace, policy: Policy) -> tuple[Finding, ...]:
    if policy.repeated_call_count < 2:
        raise ValueError("repeated_call_count must be at least 2")
    if not 0.0 <= policy.minimum_result_overlap <= 1.0:
        raise ValueError("minimum_result_overlap must be between 0 and 1")

    findings: list[Finding] = []

    for step in trace.steps:
        if step.tool not in policy.permitted_tools:
            findings.append(
                Finding(
                    kind="tool_outside_task_contract",
                    event_ids=(step.event_id,),
                    policy_revision=policy.revision,
                    reason=(
                        f"{step.tool!r} is not permitted for "
                        f"task category {trace.task_category!r}"
                    ),
                )
            )

    if trace.termination == "step_limit" and not (trace.answer or "").strip():
        final_event = trace.steps[-1].event_id if trace.steps else "no-events"
        findings.append(
            Finding(
                kind="step_limit_without_answer",
                event_ids=(final_event,),
                policy_revision=policy.revision,
                reason="the run reached its step limit without a non-empty answer",
            )
        )

    width = policy.repeated_call_count
    for start in range(0, len(trace.steps) - width + 1):
        window = trace.steps[start : start + width]
        if len({step.tool for step in window}) != 1:
            continue

        if any(step.result_ids is None for step in window):
            findings.append(
                Finding(
                    kind="insufficient_loop_evidence",
                    event_ids=tuple(step.event_id for step in window),
                    policy_revision=policy.revision,
                    reason="a repeated-tool window is missing observed result identifiers",
                )
            )
            break

        result_sets = [step.result_ids for step in window]
        assert all(result_ids is not None for result_ids in result_sets)
        overlaps = [
            overlap_coefficient(result_sets[index], result_sets[index + 1])
            for index in range(len(result_sets) - 1)
        ]

        if all(
            value >= policy.minimum_result_overlap
            for value in overlaps
        ):
            findings.append(
                Finding(
                    kind="repeated_result_loop",
                    event_ids=tuple(step.event_id for step in window),
                    policy_revision=policy.revision,
                    reason=(
                        "consecutive calls used the same tool; "
                        f"adjacent result overlaps were {overlaps!r}"
                    ),
                )
            )
            break

    return tuple(findings)

Returning event identifiers matters as much as returning the label. A boolean tells CI whether something failed. The identifiers tell an engineer where to start. Include the policy revision because a trace replayed after a threshold change may receive a different verdict for a legitimate reason.

Notice the treatment of missing results. A repeated tool window with one unobserved result list produces insufficient_loop_evidence, not a loop. An empty tuple is different: it says the tool returned an observed empty set. For empty sets, the overlap coefficient is zero, so this specific result-repetition rule does not fire. You could define a separate repeated-empty-search rule, but it needs its own name and fixtures. Hiding that second policy inside the overlap calculation would make alerts difficult to explain.

Result identifiers also need a contract. Prefer stable business keys, canonical document URLs, or provider result IDs. Do not hash an entire rendered response and call it identity. A timestamp, ranking change, or irrelevant formatting difference would make the same result appear new. If a provider exposes no stable key, normalize only fields you understand and retain the raw response for investigation.

Work through three failures that need different verdicts

Save the tests as test_planning_detector.py beside the detector:

import pytest

from planning_detector import Policy, Step, Trace, detect


SEARCH_POLICY = Policy(
    revision="supplier-search-v1",
    repeated_call_count=3,
    minimum_result_overlap=0.80,
    permitted_tools=frozenset({"web_search", "supplier_record"}),
)


def trace(
    *,
    trace_id: str,
    steps: tuple[Step, ...],
    termination: str = "completed",
    answer: str | None = "Supplier list attached",
) -> Trace:
    return Trace(
        trace_id=trace_id,
        task_category="supplier_search",
        termination=termination,
        answer=answer,
        steps=steps,
    )


def finding_kinds(run: Trace) -> set[str]:
    return {finding.kind for finding in detect(run, SEARCH_POLICY)}


def test_repeated_results_return_the_triggering_span() -> None:
    run = trace(
        trace_id="loop-fixture",
        steps=(
            Step("event-01", "web_search", ("a", "b", "c", "d", "e")),
            Step("event-02", "web_search", ("a", "b", "c", "d", "f")),
            Step("event-03", "web_search", ("a", "b", "c", "d", "g")),
        ),
    )

    findings = detect(run, SEARCH_POLICY)

    loop = next(
        finding
        for finding in findings
        if finding.kind == "repeated_result_loop"
    )
    assert loop.event_ids == ("event-01", "event-02", "event-03")
    assert loop.policy_revision == "supplier-search-v1"


def test_constraint_refinement_is_not_a_result_loop() -> None:
    run = trace(
        trace_id="healthy-refinement",
        steps=(
            Step("event-01", "web_search", ("europe-1", "europe-2")),
            Step("event-02", "web_search", ("iso-1", "iso-2")),
            Step("event-03", "web_search", ("audit-1", "audit-2")),
        ),
    )

    assert "repeated_result_loop" not in finding_kinds(run)


@pytest.mark.parametrize(
    ("termination", "answer", "expected"),
    [
        ("step_limit", None, True),
        ("step_limit", "Partial supplier list", False),
        ("cancelled", None, False),
        ("tool_error", None, False),
        ("completed", "Supplier list", False),
    ],
)
def test_step_limit_needs_its_own_evidence(
    termination: str,
    answer: str | None,
    expected: bool,
) -> None:
    run = trace(
        trace_id=f"termination-{termination}",
        termination=termination,
        answer=answer,
        steps=(Step("event-01", "web_search", ("a",)),),
    )

    present = "step_limit_without_answer" in finding_kinds(run)

    assert present is expected


def test_unknown_results_do_not_become_a_loop() -> None:
    run = trace(
        trace_id="telemetry-gap",
        steps=(
            Step("event-01", "web_search", ("a", "b")),
            Step("event-02", "web_search", None),
            Step("event-03", "web_search", ("a", "b")),
        ),
    )

    kinds = finding_kinds(run)

    assert "insufficient_loop_evidence" in kinds
    assert "repeated_result_loop" not in kinds


def test_tool_contract_reports_only_the_offending_event() -> None:
    run = trace(
        trace_id="unexpected-tool",
        steps=(
            Step("event-01", "web_search", ("a",)),
            Step("event-02", "get_weather", ("forecast-1",)),
            Step("event-03", "supplier_record", ("supplier-a",)),
        ),
    )

    finding = next(
        item
        for item in detect(run, SEARCH_POLICY)
        if item.kind == "tool_outside_task_contract"
    )
    assert finding.event_ids == ("event-02",)
    assert "get_weather" in finding.reason

The pytest parametrization guide confirms that each argument set runs as a separate case. Here, that keeps five termination states visible and makes any future contract change a reviewable fixture edit.

The loop fixture illustrates a policy boundary. Every adjacent pair shares four of the smaller five-element set, so the overlap coefficient is 0.80. That figure comes from the fixture data and demonstrates the calculation. It is not a claimed production rate. The healthy fixture uses disjoint result identities to prove restraint, not merely sensitivity.

The second failure is a forced stop. Its evidence is not result overlap. The termination must be step_limit, and the answer must contain no non-whitespace text. A cancellation has no answer too, but it belongs to a different operational path. A tool error can also end without an answer. Combining all three as “agent stalled” discards the owner and the likely repair.

The third failure is a contract mismatch. The weather event in the fixture is flagged because get_weather is absent from the supplier-search policy. If the product owner approves weather checks for port-risk analysis, the correct fix is to update and version that task policy. Lowering a similarity threshold or adding a special exception inside the detector would fix the wrong layer.

Separate planning bugs from look-alike failures

Start debugging before threshold tuning. Replay the stored trace through the detector and policy revision named by the alert, then confirm that ingestion preserved event order, attempt boundaries, tool outcomes, and missing values. Save this adapter as replay_trace.py.

from __future__ import annotations

import json
import sys
from dataclasses import asdict
from pathlib import Path
from typing import Any

from planning_detector import Policy, Step, Trace, detect


def load_object(path: str) -> dict[str, Any]:
    value = json.loads(Path(path).read_text(encoding="utf-8"))
    if not isinstance(value, dict):
        raise ValueError(f"{path} must contain one JSON object")
    return value


def load_trace(path: str) -> Trace:
    raw = load_object(path)
    return Trace(
        trace_id=str(raw["trace_id"]),
        task_category=str(raw["task_category"]),
        termination=raw["termination"],
        answer=raw.get("answer"),
        steps=tuple(
            Step(
                event_id=str(step["event_id"]),
                tool=str(step["tool"]),
                result_ids=(
                    None
                    if "result_ids" not in step
                    else tuple(str(item) for item in step["result_ids"])
                ),
            )
            for step in raw["steps"]
        ),
    )


def load_policy(path: str) -> Policy:
    raw = load_object(path)
    return Policy(
        revision=str(raw["revision"]),
        repeated_call_count=int(raw["repeated_call_count"]),
        minimum_result_overlap=float(raw["minimum_result_overlap"]),
        permitted_tools=frozenset(
            str(tool) for tool in raw["permitted_tools"]
        ),
    )


def main(arguments: list[str]) -> int:
    if len(arguments) != 3:
        print(
            "usage: replay_trace.py TRACE.json POLICY.json",
            file=sys.stderr,
        )
        return 2

    trace = load_trace(arguments[1])
    policy = load_policy(arguments[2])
    output = {
        "trace_id": trace.trace_id,
        "policy_revision": policy.revision,
        "findings": [asdict(item) for item in detect(trace, policy)],
    }
    print(json.dumps(output, indent=2, sort_keys=True))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

The following runnable diagnostic builds the same three-event fixture, prints the adapter's JSON, and verifies the exact finding coordinates. Its identifiers and overlap values come from the fixture, not production telemetry.

#!/usr/bin/env bash
set -euo pipefail

scratch_dir="$(mktemp -d)"
trap 'rm -rf "$scratch_dir"' EXIT

cat > "$scratch_dir/loop.json" <<'JSON'
{
  "trace_id": "loop-fixture",
  "task_category": "supplier_search",
  "termination": "completed",
  "answer": "Supplier list attached",
  "steps": [
    {"event_id": "event-01", "tool": "web_search", "result_ids": ["a", "b", "c", "d", "e"]},
    {"event_id": "event-02", "tool": "web_search", "result_ids": ["a", "b", "c", "d", "f"]},
    {"event_id": "event-03", "tool": "web_search", "result_ids": ["a", "b", "c", "d", "g"]}
  ]
}
JSON

cat > "$scratch_dir/policy.json" <<'JSON'
{
  "revision": "supplier-search-v1",
  "repeated_call_count": 3,
  "minimum_result_overlap": 0.8,
  "permitted_tools": ["web_search", "supplier_record"]
}
JSON

output_file="$scratch_dir/output.json"
python replay_trace.py \
  "$scratch_dir/loop.json" \
  "$scratch_dir/policy.json" \
  | tee "$output_file"

python -c '
import json
import sys

data = json.load(open(sys.argv[1], encoding="utf-8"))
finding = data["findings"][0]
assert finding["kind"] == "repeated_result_loop"
assert finding["event_ids"] == ["event-01", "event-02", "event-03"]
assert data["policy_revision"] == "supplier-search-v1"
' "$output_file"

Compare the event identifiers with the exporter records first. One emitted search stored three times points to delivery or deduplication, not poor planning. Next, check whether normalization erased meaningful document revisions and confirm that the named policy revision belongs to the trace's task category.

Read the source termination event last. A worker timeout, user cancellation, and application step limit can all end without an answer, but only the explicit step-limit state satisfies this detector's forced-stop rule. An adapter that maps every incomplete run to that state creates the false alert.

A near-miss deserves its own fixture when it appears in review. Tool retries after a transient error often resemble loops because the arguments and tool name repeat. The difference is the intervening outcome. If the first event records a retryable error and the next call succeeds, use a retry-specific rule or exclude that pair from repeated-result detection. Do not infer success or failure from a missing result list. Missing telemetry remains unknown.

Another near-miss comes from pagination. Three calls to the same search endpoint can be healthy when each request carries a different page token and each response contains new identifiers. The detector above naturally avoids a loop if results do not overlap. If your provider repeats pinned results on every page, compare the non-pinned portion or include the page token in a dedicated pagination rule. A generic threshold cannot understand provider ranking behavior.

Pytest's logging documentation describes captured logs and the caplog fixture. Use it to assert policy-selection or ingestion-error logs when those messages are contractual, while keeping the structured finding as the primary oracle.

Trace UIs are useful only when they show the fields behind the verdict. Open the flagged event span and check the raw tool name, result IDs, termination event, and policy revision. A rendered “planning” node or duration bar by itself cannot confirm this rule. If the viewer hides normalized values, include a link to the stored replay artifact so the reviewer can compare raw and derived evidence.

When the plan changes but the tool cache stands still

A second failure produces the detector's strongest loop signature even though the planner is refining the task. Imagine three supplier searches. The first requests all regional suppliers, the second adds a certification constraint, and the third narrows the delivery region. A defective adapter or cache key ignores those changing constraints and serves the same stored response each time. The trace contains one tool name, nearly identical result identifiers, and no useful result change. It looks like repeated planning, but the first wrong boundary is the tool path.

Start with the replay output. A broken fixture will show kind as repeated_result_loop, three event ids, the selected policy revision, and adjacent overlap values at or above the fixture boundary. A healthy non-loop fixture produces no finding of that kind. An evidence-gap fixture produces insufficient_loop_evidence instead. Those are detector conclusions, not root-cause labels. Even an overlap of 1.0 only proves that the normalized result sets repeated. It does not prove that the planner sent the same request or that the provider evaluated it again.

For each flagged event, compare the planner's structured constraints with the adapter's outgoing representation and the cache identity used for that call. Retain a redacted comparison rather than full customer queries. In a true planning loop, the constraints remain the same or change only cosmetically, the adapter receives the same meaning, and repeated results are expected. In the cache failure, the planner's required constraints change, but either the adapter drops one before sending or the cache treats distinct outgoing requests as the same entry. Provider request identifiers or cache disposition, when the system already exposes them, can confirm whether a fresh call occurred. Their absence leaves the cache hypothesis open, so report it as unclassified rather than asserting that the provider ignored the request.

A changed request does not guarantee changed results. The fixture needs at least one record whose eligibility actually differs under the added constraint. Without that sentinel, the same supplier ids may be the correct answer to all three searches, and the test cannot prove a cache defect. Put the expected eligible ids in the tool-contract fixture and confirm the uncached adapter returns them before testing the cache path. This keeps the oracle tied to an understood data change instead of assuming every refinement must alter the result set.

The most misleading diagnostic is a high overlap printed with many decimal places. Precision in the calculation does not add context about request meaning, caching, or result freshness. Read the event span first, then the planner constraints, adapter representation, cache evidence, and result identity contract. A long duration is also not decisive. A slow repeated response can come from the provider, while an immediate repeated response can be a legitimate cache hit for an unchanged query.

An existing suite usually breaks first at fixture construction. Old traces contain tool names and result ids but no redacted request comparison, so the new distinction cannot classify them. Do not relabel those fixtures as healthy. Extend the trace schema, update one real adapter and its test double, and make absent request evidence explicit. Re-record or carefully migrate reviewed fixtures. During that transition, keep the original repeated-result finding and add a root-cause status such as unresolved in the surrounding test report, rather than changing the detector's established finding vocabulary.

Land the changes in four steps. First, validate that request evidence and result identity are derived consistently for one task category. Second, add paired fixtures: a planner repetition and a constraint-changing cache failure with the same returned ids. Third, replay the existing corpus and review every case whose ownership changes. Fourth, enable the new triage route in shadow reports before using it to assign release failures. Threshold tuning belongs after this work. Lowering the overlap boundary cannot distinguish two causes that share the same results.

The handoff follows the first divergence. The agent team owns repeated constraints or failure to use returned state. The tool team owns argument translation and the definition of result identity. The platform or provider-integration team owns cache behavior. Observability owns duplication and event ordering, while the product owner approves the planning policy. Send the raw sanitized trace, normalized replay artifact, policy revision, event ids, per-event constraint comparison, outgoing request comparison, available cache or provider evidence, returned result ids, and expected progress condition. A screenshot of the loop badge omits every field needed to choose an owner.

This extra evidence is not free. Request comparisons can contain commercial filters or customer terms, so redaction needs a maintained per-tool schema. Aggressive redaction may erase the very constraint that separates calls. Requiring the richer comparison also leaves older traces without a root-cause classification, reducing historical coverage until fixtures are rebuilt. These costs are preferable to routing a cache outage to prompt engineers, but they need explicit owners and retention limits.

The rollout is working when the paired fixtures retain the same repeated_result_loop finding but lead reviewers to different first-divergence boundaries. The unchanged-query fixture should point to planning, while the changed-query fixture should point to argument translation or caching. A lower alert count is not proof of improvement because missing request evidence can also suppress classification. Review a sample of shadow findings and require the assigned team to reconstruct the route from the attached evidence without reopening raw customer content. Fewer ownership bounces, with unchanged positive and healthy-control verdicts, is the useful acceptance signal.

The sample rule does not catch a cycle that alternates tools. A planner can search, rank, search, rank, and return to the same state indefinitely, while no consecutive window contains one tool name. Detecting that failure requires a state-cycle rule with its own progress representation and healthy multi-tool fixtures. Do not claim that the repeated-tool detector covers it.

Roll the detector into an existing suite without flooding CI

Start by freezing the event schema. Write down which service creates event IDs, how attempts are separated, what None means, and which result identifier is stable. Add schema validation before detector evaluation. A malformed trace should fail ingestion or become unclassified, not quietly resemble a healthy run.

Build the initial corpus from reviewed incidents and healthy controls in the same task categories. Each case needs the raw trace, expected findings, expected evidence spans, and the policy revision. Keep a reason beside the expected label. When a reviewer later disputes a fixture, the team can discuss the contract rather than treating the old assertion as unquestionable truth.

Introduce the rule in shadow mode against production traces. Shadow mode means the finding is stored and reviewed but does not block a user run or fail a deployment. It does not mean “ignore alerts.” Assign an owner, capture reviewer decisions, and turn confirmed false positives into fixtures. Without that loop, the detector accumulates noise and engineers learn to dismiss it.

Keep pull-request CI deterministic. Run pure detector tests and frozen corpus checks without a live model, network call, or changing search index. The pytest temporary-path guide documents the tmp_path fixture if adapter tests need to write short-lived trace files. A test that depends on yesterday’s search ranking is an end-to-end observation, not a stable detector regression.

A minimal shell entry point can be called by any CI runner that already provides the project’s pinned Python environment:

#!/usr/bin/env bash
set -euo pipefail

mkdir -p artifacts
python -m pytest \
  tests/test_planning_detector.py \
  -q \
  --junitxml=artifacts/planning-detector.xml

Choose the CI gate carefully. A pull request should fail when a known positive fixture stops producing its required label, when a healthy control gains a prohibited label, or when evidence coordinates change unexpectedly. A change in the volume of shadow findings from live traffic should prompt review, not automatically block a release. Traffic mix, sampling, and policy adoption can change that count without a detector regression.

Version policies beside fixtures. When the overlap boundary, repeated-call count, or tool contract changes, add a new revision and state which categories adopt it. Replay old traces under both revisions during review. Once the new contract is accepted, update expected results deliberately. Mutating a shared configuration value without recording its revision makes every historical alert ambiguous.

Migrate in narrow slices. Start with one task category whose tool contract and result identity are clear. Add other categories only after they have their own healthy controls. Avoid copying the first policy globally. The policy object is small, but the business meaning behind it is not.

Redaction belongs before corpus storage. Tool arguments and answers can contain customer data. Keep the stable identifiers required for the detector, remove values that are irrelevant to the rule, and restrict access to raw evidence. A one-way hash may still be sensitive when the input space is predictable, so follow the data-handling policy rather than assuming hashing solves the problem.

Know what the fix costs and when to leave it out

Evidence-rich detection consumes storage. Result identifiers, ordered events, termination state, and policy snapshots are more data than a final answer and status. Retention and redaction work increase with that detail. The payoff is reproducibility, but a team should decide how long raw traces, normalized traces, and reviewer labels are kept instead of retaining everything indefinitely.

Pure deterministic rules are easy to replay, yet their coverage is limited. The sample detector sees repeated results, explicit forced stops, and tool-contract mismatches. It cannot decide whether a novel sequence of valid tools was strategically poor. A model-based evaluator may cover semantic questions, but it introduces prompt versioning, model variability, cost, and a second validation problem. Use that layer only when the decision cannot be expressed with observable invariants, and keep its verdict separate from deterministic findings.

Running detectors synchronously can add response latency. Most planning-quality findings are suitable for an asynchronous evaluation path after the trace is stored. A safety rule that must prevent an action belongs in execution-time policy enforcement, where it can block before the tool runs. Post-run detection can reveal a missed confirmation, but it cannot undo a money transfer or sent message.

Avoid automatic loop verdicts when result identity is unavailable. Similar search text alone is not enough to prove repeated work. Improve telemetry or classify the trace as lacking evidence. Guessing may create an attractive dashboard, but it weakens every decision based on that dashboard.

Do not use trajectory findings as proof that an agent is safe. A trace only contains observed events, and instrumentation can omit activity. These rules also say nothing about correctness of the final answer unless you add answer evaluation. Keep planning quality, answer quality, tool authorization, and runtime reliability as separate claims.

Never replace hard safety controls with post-run analysis. Confirmation before a destructive action, authorization checks, spend limits, and idempotency protections belong on the execution path. The detector can test whether the expected events were recorded and raise an incident when enforcement fails. It should not be the enforcement mechanism.

Leave subjective “best plan” judgments with reviewers when the product has not defined an observable contract. Two senior engineers can reasonably choose different research paths. Automating that disagreement before documenting acceptable behavior creates a detector that encodes one person’s taste and reports it as fact.