Stop an agent before a repeated tool call becomes a loop

Your support agent calls lookup_order for the same order, receives the same delivered status, and calls it again. The customer never gets an answer. The run eventually stops only because a global turn limit is exhausted, so the incident report says “terminated” even though the loop guard did nothing useful.

That distinction matters. A hard limit can contain cost and latency, but a QA result should tell you whether the agent recognized that it was stuck, whether the runtime stopped it, and whether any useful work happened before termination. A single pass or fail flag loses all three facts.

Recognize repeated work before the turn budget does

An agent loop usually alternates between a model decision and one or more tool executions. The model sees the updated conversation, chooses another action, and the runtime dispatches it. If the new observation does not alter the decision, the same action can recur until a separate budget ends the run. OpenAI's Agents SDK documents this loop directly and raises MaxTurnsExceeded when its configured turn budget is crossed. That behavior is a backstop, not a semantic judgment about whether a particular tool sequence was reasonable.

The useful unit for duplicate detection is an action fingerprint. At minimum, it contains the tool name and a canonical representation of the arguments. Comparing raw JSON text is unreliable because {"order_id":"A12","include_items":true} and {"include_items":true,"order_id":"A12"} express the same object. Canonicalization should parse the value, sort object keys, preserve array order, and serialize without insignificant whitespace.

Do not normalize more than the tool contract permits. Lowercasing an email address, rounding a coordinate, dropping a timestamp, or sorting an array can collapse distinct requests. The detector is allowed to know that an omitted optional field equals its documented default, but only if the tool owner has made that equivalence explicit. Generic “cleanup” code often hides a real change and creates a false loop verdict.

Consecutive repetition is the simplest pattern: A, A, A. Alternating cycles also occur: search, fetch, search, fetch. A windowed detector can find both, but the policy needs to say which patterns it considers terminal. Starting with exact consecutive repeats is usually safer because the evidence is easy to explain. Add cycle detection after production traces show a recurring pattern that matters.

Repetition alone is not enough. A status tool may return pending five times and complete on the sixth. A paginated search can call the same tool repeatedly with a different cursor. A write tool might be retried after a transport failure when the request carries an idempotency key. Each sequence contains repeated names, but only some represent stalled reasoning.

Progress therefore needs its own observable definition. For a polling tool, progress might be a changed state, a later server version, or reaching the documented terminal state before a deadline. For pagination, it is a new cursor and newly seen item identifiers. For a repair tool, it could be a smaller set of validation errors. Avoid vague signals such as output byte length. An error page and a valid response can have the same length, while harmless metadata can make two identical business results look different.

Termination reason is the third independent fact. Record values your runtime actually emits, or define a small application-owned vocabulary such as final_answer, loop_guard, turn_limit, deadline, tool_error, and cancelled. Do not infer loop_guard merely because repeated calls appear before the end. The trace needs a guard decision event or a returned application result that names that reason.

A practical score can remain deterministic. Give the run separate fields instead of an opaque decimal: whether the repetition threshold was crossed, whether progress occurred, whether the guard fired at the expected call, whether another tool executed after the guard, and whether the user received a safe terminal response. A release rule can then require all critical fields. If stakeholders insist on one number, derive it from those fields and retain the components in the report.

Build an oracle that changes when the product breaks

The detector below is application code, not an SDK API. It accepts normalized trace records, groups only consecutive calls, and asks a tool-specific function whether the latest observation represents progress. A product change that stops emitting the guard, changes the fingerprint, or permits an extra execution will change the result.

from __future__ import annotations

from dataclasses import dataclass
import hashlib
import json
from typing import Any, Callable, Iterable


@dataclass(frozen=True)
class ToolCall:
    sequence: int
    name: str
    arguments: dict[str, Any]
    observation: dict[str, Any]


@dataclass(frozen=True)
class RepeatFinding:
    fingerprint: str
    first_sequence: int
    last_sequence: int
    count: int


def fingerprint(call: ToolCall) -> str:
    canonical = json.dumps(
        {"name": call.name, "arguments": call.arguments},
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    )
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


def first_stalled_repeat(
    calls: Iterable[ToolCall],
    *,
    allowed_repeats: int,
    made_progress: Callable[[ToolCall, ToolCall], bool],
) -> RepeatFinding | None:
    if allowed_repeats < 1:
        raise ValueError("allowed_repeats must be at least 1")

    run: list[ToolCall] = []
    for call in sorted(calls, key=lambda item: item.sequence):
        if run and fingerprint(run[-1]) != fingerprint(call):
            run = []

        if run and made_progress(run[-1], call):
            run = []

        run.append(call)
        if len(run) > allowed_repeats:
            return RepeatFinding(
                fingerprint=fingerprint(call),
                first_sequence=run[0].sequence,
                last_sequence=run[-1].sequence,
                count=len(run),
            )

    return None

Notice what can make this oracle fail. If the product executes a third unchanged lookup when only two are allowed, the finding appears. If the latest observation advances the declared version, the run resets and no finding appears. If an exporter duplicates a sequence number, a separate trace-integrity check should reject the fixture before loop scoring. The detector does not quietly guess which record came first.

The tests exercise different outcomes rather than asserting facts baked into one fixture. The first mutation creates a genuine stalled call. The second proves that a higher declared version prevents the same fingerprint from being classified as stuck. The third protects pagination from a detector that groups by tool name alone.

import pytest

from loop_guard import ToolCall, first_stalled_repeat


def status_advanced(previous: ToolCall, current: ToolCall) -> bool:
    previous_version = previous.observation.get("version")
    current_version = current.observation.get("version")
    return (
        isinstance(previous_version, int)
        and not isinstance(previous_version, bool)
        and isinstance(current_version, int)
        and not isinstance(current_version, bool)
        and current_version > previous_version
    )


def call(sequence: int, arguments: dict, status: str, version: int) -> ToolCall:
    return ToolCall(
        sequence=sequence,
        name="lookup_order",
        arguments=arguments,
        observation={"status": status, "version": version},
    )


def test_flags_the_third_unchanged_lookup() -> None:
    calls = [
        call(10, {"order_id": "A12"}, "delivered", 7),
        call(20, {"order_id": "A12"}, "delivered", 7),
        call(30, {"order_id": "A12"}, "delivered", 7),
    ]

    finding = first_stalled_repeat(
        calls, allowed_repeats=2, made_progress=status_advanced
    )

    assert finding is not None
    assert (finding.first_sequence, finding.last_sequence, finding.count) == (10, 30, 3)


def test_allows_same_request_when_the_resource_version_advances() -> None:
    calls = [
        call(10, {"order_id": "A12"}, "pending", 7),
        call(20, {"order_id": "A12"}, "pending", 8),
        call(30, {"order_id": "A12"}, "delivered", 9),
    ]

    assert first_stalled_repeat(
        calls, allowed_repeats=2, made_progress=status_advanced
    ) is None


@pytest.mark.parametrize("cursor", ["page-2", "page-3"])
def test_different_pagination_cursor_is_a_different_action(cursor: str) -> None:
    calls = [
        call(10, {"query": "refund", "cursor": None}, "ok", 1),
        call(20, {"query": "refund", "cursor": cursor}, "ok", 1),
        call(30, {"query": "refund", "cursor": None}, "ok", 1),
    ]

    assert first_stalled_repeat(
        calls, allowed_repeats=1, made_progress=lambda _a, _b: False
    ) is None

The pagination test deliberately returns to the original cursor at sequence 30. It still passes because the identical calls are not consecutive. If your failure mode is an A, B, A, B cycle, write a separate cycle oracle and a fixture containing that exact sequence. Stretching a consecutive detector until it handles every pattern makes its verdict harder to trust.

Mutation tests are especially valuable for this oracle because a pleasant-looking fixture can conceal dead logic. Remove the third call from the stalled fixture and require the finding to disappear. Change only its version from 7 to 8 and expect the result to clear. Change only the arguments and expect a new fingerprint. Then simulate a product defect by moving the guard event after dispatch and require the execution-contract test to fail. Those mutations prove that the fields described in the article participate in the verdict.

Keep detector tests separate from runtime tests. Detector tests feed records directly to first_stalled_repeat and verify classification. Runtime tests use a fake tool and a scripted model boundary to verify that the dispatcher consults the detector before executing the prohibited call. Combining both layers in one test makes a failure hard to locate. A broken trace adapter, a broken fingerprint, and a broken dispatch check can all produce the same missing final answer.

A fake tool should count observable invocations outside the response it returns. For example, append each received argument object to a list owned by the test. After the scripted model requests the same action three times, assert that the list contains only the permitted two entries and that the application result names the loop guard. This assertion can fail if dispatch order regresses. Checking only the returned termination reason cannot, because a runtime could execute the third call and still label the result correctly afterward.

Use a negative control in the same suite. Script two identical status checks whose observations advance, followed by a final answer. The invocation count should reach two and the guard should remain absent. This catches the tempting implementation that increments a counter for every matching fingerprint but never resets it on progress. Another control should call the same tool for two different tenants or resources and prove that the contract-relevant identity fields keep the fingerprints distinct.

Threshold boundaries deserve exact cases. If two calls are allowed, fixtures should cover zero, one, two, and three consecutive stalled calls. The expected first blocked dispatch is call three, not call four and not call two. Off-by-one mistakes are common when one component counts completed calls while another counts proposed actions. Name variables accordingly, such as completed_identical_calls and proposed_call, instead of using a vague retry_count.

The score should also distinguish detection from containment. A detector can identify a loop correctly while the dispatcher ignores it. A dispatcher can stop after the turn limit without the repeat detector ever firing. A response formatter can then hide both failures behind a polite apology. Report at least three checkpoints: finding_created, dispatch_denied, and terminal_response_emitted. Their sequence is part of the contract. For a guarded call, the denial must precede any tool-start event carrying that call ID.

Multi-tool plans need a policy for resetting state. If the agent repeats lookup_order, calls read_refund_policy, and then returns to the same unchanged lookup, the intermediate policy call may represent a legitimate attempt to gather missing context. Resetting the consecutive counter is defensible. A broader cycle detector might still flag the overall trajectory after several passes. Choose based on the harm you are controlling, and preserve both tool fingerprints so a reviewer can see why the broader rule fired.

Parallel calls need another boundary. Two identical reads dispatched in the same model turn may be deduplication waste, but they are not a reasoning loop across turns. Group calls by decision or turn identifier before applying a sequential repeat rule. Test parallel duplication separately at the dispatcher, where it can collapse safe reads or reject duplicate writes. Treating concurrent siblings as a temporal sequence invents an order that the agent never observed.

Finally, make incomplete traces unscorable. If a tool-start event has no result because collection ended early, the detector cannot decide whether progress occurred. Returning a clean score rewards missing evidence. Emit an explicit incomplete_trace outcome, keep it out of the pass denominator, and investigate collection health. A deliberate fail-closed release policy may block on that outcome, but the report should still say that evidence was missing rather than claiming a confirmed loop.

Follow three traces that look similar at first

Consider an order assistant that receives a successful, complete tool response. The first call returns {"status":"delivered","version":7}. The next model turn requests the same order with the same options. The response is unchanged. A third request follows immediately. There is no user message between them, no tool error, and no new state.

The key evidence is not that lookup_order appears three times. It is that the canonical arguments match, the business version stays at 7, and a final state was already available after the first call. If the configured allowance is two, the guard should stop before dispatching call three. A trace that records call three and then records a guard event failed the execution-side contract even if the final response apologizes correctly. Detection after the side effect is too late for write tools and wasteful for read tools.

Now change the tool to get_export_status. The same arguments return queued, then running, then complete. The fingerprints match, but the observations advance through allowed states. This is not stalled work. It still needs a deadline and polling interval because progress can stop later. The loop guard can keep one counter for unchanged observations and another for total polls. That design distinguishes “the export stayed running too long” from “the agent kept asking despite no change.”

A third trace uses search_cases with cursor=null, then cursor=eyJwYWdlIjoyfQ, then a final answer. A detector that hashes only the tool name reports a duplicate. A detector that drops cursor values during normalization reports the same false positive. The contract says the cursor selects a different result page, so it belongs in the fingerprint. The evidence that clears this trace is a changed argument plus a response containing previously unseen case identifiers.

Write tools require a stricter reading. Suppose issue_refund times out at the client after the server accepts the request. The agent retries with the same amount and order. That may be a correct recovery only when the integration supports an idempotency mechanism and the retry reuses the same key. A different idempotency key can create a second refund. A loop scorer should not grant “progress” because the second response says success. It should check the write contract, correlate both attempts, and surface an indeterminate outcome if the first result cannot be established.

The scoring report should make these cases visibly different. For the order lookup, record stalled_repeat=true, guard_before_dispatch=false, and termination_reason=loop_guard. For export polling, record stalled_repeat=false, poll_budget_exhausted=false, and terminal_state=complete. For pagination, record distinct_action_count=2 and new_items_observed=true. Those field names are application-owned examples, not properties promised by an agent framework. Pick names that match the events your service truly emits.

Illustrative score weights can help teams discuss policy, but do not present them as measured effectiveness. One reasonable release rule is categorical: no side-effecting call may execute after a repeat guard decision; any run stopped by a turn limit because of unchanged calls fails; expected polling must reach a terminal state or an explicit deadline response. A numeric dashboard can sit on top of that rule, but it must not average a duplicate payment with several clean informational runs.

Tell a reasoning loop from bad telemetry

Duplicate trace records can imitate duplicate execution. Batch exporters retry, consumers process messages at least once, and two collectors can ingest the same span. Before scoring agent behavior, validate trace identity. The same span identifier appearing twice with identical content is usually a telemetry duplication problem. Two different span identifiers tied to two actual tool invocations are evidence of repeated execution. If the instrumentation lacks a stable call or span identifier, the QA team cannot confidently separate them.

Out-of-order delivery is another near-miss. Export time is not execution order. Use the runtime's sequence field, parent relationships, or request and result correlation identifiers when those are available and documented by your system. Do not sort solely by the timestamp at which the analytics pipeline received the record. Clock skew deserves its own validation because timestamps from separate processes can move backward without changing causal order.

Tool failures also need classification. An agent that calls a tool, receives a transient transport error, waits according to policy, and retries once is not the same as an agent that ignores a validation error and repeats the invalid arguments. Save the error category and retry metadata. Avoid parsing human error text when the adapter can emit a structured category. If the integration exposes only text, keep the matcher narrow and treat unknown messages as unknown rather than assigning them to “transient.”

Argument redaction can destroy the fingerprint. Replacing every order ID with [REDACTED] makes different customer requests look identical. Hashing the sensitive value with a keyed, environment-specific function can preserve equality without storing the raw identifier. The security team should approve that approach because even stable hashes can enable correlation. Another option is to compute the fingerprint inside the trusted service and export only the digest.

Caching can create the opposite confusion. Two calls with different request IDs may return the same cached observation. Identical output does not prove identical action. Score action repetition from tool name and contract-relevant arguments, then use observation equivalence only for progress. Keeping those roles separate prevents a cache hit from being labeled a loop by itself.

Finally, a model may repeat its explanation while making no tool call. That is a response-quality problem, not a repeated tool-call problem. Likewise, a workflow engine may intentionally fan out the same read to several tenants. The parent task or tenant identifier makes those actions distinct even when their business arguments match. Test at the boundary where the product defines uniqueness.

A small diagnostic script helps reviewers see exactly what the scorer saw. This one reads newline-delimited JSON exported by your own trace adapter and prints sequence, call ID, tool name, and the canonical digest. It fails loudly when required fields are absent instead of silently producing a partial score.

from __future__ import annotations

import argparse
import json
from pathlib import Path

from loop_guard import ToolCall, fingerprint


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("traces", nargs="+", type=Path)
    args = parser.parse_args()

    for trace_path in args.traces:
        rows = []
        for line_number, line in enumerate(trace_path.read_text().splitlines(), start=1):
            if not line.strip():
                continue
            raw = json.loads(line)
            for field in ("sequence", "call_id", "tool", "arguments", "observation"):
                if field not in raw:
                    raise ValueError(
                        f"{trace_path}:{line_number}: missing {field}"
                    )
            call = ToolCall(
                sequence=int(raw["sequence"]),
                name=str(raw["tool"]),
                arguments=dict(raw["arguments"]),
                observation=dict(raw["observation"]),
            )
            rows.append(
                (call.sequence, raw["call_id"], call.name, fingerprint(call))
            )

        print(f"TRACE {trace_path}")
        for sequence, call_id, name, digest in sorted(rows):
            print(f"{sequence:06d} {call_id} {name} {digest[:16]}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

For a real loop, the output shows different call IDs beside the same digest. For duplicate telemetry, it often shows the same call ID twice. For pagination, the digests differ. That compact view does not replace the full trace, but it tells the investigator which branch to pursue before reading model messages.

Put the guard and the score into an existing suite

Start in observe-only mode. Compute fingerprints and findings, but do not terminate runs. Review a sample from each high-volume tool with its owner. This is where you discover contract details such as meaningful cursors, safe retry keys, expected poll counts, and fields that must never enter telemetry. A global threshold chosen before this review will either miss expensive loops or break legitimate workflows.

Next, enable active termination for read-only tools with clear progress rules. Record both the decision sequence and whether dispatch occurred afterward. Keep the runtime turn limit in place. The guard should normally fire earlier, while the turn limit catches patterns the specific detector does not understand.

Move side-effecting tools only after their idempotency and outcome-reconciliation paths are tested. The safest response to an uncertain write is often escalation, not an automatic retry and not a claim that the action failed. Include fixtures for success, explicit rejection, timeout before dispatch, timeout after possible dispatch, and duplicate request keys.

CI should use committed synthetic traces because live model runs are variable and can be expensive. A separate scheduled evaluation can exercise real agents and store reviewed artifacts. The deterministic suite verifies the scorer and guard contract on every change; the scheduled job looks for new production-like patterns.

name: agent-loop-contract

on:
  pull_request:
    paths:
      - "agent_runtime/**"
      - "trace_scoring/**"
      - "tests/loop_guard/**"

jobs:
  score-fixtures:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements-test.txt
      - run: python -m pip install -r requirements-test.txt
      - run: python -m pytest tests/loop_guard -q
      - run: python -m trace_scoring.audit tests/fixtures/traces/*.jsonl

Note the cache-dependency-path line. With cache: "pip" enabled and that key left out, actions/setup-python searches for its default **/requirements.txt, falling back to **/pyproject.toml. A loop-guard project whose runtime is stdlib-only and whose test pins live in requirements-test.txt provides neither, so the setup step errors on the missing dependency file and the job dies before any fixture is scored. Naming the file that the very next step installs from keeps the cache key derived from the same input the job actually uses.

The cost is real. Fingerprinting adds code and telemetry. Progress rules create per-tool maintenance. Active guards can stop a slow but valid workflow. More detailed traces increase storage and privacy review. Scheduled model evaluations consume time and money. State those costs in the rollout proposal so teams do not quietly weaken the checks when the first false positive appears.

Version the policy with the test fixture. When a tool contract changes, reviewers should see the threshold or progress-rule change beside a trace that proves why it is needed. Do not update a dashboard threshold in isolation. That makes historical scores impossible to interpret and lets a regression disappear without a code review.

Monitor guard outcomes after release. A rising count can mean the model changed, the tool started returning stale data, an observation field stopped updating, or the detector became too aggressive. The finding is a lead, not automatic proof that the model is at fault. Route it with the normalized evidence and the owning tool version.

Know when this control should stay out of the path

Do not terminate solely on repeated calls when the workflow contract explicitly requires them and no stronger progress signal exists. Long-running job polling is the obvious example. Add a deadline, an interval, and a maximum poll budget first. Until those exist, a repeat threshold is an arbitrary availability risk.

Avoid argument fingerprinting for payloads that cannot be handled safely. Raw medical notes, access tokens, private documents, and large binary inputs should not be copied into a scoring pipeline. Compute a minimal approved digest inside the trusted boundary or rely on non-sensitive correlation fields. OpenAI's tracing documentation notes that model and function spans can contain sensitive inputs and outputs, so trace capture settings deserve an explicit security decision.

Do not use this score to judge whether the final answer is factually correct. A run can make one perfectly chosen tool call and still misread the result. Another can repeat a safe lookup once, notice a delayed update, and answer correctly. Tool-loop control and answer-quality evaluation cover different failure surfaces.

Do not let a global numeric score authorize side effects. Averages are useful for trend reporting, but a duplicated refund or message send is a discrete contract violation. Preserve hard gates for irreversible actions, and use the score to explain the path to the gate.

Finally, skip active blocking while trace integrity is unknown. Missing spans, duplicate exports, or unreliable ordering can make the guard itself unsafe. Fix instrumentation first, replay known traces through observe-only scoring, and enable termination only when the evidence distinguishes an agent loop from a telemetry loop.