Catch the retry loop before one failed tool floods your backend

An inventory tool times out, and the agent hits the backend three times before returning an error. The dashboard shows one tool call, the API logs show three requests, and nobody can say whether the third request was expected. That gap is where a retry setting becomes a production incident.

Count the attempt the dashboard hides

One tool call is not always one execution. LangChain's ToolRetryMiddleware wraps the handler that executes a selected tool. When that handler raises an exception accepted by retry_on, the middleware can call the handler again. The model does not need to make another decision between those attempts. From the agent transcript, the sequence can still look like one tool request followed by one tool result.

The first counting trap sits in the name max_retries. It is the number of retries after the initial call, not the total number of calls. A value of 2 therefore permits at most three handler executions for one matching tool call. A test that expects two total attempts has encoded the wrong contract. A dashboard that increments only when the model emits a tool call is also answering a different question from the backend's request counter.

That distinction matters whenever the tool charges money, consumes a rate limit, takes a lock, or performs a write. Even a read-only lookup can amplify an outage. If 100 logical calls all reach their three-attempt ceiling, the dependency can receive as many as 300 handler executions. That is arithmetic from the configured ceiling, not a claim about measured traffic. Real traffic will depend on how often earlier attempts succeed and whether requests reach the dependency at all.

Four settings decide the path. The optional tools list decides whether the middleware applies to this tool name. The retry_on tuple or callable decides whether a caught exception is eligible. The max_retries value sets how many eligible retries follow the initial execution. Finally, on_failure decides what happens after the eligible attempts have all failed. The LangChain middleware documentation documents those controls and their defaults.

Failure handling changes what the caller can observe. With on_failure set to error, LangChain re-raises the last exception after exhaustion. With continue, it returns a ToolMessage whose status is error, allowing the model to react. A custom failure function supplies the message content. A suite that only uses pytest.raises covers the first policy and can completely miss a broken second policy. Conversely, a suite that treats every returned message as success will turn exhausted retries into a green run.

Backoff affects duration, not the number of allowed handler calls. LangChain calculates delays from initial_delay, backoff_factor, max_delay, and jitter. With jitter disabled, an initial delay of 0.25 seconds, a factor of 2, and two retries adds scheduled waits of 0.25 and 0.5 seconds before the second and third attempts. That 0.75-second sum excludes the time spent inside the tool. It is a configuration calculation, not a benchmark. Jitter is enabled by default and intentionally makes exact wall-clock assertions unreliable.

The budget also has a boundary that is easy to overlook. It belongs to one logical tool call. If exhaustion produces an error message and the model chooses the same tool again, the new call gets its own retry cycle. ToolRetryMiddleware does not promise a total backend-attempt ceiling for the whole agent run. LangChain provides separate tool-call limiting middleware for run or thread limits, but a logical-call limit and a handler-attempt limit still measure different events.

For QA, the useful model is a short event chain: model-issued tool call, initial handler execution, zero or more eligible retries, then success, an error message, or a raised exception. Capture the tool call ID across that chain. It lets you distinguish three attempts for one request from three separate requests created by the model. Without that identifier, an exhausted retry budget and an agent loop can produce nearly identical API logs.

Prove exhaustion with an oracle that can fail

The smallest reliable test calls the middleware wrapper directly with a deterministic handler. No model, network, or real sleep is needed. The handler records every invocation and always raises a retryable exception. The assertion then checks both terminal behavior and the full attempt list.

The following test uses public types from LangChain and LangGraph. Setting initial_delay to zero keeps the contract test fast. Setting jitter to false removes randomness, although jitter has no effect when the base delay is zero. The production delay policy belongs in a separate configuration check.

from typing import Any, cast

import pytest
from langchain.agents.middleware import ToolRetryMiddleware
from langchain_core.messages import ToolCall, ToolMessage
from langgraph.prebuilt.tool_node import ToolCallRequest


def make_request(
    *,
    name: str = "inventory_lookup",
    call_id: str = "call-17",
) -> ToolCallRequest:
    return ToolCallRequest(
        tool_call=ToolCall(
            name=name,
            args={"sku": "A-17"},
            id=call_id,
        ),
        tool=None,
        state={},
        runtime=cast(Any, None),
    )


def test_exhaustion_stops_after_initial_call_plus_two_retries() -> None:
    middleware = ToolRetryMiddleware(
        max_retries=2,
        retry_on=(TimeoutError,),
        on_failure="error",
        initial_delay=0.0,
        jitter=False,
    )
    seen_call_ids: list[str | None] = []

    def timed_out(request: ToolCallRequest) -> ToolMessage:
        seen_call_ids.append(request.tool_call["id"])
        raise TimeoutError("inventory dependency timed out")

    with pytest.raises(
        TimeoutError,
        match="inventory dependency timed out",
    ):
        middleware.wrap_tool_call(make_request(), timed_out)

    assert seen_call_ids == ["call-17", "call-17", "call-17"]


def test_success_on_third_attempt_stops_the_loop() -> None:
    middleware = ToolRetryMiddleware(
        max_retries=3,
        retry_on=(TimeoutError,),
        on_failure="error",
        initial_delay=0.0,
        jitter=False,
    )
    attempts = 0

    def eventually_succeeds(request: ToolCallRequest) -> ToolMessage:
        nonlocal attempts
        attempts += 1
        if attempts < 3:
            raise TimeoutError("temporary timeout")
        return ToolMessage(
            content="17 units available",
            tool_call_id=request.tool_call["id"],
            name=request.tool_call["name"],
        )

    result = middleware.wrap_tool_call(make_request(), eventually_succeeds)

    assert attempts == 3
    assert isinstance(result, ToolMessage)
    assert result.content == "17 units available"

The first oracle can fail in several meaningful ways. A framework or configuration regression that makes four calls adds a fourth ID to the list. A filter mistake that disables retries leaves one ID. An off-by-one implementation that interprets the setting as total attempts produces two. If the middleware swallows the exception despite on_failure="error", the expected-exception assertion fails before the count is even checked. None of those branches is dead code.

The success case protects a different edge. A retry loop must stop immediately when the handler returns normally. Configuring three retries means a fourth attempt is available, but it does not mean all four executions must happen. Returning success on the third execution and asserting a count of three catches middleware or wrapper code that continues after it already has a usable result.

Keep the call ID assertion. Counting with a bare integer proves quantity but not identity. A handler that accidentally receives a reconstructed request with a new call ID on each attempt could still increment the integer three times. The repeated ID demonstrates that this test observed retries of the same logical request. It does not prove that a downstream HTTP client propagated that ID. If the dependency needs it for idempotency or tracing, add an assertion at the adapter boundary where the request headers are built.

Direct wrapper tests have a deliberate trade-off. They are quick and make failures local, but they bypass model selection, agent graph assembly, and middleware composition. One integration test should still create the agent with the same middleware list used by production and a deterministic fake model that emits a known tool call. Do not use a live model for the call-count oracle. A model can choose not to call the tool, call it twice, or change arguments, and those valid variations turn a deterministic retry contract into a probabilistic test.

A team adopting these checks in an existing suite should start with the always-fail case. It reveals the current interpretation of max_retries without depending on recovery timing. Add the succeeds-on-last-allowed-attempt row next. Then add a succeeds-one-attempt-too-late row and verify that the late success is never reached. Those three boundaries expose an off-by-one error more clearly than dozens of random failure sequences.

The late-success fixture deserves careful construction. For max_retries=2, make the handler succeed only on its fourth invocation. The correct middleware never gets there, so the expected result remains exhaustion after three calls. If a future refactor makes the test return success, that is direct evidence of one extra execution. Avoid an assertion against a hard-coded fixture that already contains the expected answer. The mutable handler state is part of the system under test, and a changed loop count changes the observed result.

Separate a spent budget from a bad exception filter

Two failures can print the same timeout line on the test runner while exercising entirely different retry paths. In the first, the timeout class is eligible and the middleware spends every retry. In the second, retry_on rejects the concrete exception class and LangChain re-raises it after the first handler call. Looking only at the final exception text cannot tell them apart.

Parameterize the exception type and assert the count associated with each policy branch. This example treats timeouts as transient and validation errors as permanent. Both errors propagate because on_failure is error, but only one should consume the budget.

@pytest.mark.parametrize(
    ("error_type", "message", "expected_attempts"),
    [
        (TimeoutError, "dependency timed out", 3),
        (ValueError, "sku must not be blank", 1),
    ],
    ids=["retryable-timeout", "non-retryable-validation"],
)
def test_retry_filter_controls_budget_use(
    error_type: type[Exception],
    message: str,
    expected_attempts: int,
) -> None:
    middleware = ToolRetryMiddleware(
        max_retries=2,
        retry_on=(TimeoutError,),
        on_failure="error",
        initial_delay=0.0,
        jitter=False,
    )
    attempts = 0

    def fails(request: ToolCallRequest) -> ToolMessage:
        nonlocal attempts
        attempts += 1
        raise error_type(message)

    with pytest.raises(error_type, match=message):
        middleware.wrap_tool_call(make_request(), fails)

    assert attempts == expected_attempts

This is more than a convenient parameter table. It defines the team's error taxonomy in executable form. A blank SKU will not become valid because the same request is sent again. Retrying it wastes latency and can obscure a product defect in argument generation. A dependency timeout may clear on another attempt, so the policy permits retries. Add rows for the exception classes your adapter actually raises, not the names an upstream API uses in its documentation.

Exception translation is a common source of near-misses. An HTTP client might raise its own timeout class, while the tool adapter catches it and raises RuntimeError. A retry_on=(TimeoutError,) policy will see only the translated exception and make one attempt. The production log may still contain the word "timeout," which sends investigators toward an off-by-one theory. Record the concrete exception class at the middleware boundary, not just the message supplied by a remote service.

The tools filter creates a similar single-attempt path. LangChain accepts tool names or tool instances in that setting. A renamed tool, a wrapper that changes the registered name, or a simple typo can bypass retry logic. Evidence should include both the name in the model's tool call and the name resolved for execution. The most useful negative test calls a second tool that is intentionally outside the filter and proves its handler runs exactly once.

Terminal behavior needs its own branch because continue is intentionally not an exception path. After the allowed attempts fail, LangChain returns an error ToolMessage. The model may use that result to apologize, choose a fallback, or issue another call. Assert the message as an error result, not merely as an object that exists.

def test_continue_returns_an_error_tool_message_after_exhaustion() -> None:
    middleware = ToolRetryMiddleware(
        max_retries=1,
        retry_on=(ConnectionError,),
        on_failure=lambda error: (
            f"DEPENDENCY_UNAVAILABLE:{type(error).__name__}"
        ),
        initial_delay=0.0,
        jitter=False,
    )
    attempts = 0

    def unavailable(request: ToolCallRequest) -> ToolMessage:
        nonlocal attempts
        attempts += 1
        raise ConnectionError("internal host name must stay out of the prompt")

    result = middleware.wrap_tool_call(make_request(), unavailable)

    assert attempts == 2
    assert isinstance(result, ToolMessage)
    assert result.status == "error"
    assert result.name == "inventory_lookup"
    assert result.tool_call_id == "call-17"
    assert result.content == "DEPENDENCY_UNAVAILABLE:ConnectionError"

The custom formatter makes the assertion stable and prevents an internal hostname from being sent back to the model. It also creates a cost: operators lose the raw exception detail in the conversational result. Keep the detailed exception in access-controlled telemetry, linked by tool call ID, while exposing only the safe classification to the agent. Do not solve the security problem by removing the ID that joins those records.

When the list assertion fails, run the single test with verbose output rather than the full agent suite. Pytest shows the parameter ID and the differing values, which immediately distinguishes three expected calls from one observed call. The exact line wrapping varies by pytest version, so do not snapshot the whole console rendering.

python -m pytest tests/contracts/test_tool_retry_budget.py::test_retry_filter_controls_budget_use -vv

For an exhaustion test with on_failure="continue", LangChain's default message includes the tool name, the number of attempts, and the exception type and text. Those fields are useful during a manual investigation, but a test tied to the entire English sentence is brittle. Assert the structured ToolMessage fields and the count you recorded at the handler. If product requirements depend on wording, supply a custom formatter and test the application-owned string instead.

Trace the near-misses that look like exhaustion

The hardest production case is not an incorrect max_retries value. It is a second logical tool call that starts immediately after the first one returns an error message. API logs show six requests. The retry policy allows three attempts per call. Without tool call IDs, the sequence looks like one call that ignored its ceiling.

Inspect the agent message history or trace before changing the retry setting. One tool-call ID repeated three times at the adapter boundary points to a single exhausted cycle. Two IDs, each repeated three times, point to two logical calls chosen by the model. A third shape, three IDs appearing once each, suggests retries were bypassed and the model independently repeated the request. All three can produce the same dependency exception and the same total request count.

ToolRetryMiddleware and ToolCallLimitMiddleware address different layers. The first retries an exception while handling one selected tool call. The second can limit tool calls within a run or thread. Pairing them is reasonable when an agent may react to error messages by selecting the same tool again. It still does not replace backend-attempt telemetry because an internal retry is not a fresh model-issued tool call.

This configuration retries only two transient exception classes for the inventory tool, returns a safe error result after exhaustion, and stops the run if the model exceeds four logical inventory calls. The model and tool are injected so the example does not require a provider credential just to import the policy.

from collections.abc import Sequence
from typing import Any

from langchain.agents import create_agent
from langchain.agents.middleware import (
    ToolCallLimitMiddleware,
    ToolRetryMiddleware,
)
from langchain_core.tools import BaseTool


retry_transient_inventory_failures = ToolRetryMiddleware(
    max_retries=2,
    tools=["inventory_lookup"],
    retry_on=(TimeoutError, ConnectionError),
    on_failure=lambda error: (
        f"Inventory lookup unavailable ({type(error).__name__})"
    ),
    initial_delay=0.25,
    backoff_factor=2.0,
    max_delay=2.0,
    jitter=True,
)

limit_inventory_calls_per_run = ToolCallLimitMiddleware(
    tool_name="inventory_lookup",
    run_limit=4,
    exit_behavior="error",
)


def build_agent(model: Any, tools: Sequence[BaseTool]) -> Any:
    return create_agent(
        model=model,
        tools=list(tools),
        middleware=[
            retry_transient_inventory_failures,
            limit_inventory_calls_per_run,
        ],
    )

Do not read the values as universal recommendations. Two retries may be too many for a slow payment service and too few for a cheap eventually consistent read. A four-call run limit may conflict with a workflow that legitimately checks four warehouses. The important part is that the two policies are named separately and can be tested against separate evidence.

Side effects create another near-miss. Suppose a tool sends a shipment request. The server commits the shipment, but the connection drops before the client receives the response. From the handler's point of view, the attempt raised a connection error and is eligible for retry. From the business system's point of view, it already succeeded. A second attempt can create a duplicate shipment even though the middleware obeyed its budget perfectly.

The evidence that separates this case is an idempotency record at the dependency, not the exception count. Use a stable operation key for every retry of the same logical action, and verify that the dependency stores or recognizes it. LangChain does not automatically make an arbitrary external API idempotent. The tool adapter and target service own that contract. The call ID can be an input to the design, but only if the adapter explicitly propagates a stable key and the service enforces it.

Timeout placement matters too. A client-side timeout before a socket is opened is different from a timeout while waiting for a response after bytes were sent. Logs that say only TimeoutError discard the evidence needed to judge retry safety. Capture whether the request was dispatched, whether a response status arrived, the tool call ID, the attempt ordinal, and the dependency's request ID when available. Redact credentials and sensitive arguments before storage.

Do not infer attempt ordinals from timestamps. Concurrent tool calls can interleave, clocks can differ across services, and jitter intentionally changes spacing. Increment the ordinal in the adapter invocation path and attach it to the same logical call ID. A sequence like call-17 attempt 1, call-22 attempt 1, call-17 attempt 2 is then unambiguous even if sorted logs look confusing.

An agent trace should also reveal the terminal object. If continue is configured, look for a ToolMessage with error status and the original tool call ID. If error is configured, expect the exception to leave the middleware instead. If neither appears but the handler count reaches the ceiling, another middleware may have transformed or swallowed the outcome. That is a composition problem, not retry exhaustion. Reproduce it once with only ToolRetryMiddleware, then restore neighboring middleware one at a time.

Roll the contract into CI without waiting on real outages

Existing suites usually start with end-to-end agent tests that hit a sandbox dependency. Those tests are the wrong place to discover an off-by-one retry. They are slower, the sandbox may recover halfway through the sequence, and a live model may choose a different tool. Move the counting oracle into a contract-test module first, while leaving one deterministic integration path to prove the middleware is wired into the built agent.

The rollout can happen without changing production behavior. Record the current policy values and package version. Add the always-fail, recover-on-last-allowed-attempt, non-retryable exception, excluded tool, and continue result tests. Run them against the current implementation. If they reveal that the team misunderstood the existing contract, treat that as a policy decision rather than editing the expected count until the test turns green.

Next, compare production telemetry with the new event vocabulary. Add separate counters for logical tool calls, handler attempts, exhausted cycles, immediate non-retryable failures, and error ToolMessages. Keep tool call ID and attempt ordinal in diagnostic events, but avoid raw arguments by default. This stage can run in observation-only mode. It exposes which tools would be affected before a narrower retry_on filter or lower budget is released.

Change one tool family at a time. Read-only, low-cost tools are a safer first group than payment, email, ticket creation, or database mutation tools. Use the tools filter to make that scope explicit. After the first release, compare exhaustion and recovery paths by exception class. A high recovery count can justify retries. A high immediate success rate with rare expensive exhaustion may justify a smaller budget. Those decisions require actual telemetry; do not paste illustrative percentages into a release threshold.

CI should pin the LangChain version through the project's dependency file and run the contract module without provider secrets. A separate scheduled or pre-release job may run the deterministic agent integration test. Keeping the contract test in every pull request gives fast feedback when someone changes max_retries, retry_on, tools, or on_failure.

name: Retry contract

on:
  pull_request:
    paths:
      - "src/agents/**"
      - "tests/contracts/test_tool_retry_budget.py"
      - "requirements-dev.txt"

jobs:
  tool-retry-budget:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install pinned test dependencies
        run: python -m pip install -r requirements-dev.txt
      - name: Verify retry boundaries
        run: python -m pytest -q tests/contracts/test_tool_retry_budget.py

The workflow assumes requirements-dev.txt exists and pins compatible LangChain, LangGraph, and pytest versions. Do not replace that file with unpinned installs in the workflow. Middleware behavior and import locations can change between releases, and a surprise dependency upgrade should not masquerade as a product regression.

Avoid wall-clock gates in the pull-request job. A test such as "completed in less than one second" is vulnerable to runner load. A test that sleeps through production backoff makes each negative case needlessly expensive. Zero-delay wrapper tests prove call count. Configuration assertions prove the intended delay values are present. If the sleep algorithm itself is critical, isolate it in one version-specific test and accept that it couples the suite more closely to framework internals.

The pytest assertion guidance supports checking expected exceptions without hiding the assertions that follow. Put the attempt count after the raises context, as in the first example. If the count sits inside the context after the call that should raise, it is unreachable and can never catch an extra retry. This small review detail prevents a surprisingly common false oracle.

Failure output should tell the engineer which layer broke. Include the selected tool name, tool call ID, concrete exception class, expected attempt ceiling, observed attempt count, and terminal mode. Do not dump an entire trace into the assertion message. Save the full sanitized trace as a CI artifact only when the integration test fails, and keep the short contract-test output readable in the job log.

There is a real maintenance cost. Direct calls to wrap_tool_call give precise, fast coverage but sit closer to framework mechanics than a user-level agent invocation. A LangChain upgrade may require adjusting the request fixture even when product behavior is unchanged. The integration check has the opposite profile: it survives some internal refactors but involves more components and produces less local failures. Keep both layers small and give each one a specific job.

Release gating should be based on contract violations, not on the existence of any transient failure. Block when a matching tool exceeds its configured attempt count, retries a prohibited exception, changes terminal mode unexpectedly, or loses the stable call identity required by the adapter. A correctly exhausted and correctly classified dependency failure is evidence that the guard worked. It may still affect product availability, but it is not automatically a retry implementation defect.

Know when retries make the system less safe

Validation failures are the clearest no-retry case. Missing required arguments, an unsupported enum, a malformed date, or a domain rule violation will not repair itself while the identical request is repeated. Let that exception propagate or convert it into a message that lets the model correct the input. Spending the retry budget only adds latency and can hit the same validation logging pipeline several times.

Authentication and authorization failures are usually permanent for the life of one tool call. Repeating a request with the same expired token or forbidden identity does not grant access. A separate credential-refresh path can be valid, but that is a state transition with its own tests, not a reason to classify every permission error as transient. Retrying authorization failures can also trigger security alerts or account lockouts.

Non-idempotent writes need stronger protection than a low retry count. A budget of one retry still permits two charges, two emails, or two ticket creations. If the downstream operation cannot deduplicate a stable key, do not automatically retry an ambiguous failure after dispatch. Route it to reconciliation or human review. The operational cost is a slower recovery path, but that is preferable when duplication is more harmful than temporary unavailability.

Capacity failures require coordination with the dependency. Immediate retries against an overloaded service can deepen the outage. Exponential backoff and jitter reduce synchronized bursts, but they do not create capacity. Respect a documented server retry signal when the client exposes it, cap the wait inside the request's overall deadline, and consider a circuit breaker outside the agent. A middleware retry loop should not outlive the user request that triggered it.

Long-running tools may have no useful time left for another attempt. If the request budget is five seconds and the first attempt consumes four, a configured retry that begins after backoff is unlikely to complete. ToolRetryMiddleware's count does not know an application's end-to-end latency target unless the application encodes that context elsewhere. A deadline-aware adapter may need to reject a retry even though its exception type is normally transient.

Tests should not enable retries merely to hide a flaky fake. A fixture that randomly fails and then passes exercises nondeterminism, not the framework's boundary. Use a scripted handler that fails a known number of times. If a test passes only because retries are active, report the original failure and decide whether it represents the intended transient class. Otherwise the suite starts calling instability resilience.

Do not apply the middleware to every tool by default just because tools=None is supported. Selection should follow the behavior of each dependency. A weather lookup and a fund transfer have different retry safety. Explicit filters cost maintenance when tools are renamed or added, but that cost forces a useful review. Add a contract test that enumerates registered tool names and the expected retry policy so a new side-effecting tool does not silently inherit a broad exception rule.

The continue terminal mode is also not universally safer than raising. Returning an error ToolMessage gives the model a chance to recover, but it may choose the same tool again, switch to a more expensive fallback, or present a confident answer without data. Raising stops the run but may produce a harsher user experience. Test the agent's reaction to the error message separately from the middleware's attempt count. The retry contract can be correct while the recovery conversation is wrong.

Broad exception matching trades availability for correctness risk. The documented default accepts Exception, which is convenient for transient network failures but can also retry programming errors and invalid input. Narrowing retry_on costs coverage when a dependency introduces a new transient exception class. Manage that cost with telemetry for immediate non-retryable failures and a deliberate review of adapter upgrades. Do not respond by restoring a catch-all without checking what it would repeat.

Finally, avoid using handler attempts as a billing oracle unless the tool's charging rules make that relationship explicit. A failed request may be free, partially billed, or fully billed. A cache can satisfy a handler without contacting the paid service. Count attempts to enforce the software contract, then reconcile charges from provider records designed for billing. Conflating those quantities creates a test that is precise about the wrong thing.

Before approving automatic retries for a tool, ask four concrete questions. Can the same inputs succeed without another state change? Can the operation be repeated without duplicate harm? Does another attempt fit inside the remaining deadline? Will the terminal result be handled safely by the agent? A "no" on any one of them is a reason to narrow the policy, add idempotency or reconciliation, or skip retries entirely.