Test LangChain Tool-Call Error Sequences

LangChain fake model tool call error testing uses GenericFakeChatModel to emit a deterministic sequence of AI messages, tool calls, and final text without a provider API. Pair that sequence with controlled tool failures to verify call IDs, error observations, revised arguments, retry decisions, recovery, and bounded stop paths.

LangChain's official unit-testing guide positions fake chat models as unit-test tools for scripted responses. The LangChain, tool-use, and trajectory challenges in src/db/seed-data/challenges/expansion-ai-agents.ts, src/db/seed-data/challenges/expansion-mcp-ai.ts, and src/lib/roadmaps.ts provide the repository context: agent quality depends on the chosen tool, valid arguments, controlled failure handling, and evidence across the complete sequence.

What is LangChain fake model tool call error testing?

It is a unit-test technique for driving an agent or message loop through a known multi-turn path. The fake model supplies the next AIMessage each time the application invokes it. A controlled tool supplies success or failure. The test then inspects every model message, tool invocation, ToolMessage, and terminal result.

The technique answers questions that a final-string assertion cannot:

• Did the model request the intended tool?
• Were arguments valid and did a retry revise the rejected field?
• Did each tool result refer to the correct tool call ID?
• Was the error observation sanitized before re-entering model context?
• Did a non-retryable error stop further calls?
• Did a successful retry lead to one terminal response?
• Did the loop remain inside its configured step and attempt bounds?

This is narrower than the broad LangChain testing guide. It owns deterministic tool-error sequences, not every chain, retriever, callback, or provider integration. It also differs from testing LangChain middleware retries and fallbacks, which focuses on middleware and model-level retry behavior. Here the agent observes a tool failure and chooses what to do next.

A fake response is not a model evaluation. It proves orchestration behavior given a scripted model decision. Keep claims at that boundary.

Why use a scripted model instead of a live provider?

A scripted model gives the test exact control over sequence and removes API credentials, provider availability, sampling variation, and model drift from the unit-test result. It can force rare paths such as a malformed first call, a domain rejection, a corrected retry, or an extra call after success.

That control improves diagnosis. If the expected second ToolCall never appears, the defect is in the message loop, fixture, middleware, or agent configuration under test. The result is not ambiguous because a provider made a different but reasonable choice.

Do not choose only one layer. The fake provides a stable contract test. A small live integration suite checks that the selected provider still accepts bound tools and returns messages the adapter can convert. Captured trajectories can then be evaluated with agent tool-call trace grading.

The testing pyramid is about evidence ownership, not speed claims. Run the fake where the requirement is deterministic sequence handling. Run the provider where the requirement is provider compatibility or actual model behavior.

How do GenericFakeChatModel sequences work?

Import GenericFakeChatModel from langchain_core.language_models.fake_chat_models. Construct it with an iterator of strings or AIMessage objects. Each invocation consumes the next item. For tool calls, use AIMessage with ToolCall entries containing name, arguments, and an ID.

from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage
from langchain_core.messages.tool import ToolCall

responses = iter(
    [
        AIMessage(
            content="",
            tool_calls=[
                ToolCall(
                    name="lookup_order",
                    args={"order_id": "BAD"},
                    id="call_lookup_1",
                )
            ],
        ),
        AIMessage(
            content="",
            tool_calls=[
                ToolCall(
                    name="lookup_order",
                    args={"order_id": "ORD-1042"},
                    id="call_lookup_2",
                )
            ],
        ),
        AIMessage(content="The order is ready for pickup."),
    ]
)

model = GenericFakeChatModel(messages=responses)

Confirm the constructor parameter against the installed LangChain version. The official test guide is the authority for the current API. The durable design is the iterator: one invocation consumes one known response.

Use unique call IDs across attempts. The corrected retry is a new tool call, not a mutation of the first call. That lets the message history preserve which observation belongs to which request. Deliberately include mismatched-ID fixtures to prove the harness does not attach a success to the failed attempt.

Strings are useful for final text, while AIMessage objects give explicit control over tool calls and metadata. Prefer the object form whenever the test asserts message fields. It keeps tool behavior visible instead of hiding it in a provider-specific text representation.

How do you create tools with controlled failures?

Build a deterministic tool whose failure depends on an explicit input or test fixture. Do not use a real order service to create a unit-test error. The tool should record calls, classify failures, and return or raise according to the agent's selected error-handling policy.

LangChain's official tools documentation describes the current tool API. A simple controlled tool can be defined as follows:

from dataclasses import dataclass, field
from langchain_core.tools import tool

@dataclass
class CallLog:
    arguments: list[str] = field(default_factory=list)

call_log = CallLog()

class ToolDomainError(Exception):
    """Expected tool failure that may be shown to the model."""

class InvalidOrderId(ToolDomainError):
    pass

class OrderAccessDenied(ToolDomainError):
    pass

@tool
def lookup_order(order_id: str) -> str:
    """Return a controlled order status for a test identifier."""
    call_log.arguments.append(order_id)
    if order_id == "BAD":
        raise InvalidOrderId("order_id must use the ORD- prefix")
    if order_id == "ORD-DENIED":
        raise OrderAccessDenied("order access denied")
    return "ready for pickup"

Keep the exception messages free of real secrets even in tests. Add canary values in a dedicated redaction case and assert they do not enter model messages or snapshots.

The agent's tool error policy determines whether an expected domain exception becomes a ToolMessage, propagates to the caller, or is transformed by middleware. Give expected failures dedicated types and freeze that policy in the test. Do not catch every exception or return a success-looking string such as "error"; either choice can erase the difference between a handled domain failure and a programming defect.

For tools with side effects, provide a fake transaction ledger or operation record. A retry after an uncertain failure should not silently perform the action twice. Secure agent tool execution architecture covers approval and side-effect controls beyond this local sequence.

How do you test a fail, retry, and recover sequence?

First write the expected chronology in behavior language. The model calls lookup_order with an invalid identifier. The tool failure enters history with the first call ID. The model calls the same tool with a corrected identifier and a new ID. The successful observation enters history. The final model message explains the result and no additional tool call occurs.

The numbered workflow is:

1. Define the allowed tool, arguments, retry limit, and terminal policy.
2. Script AI messages with unique tool call IDs for each attempt.
3. Configure one controlled failure and one controlled success.
4. Invoke the same agent or message-loop entry point used by production code.
5. Collect AI messages, tool calls, tool observations, and terminal output.
6. Correlate every ToolMessage to the originating tool call ID.
7. Assert revised arguments, attempt count, recovery, and final text.
8. Assert no extra model or tool invocation after the terminal response.

An application-specific loop can expose the orchestration clearly:

from langchain_core.messages import HumanMessage, ToolMessage

class ScriptLimitExceeded(RuntimeError):
    pass

def run_scripted_lookup(
    model,
    tool_fn,
    *,
    max_steps: int = 4,
    max_tool_attempts: int = 2,
):
    history = [HumanMessage(content="Check order ORD-1042")]
    model_steps = 0
    tool_attempts = 0

    while model_steps < max_steps:
        model_steps += 1
        ai = model.invoke(history)
        history.append(ai)
        if not ai.tool_calls:
            return history

        for call in ai.tool_calls:
            if tool_attempts >= max_tool_attempts:
                raise ScriptLimitExceeded(
                    f"Tool attempt limit reached: {max_tool_attempts}"
                )
            tool_attempts += 1

            try:
                result = tool_fn.invoke(call["args"])
                history.append(
                    ToolMessage(
                        content=result,
                        tool_call_id=call["id"],
                    )
                )
            except ToolDomainError as exc:
                history.append(
                    ToolMessage(
                        content=(
                            f"Tool failed: {type(exc).__name__}: {exc}"
                        ),
                        tool_call_id=call["id"],
                        status="error",
                    )
                )

    raise ScriptLimitExceeded(f"Model step limit reached: {max_steps}")

ToolDomainError is the only transformed exception boundary. An assertion failure, TypeError, KeyError, or other programming defect escapes to fail the test. The model and tool counters also make an extra scripted turn fail deterministically instead of depending on iterator exhaustion. Align supported ToolMessage fields with the installed package and official documentation; the key contract is correlation, bounded execution, and sanitized error status.

Test the history, not only the returned final message. Evaluate AI agent tool selection correctness extends this into model behavior evaluation, while the fake sequence proves the application handles the scripted choice correctly.

Which message and tool-call fields should you assert?

Assert the tool name, argument mapping, tool call ID, correlated tool message ID, error classification, successful result, message order, and final absence of tool calls. If the application stores attempt metadata or trace spans, assert those identifiers match the same chronology.

Avoid asserting incidental object representations. Convert messages into a small evidence record:

def summarize_history(history):
    summary = []
    for message in history:
        item = {"type": message.type, "content": message.content}
        if getattr(message, "tool_calls", None):
            item["tool_calls"] = [
                {
                    "name": call["name"],
                    "args": call["args"],
                    "id": call["id"],
                }
                for call in message.tool_calls
            ]
        if getattr(message, "tool_call_id", None):
            item["tool_call_id"] = message.tool_call_id
        summary.append(item)
    return summary

Then compare stable fields individually or with a reviewed inline structure. This avoids snapshotting internal serialization fields unrelated to the contract.

Call IDs deserve negative tests. Return a tool message with an unknown ID, use the first ID for the second result, or duplicate one ID across two active calls. The harness should reject or clearly classify each case. A final answer that happens to be correct does not repair corrupted correlation.

Also inspect argument revision. A retry with the same invalid arguments is a loop, not recovery. A retry that changes unrelated fields may still fail the intended contract. Evaluate AI agent tool use covers broader efficiency and appropriateness, while this test can use exact deterministic assertions.

How do you test stop paths and exhausted fake responses?

Every sequence needs a maximum number of model turns and tool attempts. Test a non-retryable permission error, repeated invalid arguments, success followed by an illegal extra call, and fake-response exhaustion. Each should have one named terminal classification.

import pytest

@pytest.mark.parametrize(
    ("script_name", "expected_calls", "terminal"),
    [
        ("recover-after-validation", 2, "answered"),
        ("stop-on-permission", 1, "tool-error"),
        ("repeat-invalid-until-limit", 2, "attempt-limit"),
        ("responses-exhausted", 1, "fixture-exhausted"),
    ],
)
def test_scripted_tool_paths(
    script_name,
    expected_calls,
    terminal,
    run_script,
):
    result = run_script(script_name, max_steps=4, max_tool_attempts=2)
    assert result.tool_call_count == expected_calls
    assert result.terminal == terminal
    assert result.open_tool_calls == []

The limits above are test fixture inputs, not universal recommendations. Select product limits from side-effect risk, cost, user experience, and tool semantics. The assertion is that the selected limit is enforced exactly.

Iterator exhaustion often signals an unexpected extra model call. Do not catch it and return an empty answer. Surface a fixture-specific failure that includes the consumed response count and sanitized history. For a test intentionally covering exhaustion, assert the application's terminal error and cleanup.

Stop-path tests should also prove no dangling tool call remains. Every emitted call has a correlated observation or an explicit terminal cancellation/error classification. Debugging missing tool spans in agent traces helps when the execution occurred but observability lost that pairing.

When should the same scenario call a real model?

Use a live provider when the requirement depends on actual tool binding, provider message conversion, model support for tool calls, streaming behavior, or the probability that a model selects and revises the right tool. A fake cannot prove any of those properties because the test authors choose every model response.

The official LangChain integration-testing guide defines the complementary boundary for tests that call real integrations. The LangChain agents documentation and provider integration docs should guide live setup. Keep credentials outside fixtures, bound cost and steps, sanitize captured messages, and record provider and model identifiers. Avoid copying a unit-test expected sequence into a live test and demanding exact wording.

A useful division is:

• Unit test: the scripted failure sequence is handled correctly.
• Provider integration: bound tools convert to and from the selected API.
• Agent evaluation: the real model chooses, revises, and stops acceptably.
• End-to-end test: the complete system applies policy and produces the user outcome with trace evidence.

The test in this article owns the first item. Testing AI agents describes the larger strategy, and building red-team datasets for tool abuse adds adversarial paths that should not all be reduced to scripted unit fixtures.

How do you keep deterministic agent tests useful in CI?

Keep fixtures short, named by behavior, and close to the orchestration contract. Pin the LangChain package through the project's dependency process and run message-shape tests when upgrading. Avoid private implementation attributes when public message fields provide the needed evidence.

CI should cover one success, one recoverable validation error, one non-retryable error, one repeated invalid call, one mismatched ID, one exhausted sequence, and one side-effect uncertainty path if relevant. Preserve the sanitized message summary and terminal classification on failure.

Review a fixture whenever production policy changes. A deterministic test can remain green while encoding an obsolete retry decision. Link each script to an owned requirement, incident, or risk. Contract testing tool schema evolution is especially relevant when argument names or required fields change.

Do not allow fakes to leak into production dependency wiring. Inject the model at the application boundary and assert that deployment configuration selects a real provider. A fake is valuable because its boundary is explicit.

Test parallel and multiple tool calls deliberately

An AIMessage can contain more than one tool call. Do not let a single-call fixture define behavior accidentally for all cases. Script two independent read-only calls, return their ToolMessage values in the order chosen by the application, and verify correlation uses IDs rather than list position.

Then make one call fail while the other succeeds. Freeze the product decision: does the agent receive both observations, stop the batch, retry only the failed call, or discard partial results? LangChain does not choose the business policy for every application. The unit test should state the selected rule and prove that one failed ID cannot overwrite the successful observation.

For side-effecting tools, avoid parallel execution unless the application has an explicit ownership model. A scripted model can request two actions, but the orchestrator may need to serialize them, request confirmation, or reject the combination. Add an assertion at that boundary instead of allowing test execution order to decide the result.

Distinguish tool failure from harness failure

A controlled tool exception is part of the scenario. A fixture bug, exhausted iterator, invalid message object, missing tool registration, or assertion failure is a test harness problem. Give them different terminal classifications. Otherwise a broken fixture can look like evidence that the agent handled a tool failure.

Wrap only the exception boundary the production loop is meant to transform. Do not catch Exception around model invocation, tool lookup, message construction, assertions, and cleanup as one block. A narrow wrapper lets the test prove exactly which tool exception became a correlated error observation.

Use a canary exception type from the controlled tool and assert it does not escape when the policy handles it. Then raise an unexpected programming error from the harness and assert it does escape. This negative control keeps the test from normalizing defects into model-visible text.

Review the prompt and history supplied on each turn

The fake controls outputs, but the application still builds model inputs. Capture the message list passed to each invocation. After the first failure, the next model call should include the original user request, the first AIMessage, and the correlated error ToolMessage in the expected order. After the successful retry, the final invocation should include the second call and its success.

Assert absence as well as presence. The error history should not contain raw stack traces, credentials, internal object representations, or unrelated tool results from another test. If the application summarizes older messages, test the summary contract rather than demanding full history, and preserve IDs needed to explain the active recovery.

This input inspection catches a class of false tests where the fake emits the correct next response regardless of missing history. The final answer looks right even though a real model would never have seen the tool error. Verifying each invocation's input proves the orchestration supplied the evidence that the script assumes.

Use scenario builders without hiding the chronology

Small builders can reduce fixture noise:

• tool_call(name, args, id) creates an AIMessage.
• tool_error(id, category) creates sanitized expected evidence.
• tool_success(id, value) creates a successful observation.
• final_text(value) creates the terminal model response.

Keep the resulting sequence visible in each test. A generic builder that accepts dozens of flags makes reviewers reconstruct the path mentally and can reuse IDs accidentally. Prefer named scenario functions such as validation_error_then_recover and permission_error_then_stop, each returning a short sequence plus expected evidence.

Validate scenario fixtures before execution. Require unique call IDs, expected observations for every scripted call, an explicit terminal condition, and attempt counts within the scenario's limit. This catches malformed tests before they reach agent code.

Promote real incidents without copying sensitive traces

When production reveals a recovery defect, translate it into the smallest deterministic sequence. Preserve the tool category, argument shape, error classification, decision, and ordering. Replace user data and provider payloads with synthetic values. Link the fixture to an internal incident identifier without embedding restricted text.

The regression should fail against the pre-fix orchestration and pass after the repair. Keep a separate integration or evaluation case when real-model selection was also part of the incident. One incident can produce several tests because provider choice, message conversion, agent decision, and tool execution have different owners.

Review incident-derived fixtures after schema or policy changes. Retire a case only when its protected behavior no longer exists, and document the replacement coverage. Determinism makes a regression durable, but governance keeps it relevant.

Run a negative control against the final-answer assertion. Change the scripted success observation while keeping the final text unchanged, then require the history or evidence assertion to fail. This proves the test is not merely accepting a canned answer disconnected from tool results.

Also verify cleanup after every scenario. Reset the call log, dispose callbacks, and create a fresh response iterator per test. Reusing an exhausted iterator or mutable call list can make execution order part of the result. Isolation should be visible in fixtures so parallel CI workers do not share scripted state.

FAQ

Can GenericFakeChatModel emit tool calls?

Yes. Supply AIMessage instances containing ToolCall values with a name, argument mapping, and call ID. The agent or test harness can execute the matching controlled tool and return a correlated ToolMessage. Assert the exact sequence, not only the final answer.

What happens when its iterator is exhausted?

The iterator cannot produce another response, so treat exhaustion as a fixture or stop-path signal rather than silently continuing. Add explicit bounds on agent steps and tool attempts, then assert whether exhaustion means an unexpected extra model call or an intended terminal case.

Should unit tests assert exact final prose?

Usually no. Assert exact tool names, arguments, call IDs, message order, terminal class, and required meaning. Exact final prose is appropriate only when wording itself is the contract. Otherwise, a scripted sentence snapshot can create noise while missing a broken tool observation or recovery path.

How are tool-call IDs verified?

Capture every AIMessage tool call and resulting ToolMessage, then assert each observation references the originating call ID exactly once. Use unique IDs for retries and add negative fixtures for unknown, duplicated, or mismatched IDs. Final-answer correctness must not hide corrupted correlation.

When should integration tests use a real provider?

Use a real provider when the requirement concerns actual message conversion, tool binding, provider compatibility, streaming, or probabilistic tool selection. Keep deterministic fake sequences for orchestration contracts, then run a smaller controlled integration suite with credentials, execution bounds, redaction, and recorded provider and model metadata.

Can fake models test streaming?

GenericFakeChatModel supports regular and streaming test paths, so it can exercise application handling of scripted streamed content without a provider call. It still does not prove the selected provider's event format, timing, interruption behavior, or tool-call streaming. Cover those properties in provider integration tests.

Conclusion

Scripted fake models are precise tools for orchestration tests. They let a team force a tool failure, observe the correlated error message, require a revised call, and prove termination without asking a live model to choose the desired path on demand.

Keep the claim honest: the fake proves application behavior under a supplied sequence. Pair it with provider integration and agent evaluation where real model behavior matters. That layered evidence keeps fast CI tests deterministic without mistaking a script for proof of model quality.