Catch hidden structured-output repairs before they ship

Your assertion receives a valid ProductRating(rating=5), so the test passes. The model's first attempt contained a value outside the schema, LangChain sent repair feedback, and a second model call produced the accepted object. The final type is valid, but the test says nothing about the hidden retry, its cost, or whether the repair changed the meaning.

That blind spot is common in structured-output tests. Teams assert only result["structured_response"] and lose the trajectory that produced it. A useful suite treats first-attempt success, repaired success, exhausted repair, and non-repairable failure as different outcomes.

Test the mechanism LangChain actually uses

In current LangChain Python documentation, create_agent accepts a response format and places the validated result in the final state's structured_response key. ToolStrategy uses artificial tool calling for that response. When a structured-output tool call fails schema validation, the documented flow adds error feedback as a ToolMessage and asks the model to try again.

The control for that behavior is ToolStrategy(..., handle_errors=...). It accepts True, False, a string, an exception type, a tuple of exception types, or a callable that returns feedback text. The default is True. False lets errors propagate. A string supplies fixed feedback. Exception classes narrow which errors are handled. A callable can choose a message based on the exception.

Product teams sometimes call this repair middleware because it sits between a model attempt and the next attempt. That description is informal. It must not be confused with LangChain's middleware=[...] hooks. The LangChain v1 migration guide explicitly distinguishes wrap_tool_call handling for errors during real tool execution from structured-output schema mismatches, which the structured-output machinery handles. Putting schema repair in generic tool middleware tests the wrong boundary.

Provider-native structured output is another path. ProviderStrategy relies on a provider's native capability, while ToolStrategy represents the result as a tool call. Tests that expect repair ToolMessage records are therefore strategy-specific. If production selects a strategy automatically from a schema type, record the selected strategy or pin it when trajectory behavior is part of the contract. Do not run a ToolStrategy fixture and claim it proves a ProviderStrategy integration.

Break the requirement into observable properties. The schema must accept intended values and reject prohibited values. The repair policy must handle only approved error classes. Feedback must be actionable without leaking sensitive input. The number of attempts must stay within the application's run budget. No structured value may be published before validation. The final object must remain faithful to source data, because schema validity alone cannot prove meaning.

Keep attempt outcome separate from run outcome. valid_first_attempt means no repair was needed. valid_after_repair means the application can continue but the trajectory changed. repair_exhausted means all permitted attempts failed. unhandled_error means the error was outside the repair policy. Those labels are application-owned examples, not fields promised by LangChain.

The difference affects release policy. A low-risk extraction feature may permit one repair and monitor its frequency. A financial or safety workflow may reject any attempt that requires the model to invent a corrected value. Another product may allow format repair but prohibit factual repair. One global “structured output passed” metric cannot express those decisions.

Configure a narrow repair policy and test its text

The configuration below uses documented exception types and a callable handler. It gives specific, non-sensitive instructions for validation and multiple-output errors. Any unexpected exception is re-raised instead of being converted into a misleading schema prompt.

from __future__ import annotations

from pydantic import BaseModel, Field
from langchain.agents import create_agent
from langchain.agents.structured_output import (
    MultipleStructuredOutputsError,
    StructuredOutputValidationError,
    ToolStrategy,
)


class ProductRating(BaseModel):
    rating: int | None = Field(
        description="Rating from 1 to 5, or null when the source has no compatible rating",
        ge=1,
        le=5,
    )
    comment: str = Field(description="Short comment supported by the source text")


def structured_output_feedback(error: Exception) -> str:
    if isinstance(error, StructuredOutputValidationError):
        return (
            "Return one ProductRating. Use an integer from 1 to 5 only when "
            "the source supports that scale; otherwise use null."
        )
    if isinstance(error, MultipleStructuredOutputsError):
        return "Return exactly one ProductRating for the supplied review."
    raise error


def build_rating_agent(model):
    return create_agent(
        model=model,
        tools=[],
        response_format=ToolStrategy(
            schema=ProductRating,
            handle_errors=structured_output_feedback,
        ),
        system_prompt=(
            "Extract only information supported by the review. "
            "Do not clamp, rescale, or invent a rating."
        ),
    )

This policy does not promise that the model will obey the feedback. It only defines what the agent sends when those documented errors reach the handler. The tests still need an exhausted path. The surrounding runtime also needs a finite run budget so a model that repeatedly emits invalid output cannot continue indefinitely.

Do not interpolate str(error) into feedback by default. Validation errors can contain rejected field values, and those values may include customer data. A fixed, schema-aware message often gives the model enough direction. Preserve detailed errors in protected diagnostics only when the security policy allows it.

Test the callback as an ordinary function. A StructuredOutputValidationError can be awkward to construct directly because it carries schema and tool-call context, so the agent-level fake test below proves that route. The multiple-output route does the same. For unexpected errors, a small unit test can pass a RuntimeError and assert that the exact object is raised.

The handler's message is part of model context. Treat changes like prompt changes. A broad instruction such as “fix it” may work with one model and fail with another. A message that tells the model to clamp an invalid value can produce schema-valid fabrication. Keep the instruction tied to the source-of-truth policy.

Script invalid, repaired, and rejected trajectories

LangChain's official unit-testing guide provides GenericFakeChatModel, which consumes a supplied iterator of strings or AIMessage objects. The base fake does not implement the tool binding needed by create_agent, so the example adds a minimal test-only override that binds and ignores the advertised tools while preserving the scripted messages. The fake does not prove provider serialization. It proves agent logic and repair handling deterministically.

This fixture supplies an invalid structured-output call followed by a valid one. A counting iterator makes the retry observable without relying on private model fields. The test also checks the error ToolMessage linked to the invalid call and the final Pydantic object.

from __future__ import annotations

from collections.abc import Iterator

import pytest
from langchain.agents import create_agent
from langchain.messages import AIMessage, ToolMessage
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain.agents.structured_output import StructuredOutputValidationError, ToolStrategy

from rating_agent import ProductRating, build_rating_agent


class ToolBindingFakeChatModel(GenericFakeChatModel):
    """Scripted model that accepts tool binding for agent unit tests."""

    def bind_tools(self, tools, *, tool_choice=None, **kwargs):
        return self.bind(tools=tools, tool_choice=tool_choice, **kwargs)


class CountingResponses(Iterator[AIMessage]):
    def __init__(self, messages: list[AIMessage]) -> None:
        self._messages = iter(messages)
        self.consumed = 0

    def __iter__(self) -> "CountingResponses":
        return self

    def __next__(self) -> AIMessage:
        message = next(self._messages)
        self.consumed += 1
        return message


def rating_call(call_id: str, rating: int | None, comment: str) -> AIMessage:
    return AIMessage(
        content="",
        tool_calls=[{
            "name": "ProductRating",
            "args": {"rating": rating, "comment": comment},
            "id": call_id,
            "type": "tool_call",
        }],
    )


def test_invalid_rating_is_repaired_on_the_second_model_attempt() -> None:
    responses = CountingResponses([
        rating_call("rating-invalid", 10, "Excellent"),
        rating_call("rating-valid", None, "Excellent, but no five-point scale was given"),
    ])
    agent = build_rating_agent(ToolBindingFakeChatModel(messages=responses))

    result = agent.invoke({
        "messages": [{"role": "user", "content": "Review: Excellent. Score shown: 10."}]
    })

    assert responses.consumed == 2
    assert result["structured_response"] == ProductRating(
        rating=None,
        comment="Excellent, but no five-point scale was given",
    )
    feedback = [
        message
        for message in result["messages"]
        if isinstance(message, ToolMessage)
        and message.tool_call_id == "rating-invalid"
    ]
    assert len(feedback) == 1
    assert "integer from 1 to 5" in str(feedback[0].content)


def test_handle_errors_false_propagates_validation_failure() -> None:
    responses = CountingResponses([
        rating_call("rating-invalid", 10, "Excellent"),
    ])
    model = ToolBindingFakeChatModel(messages=responses)
    agent = create_agent(
        model=model,
        tools=[],
        response_format=ToolStrategy(ProductRating, handle_errors=False),
    )

    with pytest.raises(StructuredOutputValidationError):
        agent.invoke({
            "messages": [{"role": "user", "content": "Review: Excellent. Score: 10."}]
        })

    assert responses.consumed == 1

The first test will fail if repair stops occurring, if a third model response is consumed, if feedback is not correlated to the invalid call, or if the final value changes. The second will fail if handle_errors=False begins swallowing the documented validation error. Neither test makes a network call.

Add a multiple-output fixture with two ProductRating calls in one AIMessage, followed by one valid call. Assert that the first call's two identifiers receive error feedback and that exactly two model responses are consumed. Follow the current LangChain message behavior in your installed version rather than guessing the number or wording of generated messages. The stable product claim is one accepted final structured response after approved feedback.

Also test exhaustion. Supply only invalid responses up to the application run budget and assert the terminal exception or failure result your wrapper exposes. ToolStrategy controls whether a qualifying error is sent back for retry, but your application still owns how an exhausted agent run is converted into an HTTP response, job state, or user message.

Make the attempt boundary exact. If the application permits one repair, the fixture needs two invalid responses and a third unused valid response. Assert that the valid response remains unconsumed when the budget closes. Otherwise an off-by-one implementation can make one more paid model call and still return the expected failure. The counter must represent model attempts, not the number of error messages, because one attempt can emit multiple invalid structured calls.

Correlate every feedback message to the call that caused it. LangChain ToolMessage uses tool_call_id for that relationship. In a multiple-output failure, do not accept feedback attached to an unrelated ordinary tool call. A model presented with mismatched history may behave unpredictably, and a trace viewer can make the repair appear to follow the wrong action.

Check message order as well as membership. The invalid AIMessage must precede its error ToolMessage, which must reach the model before the valid response. A test that collects all messages and uses any(...) can pass when the feedback is appended after the repaired output. Build a small index by message identity and assert required positions. Leave unrelated messages unordered unless the contract requires a relationship.

Use different call IDs in every scripted attempt. Reusing one ID makes the transcript invalid and can hide a bug that associates the second result with the first request. The IDs need not resemble provider-generated values; they only need to be unique and consistently correlated inside the fixture.

Test fixed-string handling separately from callable handling. A string policy intentionally returns the same instruction for every handled structured-output error. That can be suitable for a narrow schema but unhelpful for unions where “choose one type” and “fix a field” need different guidance. The test should reflect the configured mode instead of assuming all handle_errors values behave like the callback example.

Exception-tuple policies need a negative control. Configure only StructuredOutputValidationError, drive a multiple-output response, and assert that MultipleStructuredOutputsError propagates. Then include both exception types and require the repair. This proves that the exception filter is active. Merely checking the ToolStrategy.handle_errors attribute would show configuration but not agent behavior.

Schema evolution needs paired fixtures. Adding a required field can turn every previously valid first attempt into repaired success. Removing a constraint can eliminate repair and allow values the consumer rejects later. Run old accepted examples, old rejected examples, and new boundary cases against the changed schema. The change report should show which trajectories moved between outcome categories.

Pydantic validation order can affect feedback detail. Avoid assertions on a full framework-generated error string unless the wording is a supported interface your application consumes. Assert your callback's returned text, the exception category, the rejected field at an approved normalized layer, and the final outcome. This keeps patch releases from causing meaningless snapshot churn.

Streaming deserves a separate suite. Partial model output may contain incomplete tool-call arguments before the final AIMessage is assembled. The fake message test above exercises completed messages. If the application streams progress to a client, verify that it does not expose an unvalidated partial structured value as final and that a repair attempt does not leave the UI showing the rejected object.

Cancellation during repair is another distinct outcome. A user may cancel after invalid output and before the next model response. The wrapper should report cancellation, not repair exhaustion, and it should not continue consuming the fake response iterator. A deterministic cancellation seam can assert that the second model call never begins.

Parallel structured calls should remain tied to the documented multiple-output behavior. Do not reinterpret them as two valid final objects unless the schema and product deliberately support a collection. If the desired result is a list, model that list inside one structured response. Treating a protocol error as successful fan-out changes consumer semantics without a schema review.

Finally, save a normalized trajectory fixture that contains message type, call ID, schema name, validation category, repair decision, and final outcome. Exclude raw review text and rejected values unless the test needs them. The same fixture can drive the audit formatter and give reviewers a stable artifact when framework message serialization changes.

The diagnostic path should print message types, tool-call IDs, tool names, and redacted feedback, not entire prompts. Running the focused test with verbose output is a fast first step. Add an application-owned formatter if CI needs a stable artifact.

python -m pytest \
  tests/structured_output/test_rating_repair.py \
  -vv

Do not claim this command reproduces a provider's logs. It exercises the fake model and the agent's deterministic repair path. A live integration test is still needed to prove that the selected provider model supports the bound structured-output tool and that messages round-trip as expected.

Check semantic fidelity after schema repair

Schema validation answers whether data fit a declared shape. It cannot determine whether the data are supported by the source unless that relationship is encoded as a deterministic invariant or checked separately. A repaired rating of 5 fits a one-to-five field. If the source merely contained an unexplained 10, changing it to 5 may be fabrication.

Write source-aware assertions for exact extraction tasks. If an invoice says subtotal 100, tax 8, and total 109, all three fields are individually valid decimals, but the relationship is inconsistent. You can put an exact arithmetic invariant in the Pydantic model so the same structured-output repair path receives a validation error.

from __future__ import annotations

from decimal import Decimal
from pydantic import BaseModel, Field, model_validator


class InvoiceSummary(BaseModel):
    subtotal: Decimal = Field(ge=0)
    tax: Decimal = Field(ge=0)
    total: Decimal = Field(ge=0)

    @model_validator(mode="after")
    def total_matches_parts(self) -> "InvoiceSummary":
        if self.subtotal + self.tax != self.total:
            raise ValueError("total must equal subtotal plus tax")
        return self

This invariant is appropriate only if the source contract guarantees that equation and does not include shipping, discounts, rounding adjustments, or other components. A generic invoice schema often needs those fields or a tolerance policy. Copying the validator without the domain rule would reject valid documents.

Cross-field validators can catch enum combinations, date order, totals, and required companion fields. They cannot verify that a comment faithfully summarizes prose or that a name was copied from the right person. For those cases, compare against labeled fixtures when an exact answer exists, or use a separate evaluator with an explicit uncertainty policy.

Keep repair behavior from silently changing source facts. One safe pattern is to instruct the model to use None when an incompatible value cannot be converted under a documented rule. Another is to return a union with an explicit needs_review outcome. The right schema depends on the consumer. What matters is that “make validation pass” is not the only objective.

The first worked failure is format-only. The model returns rating as a valid numeric string while the contract requires a strict integer. If your Pydantic configuration permits coercion, no repair occurs. Decide whether coercion is acceptable and test the actual model configuration. Do not call the value repaired when the validator accepted it on the first attempt.

The second failure is multiple structured outputs. A prompt contains contact details and event details, and the model calls two response schemas when the selected union expects one. LangChain documents MultipleStructuredOutputsError for this case and can send corrective feedback. The test should prove that exactly one final value remains and that the chosen branch matches the source, not simply that a retry happened.

The third failure is an unrelated model or network exception. It should not receive “return an integer from 1 to 5” feedback. The callable handler above re-raises unknown exceptions. Retry policy for transient model calls belongs at the model-call boundary, with its own limits and diagnostics. Mixing infrastructure retry with schema repair makes attempt counts and user messages misleading.

A fourth near-miss is a real tool execution error. An agent may use ordinary tools in addition to the artificial structured-output tool. A database tool raising an execution exception is not a ProductRating schema mismatch. The v1 migration documentation points tool execution handling to wrap_tool_call. Keep tests for that middleware in the tool-error suite and verify that structured-output repair does not relabel the database failure.

Observability should show all four. Record strategy, schema version, attempt number, error category, repair decision, final validation result, and application outcome. Redact rejected values unless approved. Counting only successful final objects makes a prompt regression look like a latency increase with no explanation.

Delay irreversible consumers until every required check finishes. A Pydantic object should not trigger a payment, database write, or outbound message merely because field validation passed. Run source-aware and authorization checks first, persist an approved decision record, and only then cross the side-effect boundary. If a later check fails, report the structured response as rejected rather than “repaired successfully.”

Caching needs an explicit rule too. Caching only the final repaired object removes the failed attempt from later diagnostics and can make every replay appear clean. If caching is appropriate, store the final value with schema version and repair outcome, while keeping sensitive trajectory evidence under its shorter approved retention. Never reuse a repaired value for a different source document simply because the structured fields match.

Batch extraction can contain mixed outcomes. One item may validate on the first attempt, another after repair, and a third may exhaust its allowance. A top-level success flag loses that distinction. Return per-item status when consumers can handle partial results, or fail the batch atomically when the business operation requires all items. Test whichever contract the caller actually implements.

Repair feedback can also become stale after a schema change. A message that says “return one to five” is wrong after a product moves to a different scale or makes the field nullable for a new reason. Keep feedback text beside the schema, review them in the same change, and include a unit test that exercises each constraint named in the message.

Separate deterministic unit proof from live integration

The fake-model tests give exact trajectories at no provider cost. They are the right place for error scope, message text, attempt boundaries, multiple calls, exhaustion, and wrapper behavior. They do not prove that a live provider advertises or implements the required tool-calling capability.

Add a small integration suite using the model configuration production actually selects. LangChain's testing guide recommends separating these tests because they require credentials, incur cost, and face nondeterminism. Assert response structure and schema invariants rather than exact prose. Save the strategy and model identifier with failures.

Do not depend on a prompt reliably forcing one invalid attempt in a live test. That is nondeterministic and can become impossible as a model improves. Use the fake for the repair branch. Use the live test to prove a valid structured response can travel through the provider and that your tracing or callback adapter records the fields needed for operational monitoring.

If you need live evidence of historical repair frequency, analyze actual traces under an approved privacy policy. Do not manufacture an “expected 12 percent repair rate.” Any illustrative distribution must be labeled illustrative, and release thresholds should come from reviewed production or evaluation data.

Record package versions in failure output. Structured-output behavior and testing utilities can change between LangChain releases. A test that passes against one lockfile and fails in another environment is first a reproducibility problem. Align the environment before editing error assertions.

CI can run deterministic tests on every relevant change and reserve live integration for a protected environment or schedule. The job below intentionally installs from the repository's locked test requirements and does not put provider keys in source.

name: langchain-structured-output

on:
  pull_request:
    paths:
      - "agents/rating/**"
      - "structured_output/**"
      - "tests/structured_output/**"
  workflow_dispatch:

jobs:
  deterministic-repair:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements-test.txt
      - run: python -m pip install -r requirements-test.txt
      - run: python -m pytest tests/structured_output -m "not integration" -q

  provider-contract:
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    environment: agent-integration
    timeout-minutes: 15
    env:
      LANGCHAIN_TEST_MODEL: ${{ vars.LANGCHAIN_TEST_MODEL }}
      MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements-test.txt
      - run: python -m pip install -r requirements-test.txt
      - run: test -n "$LANGCHAIN_TEST_MODEL"
      - run: test -n "$MODEL_API_KEY"
      - run: python -m pytest tests/structured_output -m integration -q

Both jobs declare cache-dependency-path because the locked file is named requirements-test.txt. Pip caching hunts for **/requirements.txt and **/pyproject.toml on its own, finds neither, and ends the run at setup rather than at an assertion about repair behavior.

The generic MODEL_API_KEY name is application wiring. Provider integrations usually expect their own documented environment variable, so map the protected secret to the variable your selected integration reads or construct the model with an explicit credential object. Do not assume LangChain consumes MODEL_API_KEY automatically.

The trade-offs are visible. Narrow error handling returns more failures to callers. Richer schemas take more maintenance. Extra model attempts add latency and provider usage. Fake trajectories can drift from provider behavior. Live tests cost money and can flake. Capturing repair messages improves diagnosis but increases privacy review.

Roll out reporting before blocking. Count first-attempt success, repaired success, exhausted repair, and unhandled errors by schema version. Review examples of each. Then set per-workflow policy. A rising repaired-success rate can reveal a prompt regression even while every final object remains valid.

Version the schema, prompt, feedback text, strategy, and result adapter together. A migration from ToolStrategy to ProviderStrategy changes the expected trajectory even when the Pydantic class remains the same. Update tests to reflect the new mechanism, and preserve old trace interpretation under its original version.

Know when automatic repair is the wrong response

Disable automatic repair when correction could invent a consequential fact. If a payment amount, medication dose, approval identity, or legal status is incompatible with the schema, an explicit review outcome is safer than asking the model to make it fit.

Do not repair missing source information by filling a required field with a plausible value. Change the schema to represent absence, request clarification, or reject the operation. A valid object containing invented data is worse than a validation error.

Avoid parsing feedback text as your only production metric. Error wording can change, and custom messages may be identical across categories. Capture typed exceptions or normalized events at the boundary your installed version exposes. Use text assertions only for feedback that your own callback owns.

Do not route structured-output schema errors through ordinary tool middleware. That hook has legitimate uses for registered tool execution, but it is not the documented ToolStrategy repair control. Tests should fail if a database error is mislabeled as a response-format problem.

Finally, do not count a repaired object as equivalent to a clean first attempt without an explicit policy decision. The user may receive the same final shape, but the system took a different path, spent another model turn, and exposed another chance to alter meaning. Keep that evidence visible.