Stop testing LangChain, LangGraph, and Langflow as one black box

The shipping assistant quotes the correct price in a LangChain unit test, and its LangGraph route test is green. The deployed Langflow endpoint still returns an error because the flow points at an old component and the API key is missing. One end-to-end assertion reports “agent failed,” but it does not tell you which of three very different systems broke.

Treat these tools as layers, not synonyms. LangChain shapes model and tool interactions. LangGraph owns explicit state transitions and durable workflow behavior. Langflow packages components and wiring into a visual flow served by a runtime. A practical test strategy gives each layer an oracle it can actually satisfy.

Draw the boundary before choosing the test

Start with the behavior your product owns. A shipping rule, permission check, or document lookup remains your code even when a framework exposes it as a tool. Test that behavior without a model. If the function returns the wrong rate for a known zone, no graph trace or prompt evaluation will make the answer correct.

Move one boundary outward for framework adaptation. Check that the LangChain tool accepts the intended arguments and returns a value your agent can consume. This is where renamed fields, serialization mistakes, and accidental exception wrapping appear. A model is unnecessary when the input is already known.

At the graph layer, the important contract is state plus route. Given a state before a node, what update does the node produce? Given that update, which edge runs next? Checkpoint behavior belongs here too, but only when the case involves multiple invocations, interrupts, or restored state. Official LangGraph test guidance recommends creating the graph for a test and compiling it with a fresh checkpointer, which prevents state from one case leaking into another.

The deployment layer answers another question: did the flow artifact, runtime, credentials, components, and server configuration come together? A Langflow API smoke test should cross the network to the running instance. It should not become the only place where shipping arithmetic or graph routing is checked. When it fails, you want lower-level green tests to narrow the search.

Keep model behavior in its own lane. A scripted or fake model is useful for proving how your application handles a specific tool call, refusal, or malformed response. A real-provider check is useful for detecting SDK, credential, and model compatibility. Neither is a stable oracle for exact prose. Official LangChain guidance makes the same practical split: unit tests can use in-memory fakes, while integration tests that call providers should be marked and run separately.

This produces a small pyramid with four kinds of evidence:

1. Domain tests prove deterministic rules and side effects.
2. LangChain adapter tests prove tool inputs and outputs.
3. LangGraph tests prove state transitions, routing, and persistence.
4. Langflow smoke tests prove a versioned flow runs in a deployed environment.

The model can appear at the third or fourth level, but it should not decide whether basic code is correct. If every test starts with a prompt and ends with a string comparison, failures will be slow, costly, and difficult to own.

Keep LangChain tests deterministic

Consider a tool that quotes shipping. The business rule has clear inputs, so put it in an ordinary function and expose a thin LangChain wrapper. The wrapper's docstring helps a model choose the tool, but the rate table does not need model judgment.

from decimal import Decimal

from langchain.tools import tool


RATES = {
    "local": Decimal("4.50"),
    "national": Decimal("9.75"),
}


def calculate_shipping(zone: str, weight_kg: Decimal) -> dict[str, str]:
    if zone not in RATES:
        raise ValueError("unsupported_zone")
    if weight_kg <= 0:
        raise ValueError("weight_must_be_positive")

    surcharge = max(Decimal("0"), weight_kg - Decimal("2")) * Decimal("1.25")
    total = (RATES[zone] + surcharge).quantize(Decimal("0.01"))
    return {"currency": "USD", "amount": str(total)}


@tool
def quote_shipping(zone: str, weight_kg: float) -> dict[str, str]:
    """Return a shipping quote for a supported zone and positive weight."""
    return calculate_shipping(zone, Decimal(str(weight_kg)))

Three tests belong beside this code. One checks a base rate, one checks the overweight branch, and one proves invalid input cannot turn into a quote. The adapter gets one focused check through its documented invoke interface. If a framework upgrade changes tool construction, the adapter check fails while the domain cases still establish that pricing logic is sound.

from decimal import Decimal

import pytest

from shipping import calculate_shipping, quote_shipping


def test_local_base_rate():
    assert calculate_shipping("local", Decimal("1.5")) == {
        "currency": "USD",
        "amount": "4.50",
    }


def test_national_overweight_rate():
    assert calculate_shipping("national", Decimal("3")) == {
        "currency": "USD",
        "amount": "11.00",
    }


def test_non_positive_weight_is_rejected():
    with pytest.raises(ValueError, match="weight_must_be_positive"):
        calculate_shipping("local", Decimal("0"))


def test_langchain_tool_adapter_returns_domain_result():
    assert quote_shipping.invoke({"zone": "national", "weight_kg": 3.0}) == {
        "currency": "USD",
        "amount": "11.00",
    }

Each assertion can detect a product change. Replacing max(0, weight - 2) with weight breaks the overweight expected value. Removing validation breaks the exception case. Renaming weight_kg in the tool breaks the adapter invocation. These are real oracles, not checks that a constant appears in a constant list.

The nearest false friend is a model selection test. If a prompt says “quote national shipping for three kilograms” and a live model sometimes answers without calling the tool, that may expose prompt or model behavior, but it does not disprove the tool. Preserve the model request, returned message type, tool-call name, and parsed arguments. Report it as a selection failure. Do not rewrite the shipping test until evidence shows the tool itself received bad data or returned a bad result.

Scripted model responses are best for branches your application must always handle. Feed one well-formed tool call, one unknown tool name, and one invalid argument object through your dispatcher. Assert the resulting error codes and ensure no tool runs for rejected calls. A live model belongs in a smaller compatibility suite where the oracle is structural, such as at least one call to quote_shipping with schema-valid arguments, rather than an exact sentence.

Keep schema rejection separate from tool execution failure. A call missing weight_kg should be rejected before shipping code runs. A call with weight_kg=0 reaches the domain rule and returns weight_must_be_positive. A call with valid data that times out while fetching a remote rate is an execution dependency failure. These outcomes may all become tool error messages in a conversation, but they point to different owners and retry decisions. Instrument the adapter with a call ID and a stage code so the trace preserves the difference.

Tool descriptions need focused regression coverage too. A renamed concept can make a live model stop selecting a tool even while direct invocation remains green. Store a small set of user intents and check the selected tool structurally with the supported models you deploy. Do not demand one exact argument value when several interpretations are valid. For the shipping example, “three kilograms” has a deterministic weight, while “a heavy parcel” does not. The latter belongs in a clarification test, not the rate calculation suite.

Side-effecting tools require an additional boundary. A fake model can ask for cancel_order twice, and your dispatcher should prove that an idempotency key prevents a duplicate cancellation. That is not a LangChain guarantee. It is an application contract around the tool. Record both attempted calls, the shared business key, and the one accepted target operation. A test that checks only the final chat answer can miss the duplicate entirely.

Mocks have a cost. A fake cannot reveal a provider-side schema rejection, authentication error, streaming change, or rate limit. Keep at least one real integration case for the model and SDK versions you deploy. Mark it so a developer can run fast tests without secrets and CI can schedule the live check deliberately.

Test LangGraph state and routing directly

Graphs deserve tests below the full agent loop. Suppose expensive refunds require review while smaller refunds can execute automatically. The route is deterministic and should not depend on a model. A node normalizes state, then a conditional edge selects the next node.

from typing import Literal, TypedDict

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import END, START, StateGraph


class RefundState(TypedDict):
    amount_cents: int
    route: str
    outcome: str


def choose_route(state: RefundState) -> dict[str, str]:
    route = "review" if state["amount_cents"] >= 10000 else "automatic"
    return {"route": route}


def next_node(state: RefundState) -> Literal["review", "automatic"]:
    return "review" if state["route"] == "review" else "automatic"


def review(state: RefundState) -> dict[str, str]:
    return {"outcome": "waiting_for_review"}


def automatic(state: RefundState) -> dict[str, str]:
    return {"outcome": "ready_to_refund"}


def build_refund_graph():
    builder = StateGraph(RefundState)
    builder.add_node("choose_route", choose_route)
    builder.add_node("review", review)
    builder.add_node("automatic", automatic)
    builder.add_edge(START, "choose_route")
    builder.add_conditional_edges("choose_route", next_node)
    builder.add_edge("review", END)
    builder.add_edge("automatic", END)
    return builder.compile(checkpointer=InMemorySaver())

Test values immediately below and exactly at the boundary. Those two rows catch an off-by-one change that a generic “small” and “large” pair may miss. Give every case a unique thread ID and compile a fresh graph. Reusing a checkpointer can preserve prior state and create a pass that disappears when the test runs alone.

import pytest

from refund_graph import build_refund_graph


@pytest.mark.parametrize(
    ("amount_cents", "expected_route", "expected_outcome"),
    [
        (9999, "automatic", "ready_to_refund"),
        (10000, "review", "waiting_for_review"),
    ],
)
def test_refund_route(amount_cents, expected_route, expected_outcome):
    graph = build_refund_graph()
    result = graph.invoke(
        {"amount_cents": amount_cents, "route": "", "outcome": ""},
        config={"configurable": {"thread_id": f"refund-{amount_cents}"}},
    )

    assert result["route"] == expected_route
    assert result["outcome"] == expected_outcome

This graph stops at a status node; it does not pretend to test human approval. An interrupt test would need a checkpointer, an initial invocation that exposes the interrupt, and a second invocation using the same thread ID with Command(resume=...). Keep that as a separate case because it exercises persistence and two-call sequencing. When a one-invocation route test fails, adding resume mechanics only obscures the branch defect.

Test individual nodes when their transformation is substantial. LangGraph's compiled graph exposes nodes for direct invocation, but doing so bypasses checkpointer behavior. That is appropriate for a pure normalization node and inappropriate for proving restoration after a restart. Name tests by the contract they cover so a future maintainer does not read “node passed” as “checkpoint passed.”

A graph can return the expected final state through the wrong path. If route choice matters for cost, permission, or side effects, record a path event or assert a node-specific field. For the refund graph, outcome distinguishes the two terminal nodes. In a real system, use an execution spy to prove the automatic refund function was not called on the review route.

Persistence needs a worked case that crosses process-shaped boundaries. Invoke a graph until an interrupt, retain only the durable checkpoint and thread ID, construct a new graph instance with access to the same test database, and resume it. Assert the restored request identifier, the pending action, and the final side effect. An in-memory saver cannot prove this because a new process cannot share its memory. Use the same checkpointer technology and schema version as production for a small nightly or pre-release suite.

The near-miss is a test that creates a new compiled graph but accidentally shares a module-level in-memory saver. It passes when cases run in file order and fails under random order or parallel workers. Evidence includes a state field that no fixture supplied and a thread ID reused across parameter rows. Print the thread ID in assertion messages, generate it per case, and make the checkpointer an explicit fixture. Clearing a global dictionary in teardown treats the symptom; removing global persistence fixes the isolation boundary.

Routing and state-update failures can also imitate one another. If choose_route writes review correctly but the conditional function reads an older field, the state assertion passes while the terminal outcome is wrong. If the conditional edge is correct but the review node overwrites the route, the final state can hide the earlier decision. Preserve a compact transition event with node name, input state version, chosen label, and output state version. That evidence tells you whether the bad value was produced, consumed, or overwritten.

Treat Langflow as a deployed application

The visual editor introduces artifacts and environment state that Python unit tests cannot see. A saved flow can point at a component ID that no longer exists, depend on a package missing from the runtime, or reference a secret available only on a developer laptop. The server may be healthy while a particular flow is broken. That is why one health request is necessary but insufficient.

Version the exported flow JSON or use Langflow's supported flow tooling for your deployed version. Review changes as configuration changes, not screenshots. A node moving on the canvas is usually noise; a component type, connection, prompt, model setting, or input mapping changing is behavior. If raw export diffs are too noisy, generate a stable semantic manifest in your own build step and test that manifest with code you own.

For a deployed smoke test, use the endpoint and payload documented by the Langflow version you operate. Current official examples send a POST request to /api/v1/run/{flow_id}?stream=false with input_value, input_type, and output_type, plus an x-api-key header when authentication is enabled. The flow's own inputs can require a different payload, so copy the generated API example from that flow rather than assuming every flow is chat-shaped.

The test below checks a chat flow contract without asserting the model's prose. It requires explicit environment variables, gives the run a unique session ID, verifies the documented top-level response fields, and writes the response body into the assertion if the shape is wrong.

import os
from uuid import uuid4

import pytest
import requests


@pytest.mark.langflow
def test_deployed_shipping_flow_runs():
    server = os.environ["LANGFLOW_SERVER_URL"].rstrip("/")
    flow_id = os.environ["LANGFLOW_FLOW_ID"]
    api_key = os.environ["LANGFLOW_API_KEY"]
    session_id = f"qa-shipping-{uuid4()}"

    response = requests.post(
        f"{server}/api/v1/run/{flow_id}?stream=false",
        headers={"x-api-key": api_key},
        json={
            "input_value": "Quote local shipping for one kilogram.",
            "input_type": "chat",
            "output_type": "chat",
            "session_id": session_id,
        },
        timeout=60,
    )

    assert response.status_code == 200, response.text
    payload = response.json()
    assert payload["session_id"] == session_id
    assert isinstance(payload["outputs"], list)
    assert payload["outputs"], response.text

This proves the named flow is reachable and returns output in the deployment. It does not prove the quote is correct. If price accuracy matters, use a deterministic component or test environment where the response contains structured tool output you can extract reliably. Do not parse a conversational sentence with a regular expression and call that a pricing oracle.

Use a unique session ID because conversational flows can retain context by session. Sharing one ID across CI runs makes outcomes depend on history and can expose other tests' messages. Keep the flow ID outside the code so staging and production-like environments can deploy different identifiers. A missing environment variable should fail at setup rather than quietly hitting an empty URL.

Authentication deserves a negative smoke case in a non-production environment. Omit or replace the key and assert the server refuses the request. Do not hard-code one exact status without checking the authentication configuration and the version you run; instead define your deployment's accepted denial statuses as an explicit contract. The positive case should never print the key, even on failure.

One common deployment defect begins with an innocent editor change. A teammate replaces the Chat Input component, its generated component ID changes, and an environment-specific tweak still targets the old ID. The flow loads, the health endpoint is green, and a run may even return output, but the intended override never reaches the new component. Compare the exported flow's component IDs with every tweak key during artifact validation. The deployed smoke should then prove the behavior controlled by that tweak, not merely that some text came back.

Another failure appears only with conversation history. A local playground uses a clean session while CI reuses the flow ID as its default session, so a prior message changes the next response. The unique session_id in the example removes that dependence. Add a separate memory test if continuity is a product requirement: send two requests with the same generated session ID, then assert a stable structural sign that the second turn received context. Do not let the ordinary smoke test inherit memory accidentally.

Streaming changes the transport contract. The non-streaming example expects one JSON document because it sends stream=false. If production enables streaming, write a separate client test for event framing, terminal completion, and error handling. Do not point the JSON smoke test at a streaming endpoint and loosen assertions until it passes. A proxy can buffer or terminate a stream while the underlying flow remains correct, so preserve proxy status, response headers, received event count, and the last complete event when diagnosing that path.

Custom components add packaging risk. Validate imports in the same image that will serve the flow, not only in the editor environment. A component may import successfully on a laptop because an unrelated package installed it transitively. Generate or review the runtime requirements, build the image, start the server, and execute the smallest flow that uses the component. The trade-off is build time, which is why this check belongs after fast Python tests rather than inside every tool case.

Diagnose failures without blaming the wrong layer

Read evidence from the outside inward. If the Langflow request cannot connect, inspect DNS, TLS, port exposure, and service health. A Python tool unit test cannot explain a connection refusal. If /health_check succeeds but the run returns an authentication failure, examine key injection and server authentication. If authentication succeeds but the flow ID is not found, compare the deployed artifact and environment variable.

Once the server accepts the run, capture the response body, flow ID, session ID, deployment version, and component error. A component import failure points toward packaging. A validation error on input_value or the selected input type points toward the API contract. A provider authentication error belongs to the model component's secret. Keep secrets redacted while preserving the provider error code.

If Langflow finishes but the product route is wrong, reproduce the same normalized state in the LangGraph test. A graph failure with that state gives the graph owner a deterministic case. A passing graph plus a failing deployed flow suggests the visual wiring or input adapter did not construct the same state. Log the state at the boundary, with sensitive values removed, instead of comparing two final chat sentences.

If the route is correct but a tool result is wrong, invoke the LangChain tool with the arguments recorded in the trace. A failing tool invocation isolates business code or serialization. A passing tool invocation means the deployed flow may have selected another tool, altered arguments, or used a different package version. Record tool name, schema version, normalized arguments, and result type for this reason.

Provider variation is the last category, not the first excuse. Replay a scripted response through your dispatcher. If application handling fails, fix deterministic code. If scripted handling passes and repeated live calls produce different valid choices, adjust the prompt, model configuration, or evaluation policy. Never add blind retries until you know whether the first call was a transport failure, a valid refusal, or an unsafe tool choice.

The most deceptive near-miss is stale deployment. Local LangChain and LangGraph tests run the current source, while Langflow serves yesterday's flow or dependency image. Compare a build identifier exposed by your application boundary, not a timestamp guessed from the UI. A green local suite and a reproducible old behavior at the API are strong evidence of version drift.

Migrate a mixed end-to-end suite

Inventory existing tests by the first useful failure they can produce. A case that prompts the full system only to verify a fixed shipping rate belongs at domain level. A case that checks review routing belongs at graph level. A case that proves the packaged flow can load a custom component belongs at deployment level. Preserve a few critical journeys end to end, but stop asking them to diagnose every rule.

Extract deterministic fixtures from traces carefully. Keep tool arguments, relevant state, and expected structural outcomes. Remove provider IDs, timestamps, and wording that are not part of the contract. Do not copy sensitive prompts into the repository. A fixture should explain why it exists and which incident or requirement it protects, without requiring production data.

Move tests in small groups and run old and new checks together for a short period. When they disagree, inspect which oracle is closer to the behavior. A lower test may reveal that the old end-to-end case passed for the wrong reason, such as a model answering from its own knowledge instead of calling the shipping tool. Do not preserve that accidental behavior to avoid changing a dashboard.

Register markers for live suites, then turn an unknown marker into an error, because registration by itself does not do that. An unregistered mark raises PytestUnknownMarkWarning: Unknown pytest.mark.langfow - is this a typo? and the run still reports the test as passed. Registering the correct names does not change that outcome for the misspelled one, since the warning is attached to marks pytest does not recognise, and a warning is not a failure. --strict-markers is what promotes it: with the flag, the same typo stops collection with an error reading 'langfow' not found in `markers` configuration option. The distinction matters more than it looks, because the deterministic job selects with -m "not langflow and not live_model". A mark spelled langfow is excluded by neither clause, so a test intended for the Langflow job silently joins the pull-request job and starts calling a deployed flow from a job that has no credentials for it.

# pyproject.toml
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
    "langflow: exercises a deployed Langflow flow over HTTP",
    "live_model: calls a real model provider",
]

Keep deterministic tests as the default pull-request job. Run Langflow smoke tests against an ephemeral or controlled staging deployment after packaging. Schedule real-provider tests separately or gate them on changes to prompts, models, adapters, and dependencies.

name: agent-test-layers

on:
  pull_request:

jobs:
  deterministic:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: python -m pip install -r requirements.txt
      - run: python -m pytest -q --strict-markers -m "not langflow and not live_model"

  langflow-smoke:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements.txt
      - run: python -m pytest -q --strict-markers -m langflow
        env:
          LANGFLOW_SERVER_URL: ${{ secrets.LANGFLOW_SERVER_URL }}
          LANGFLOW_FLOW_ID: ${{ secrets.LANGFLOW_FLOW_ID }}
          LANGFLOW_API_KEY: ${{ secrets.LANGFLOW_API_KEY }}

The fork check prevents secrets from being exposed to untrusted pull-request code. Your CI provider and repository policy still need a security review; the example is wiring, not a universal secret-management guarantee. Some teams run deployment smoke tests only after merging into a protected environment, which trades earlier feedback for tighter credential isolation.

Track each layer separately. Domain failures should be near-zero and fast. Graph failures should identify a state and route. Langflow failures should identify deployment and flow IDs. Live-model failures should retain provider response structure and evaluation criteria. A single “agent pass rate” hides which investment would improve reliability.

Know when the three-layer split is not useful

Do not create framework tests for a one-function prototype that has no graph or deployed Langflow flow. Test the function and add the next layer when the architecture earns it. A ceremonial test for every library adds maintenance without catching a new class of failure.

Avoid duplicating the same assertion at all three levels. One domain case can exhaust a rate table; one graph case can prove that tool result is carried into state; one smoke case can prove the deployment executes. Repeating every rate combination through a live model multiplies cost and flakiness without increasing boundary coverage.

Do not mock the exact integration you need to validate. A fake Langflow response cannot prove the flow artifact loaded. An in-memory checkpointer cannot prove your production database survives a process restart. A direct tool invocation cannot prove the model provider accepts the schema. Choose at least one realistic test for each high-risk boundary and keep its scope narrow.

The split also does not excuse missing product evaluation. Correct tools and routes can still produce an unhelpful agent experience. Use separate evaluation for answer relevance, groundedness, and safety, with criteria tied to user needs. Keep those scores out of deterministic authorization and arithmetic gates unless you have calibrated an evaluator for that exact decision.

Finally, do not keep a visual flow merely to complete the stack. If Langflow is not part of production, it should not be in the release gate. If LangGraph adds no stateful orchestration, ordinary application code may be easier to test. The right strategy follows the deployed boundaries, not the number of framework names in an architecture diagram.