Unit Test CrewAI Tools Without LLM Calls

To unit test CrewAI tools without LLM calls, invoke the tool's Python interface with controlled inputs. Mock outbound I/O, assert typed results and failures, validate the input schema, and reserve model-backed checks for questions about tool selection or use. This split makes a failing test point to tool code instead of a changing model response.

The approach follows the tool surfaces in the official CrewAI tools documentation: decorated functions and BaseTool classes expose callable behavior, descriptions, schemas, synchronous execution, and asynchronous execution. Those are ordinary software contracts even when an agent will eventually use them. A crew is not required to prove that a search adapter handles a timeout or that an inventory lookup rejects an empty SKU.

This article focuses on the cold tool layer. For the wider system, testing AI agents covers model, retrieval, memory, and trajectory concerns, while evaluating an AI agent's tool use asks whether the agent chose and applied the tool correctly. Keep those questions in separate tests so each failure remains useful.

What does it mean to unit test CrewAI tools without LLM calls?

A CrewAI tool combines executable Python with agent-facing metadata. The executable part receives arguments, calls dependencies, computes or retrieves a result, and either returns a value or reports a failure. The metadata includes a name, description, input schema, and sometimes a result schema. Unit testing without an LLM means exercising these parts directly instead of asking a model to discover and invoke them.

The distinction matters because tool logic and tool use fail for different reasons. A postcode lookup can send the wrong query parameter even when an agent selects it perfectly. An agent can choose the wrong lookup even when both available tools are correct. A full crew run mixes those causes with provider availability, sampling, prompt wording, context, and orchestration. Direct invocation removes those variables from the first diagnosis.

The repository evidence behind this article is AI_AGENTS_CHALLENGES in src/db/seed-data/challenges/expansion-ai-agents.ts, especially ai-agents-crewai-tools-quiz. It identifies direct function or _run invocation, mocked HTTP, schema rejection, actionable errors, accurate descriptions, and async behavior as distinct test surfaces. That is also the practical boundary between this guide and function-calling LLM testing, which studies the model-generated call rather than the Python implementation.

A cold test does not claim that the whole agent works. It proves a smaller statement: for a defined input and dependency response, the tool produces the specified result or failure. That statement is repeatable, cheap to diagnose, and suitable for every code change. A later agent test can then assume the tool implementation already has basic contract coverage.

Which CrewAI tool contracts should a unit test protect?

Start from observable contracts, not framework internals. A useful tool test suite protects input validation, dependency interaction, returned shape, error semantics, side effects, and agent-facing metadata. Each item deserves an explicit expected behavior because a plain "does not raise" assertion can pass while the tool returns unusable data.

Input tests should cover required fields, types, boundaries, normalization, and prohibited values. If an args_schema says a query cannot be blank, assert that rejection occurs before the HTTP client is called. Schema changes are client changes because an agent constructs arguments from that contract. The same principle appears in contract testing tool schema evolution: an accidental rename can break callers even when the implementation still runs.

Result tests should assert stable fields and meanings rather than incidental serialization. For a lookup result, inspect the identifier, status, and items. For a command tool, assert both the returned receipt and the committed side effect. If the tool has an optional typed result schema, compare the Python object or dictionary that direct callers receive. Do not reduce the assertion to a substring when the contract is structured.

Failure tests need an agreed policy. A validation error, dependency timeout, not-found response, forbidden action, and malformed provider payload are different conditions. Decide which exceptions are allowed to cross the boundary and which become agent-actionable results. Then test that stack traces, secrets, and raw credentials are absent. Secure agent tool execution architecture adds authorization and policy checks around dangerous actions; the unit suite should prove the local enforcement points that architecture depends on.

Metadata is executable interface text for the eventual model. Assert that the name and description are present, that the description matches the action, and that a write tool does not sound like a read tool. This is not a semantic model evaluation. It is a contract check that prevents blank or stale metadata from reaching the agent layer.

Assign ownership to each contract before writing cases. The tool module owns argument normalization, calls to its adapter, local policy checks, and conversion into the published result. The adapter owns provider request and response details. The crew owns selection and orchestration. When a test crosses all three without naming the boundary, its failure can send three teams looking in different places. A short contract note beside the fixture prevents that ambiguity.

Version-sensitive behavior also needs an explicit home. CrewAI may change wrapper validation, result formatting, or async invocation across releases. A public-interface test should expose such a change during dependency review. Framework-neutral helper tests should continue to protect business rules. This combination lets a team adopt an intentional framework change without rewriting every local computation assertion.

How do @tool functions and BaseTool classes differ in tests?

The official custom tool guide documents two creation styles. The @tool decorator turns a function into a CrewAI tool with a name, description, inferred arguments, and an invocation interface. A BaseTool subclass defines attributes such as name, description, and args_schema, then supplies _run for synchronous behavior and, where needed, _arun for asynchronous behavior.

For a decorated function, keep most tests at the supported run interface. This includes decorator-generated validation and formatting that a raw helper call might skip. If the function delegates its business logic to an ordinary helper, test that helper independently and retain a small decorated-contract test. The helper remains framework-neutral, while the contract test proves the CrewAI wrapper exposes it correctly.

For a BaseTool, distinguish public behavior from implementation behavior. Calling run is appropriate when the test protects the public tool contract. Calling _run can be appropriate when diagnosing or isolating the subclass computation, but it bypasses wrapper behavior. Name such tests precisely so future maintainers do not mistake an _run pass for proof that schema validation and output handling also pass.

This layering prevents duplicated tests from pretending to add coverage. A public-interface case and an _run case are useful only if they prove different contracts. The same caution applies to AI agent tool argument correctness: agent-generated arguments belong above the tool's own schema and behavior tests.

How do you call a CrewAI tool directly in pytest?

The first example creates a typed result and a decorated tool, then invokes the tool without an agent, task, crew, API key, or model. The dependency is a module-level adapter so the test can patch the exact lookup used by production code.

# tools/search_kb.py
from pydantic import BaseModel
from crewai.tools import tool

from app.clients import kb_client


class SearchResult(BaseModel):
    query: str
    document_ids: list[str]


@tool("Search Knowledge Base", result_schema=SearchResult)
def search_kb(query: str) -> dict[str, object]:
    """Find approved knowledge-base documents for a user query."""
    rows = kb_client.search(query=query)
    return {
        "query": query,
        "document_ids": [row["id"] for row in rows],
    }

# tests/tools/test_search_kb.py
from unittest.mock import Mock

from tools.search_kb import search_kb


def test_search_kb_returns_document_ids(monkeypatch):
    search = Mock(return_value=[{"id": "KB-17"}, {"id": "KB-21"}])
    monkeypatch.setattr("tools.search_kb.kb_client.search", search)

    result = search_kb.run(query="refund window")

    assert result.query == "refund window"
    assert result.document_ids == ["KB-17", "KB-21"]
    search.assert_called_once_with(query="refund window")

Patch where the name is used. Patching a library's original definition will not replace a reference already imported into the tool module. This small detail accounts for many tests that unexpectedly contact a real service. The test also asserts the dependency call, which catches a tool that returns the right fixture after sending the wrong query.

Avoid creating a crew merely to reach the function. A crew adds orchestration and may add model configuration even though the assertion concerns two document IDs. If the decorated interface changes across a pinned CrewAI upgrade, the contract test should fail and direct you to the wrapper. The helper's own tests can remain unchanged.

Keep test data semantic. IDs such as KB-17 and a query such as refund window make a failure readable. Do not embed production records or secrets. If the tool filters by tenant or role, put those fields in the fixture and assert the dependency receives them.

How do you test args_schema validation and typed outputs?

args_schema is a boundary, not documentation. It should reject malformed input before the tool performs I/O. Build invalid cases for missing required values, blank strings, values outside allowed ranges, illegal enum members, and extra fields according to the schema policy. Assert that the client mock has no calls after each rejected input.

This example tests a BaseTool through its public interface and checks one direct implementation path. The public case proves validation and returned data. The _run case isolates deterministic normalization.

from typing import Type

import pytest
from crewai.tools import BaseTool
from pydantic import BaseModel, Field, ValidationError


class InventoryInput(BaseModel):
    sku: str = Field(min_length=3, pattern=r"^[A-Z0-9-]+$")


class InventoryTool(BaseTool):
    name: str = "Inventory Check"
    description: str = "Return current availability for one product SKU."
    args_schema: Type[BaseModel] = InventoryInput

    def _run(self, sku: str) -> dict[str, object]:
        normalized = sku.strip().upper()
        return {"sku": normalized, "available": normalized == "SKU-17"}


def test_inventory_tool_public_contract():
    result = InventoryTool().run(sku="SKU-17")
    assert result == {"sku": "SKU-17", "available": True}


def test_inventory_schema_rejects_blank_sku():
    with pytest.raises(ValidationError):
        InventoryInput(sku="")


def test_inventory_implementation_normalizes_sku():
    result = InventoryTool()._run(" sku-17 ")
    assert result["sku"] == "SKU-17"

If the public run interface returns a formatted representation rather than the raw Python value for the pinned CrewAI version, assert that documented contract and test the typed helper separately. Do not force a copied example to fit a different version. The repository should pin dependencies, and the test should describe the behavior of that pinned version.

Schema snapshots can be useful when an external agent host depends on the exact schema. Review snapshot changes rather than updating them automatically. For security-sensitive tools, pair schema checks with red-team datasets for agent tool abuse, but keep malicious model prompts outside the direct unit suite.

How do you mock HTTP, database, and file boundaries?

Mock the narrow boundary owned by another system. For HTTP, that is often an adapter method such as catalog.get_product, not the whole requests or aiohttp package. For a database, inject a repository or transaction interface. For files, use pytest's temporary directory when file semantics matter and mock only remote storage clients.

Representative dependency fixtures include success, empty result, not found, permission denied, timeout, malformed payload, and unavailable service. Each fixture should map to a documented tool result or exception. A timeout should not accidentally become an empty successful list, and a malformed payload should not expose a parser stack trace to an agent.

Side effects require stronger assertions. For a tool that creates a ticket, assert the exact request passed to the adapter, the idempotency key, and the returned receipt. Then test duplicate invocation with the same operation key. Idempotency and retry safety in agent tool calls explains why agent retries make this contract especially valuable.

Do not assert implementation trivia such as the number of private helper calls unless it is part of the contract. Assert what crossed the dependency boundary and what came back. This allows internal refactoring without weakening protection.

Keep one separate provider integration test when contract drift is plausible. It can run in a controlled environment with test credentials and known cleanup. Label it clearly so unit runs do not depend on a network, quota, or shared account. A live check complements mocks; it does not replace the error matrix that mocks can exercise reliably.

How do you test async tools and actionable failures?

Async tool tests should preserve async behavior from test to dependency. Use an async pytest plugin, await the tool's async path, and replace external coroutines with AsyncMock. Test successful completion, provider exception, cancellation if the application supports it, and cleanup of clients or temporary state.

CrewAI's documented async decorator accepts an async def function. In the current API, the decorated Tool exposes arun for direct asynchronous execution. The following example calls that public async path without creating an agent and verifies that the outbound coroutine receives the normalized argument.

# tools/fetch_stock.py
from crewai.tools import tool

from app.clients import catalog_client


@tool("Fetch Stock")
async def fetch_stock(sku: str) -> dict[str, object]:
    """Fetch current stock for one product SKU."""
    return await catalog_client.fetch_stock(sku=sku)

# tests/tools/test_fetch_stock.py
from unittest.mock import AsyncMock

import pytest

from tools.fetch_stock import fetch_stock


@pytest.mark.asyncio
async def test_fetch_stock_awaits_catalog_client(monkeypatch):
    fetch = AsyncMock(
        return_value={"sku": "SKU-17", "available": 8}
    )
    monkeypatch.setattr(
        "tools.fetch_stock.catalog_client.fetch_stock",
        fetch,
    )

    result = await fetch_stock.arun(sku="SKU-17")

    assert result == {"sku": "SKU-17", "available": 8}
    fetch.assert_awaited_once_with(sku="SKU-17")

Add a second case with fetch.side_effect set to the application's timeout exception. Assert the exception or structured failure promised by the tool contract. AsyncMock.assert_awaited_once_with matters here because an unawaited coroutine can exist without proving that the dependency completed.

An actionable failure tells the caller what category occurred without exposing internal secrets. For example, a search tool can report that the knowledge service is temporarily unavailable and include a stable error code. It should not return a bearer token, raw response headers, database connection string, or full stack trace. The exact error model is an application decision, so encode it in a type and assert it.

Do not catch every exception and return generic success-shaped text. That makes the tool appear to have completed and leaves the agent to infer failure from prose. Either raise a documented exception or return a discriminated result such as {"ok": false, "error_code": "KB_TIMEOUT"}. Then test both branches structurally.

An agent may retry a failed tool. That policy belongs in the orchestration layer, but the tool must provide enough stable information for the policy to act. Agent tool-call trace grading can later verify the trajectory, including whether the agent retried or stopped. The unit test only proves the error emitted by the tool.

CrewAI agents also have iteration controls, but do not use an iteration cap as a substitute for a correct tool failure. The cap limits damage after repeated calls. It does not make a misleading error result acceptable.

What belongs in unit, integration, and agent-level suites?

Unit tests own schema validation, local computation, adapter calls, result types, error conversion, metadata presence, and side-effect preparation. They run without provider credentials. Integration tests own compatibility with an actual dependency, transport, authentication, and serialization across the boundary. Agent-level tests own selection, argument generation, interpretation, retry, and stopping behavior.

Put a case at the lowest layer that can prove it. A malformed SKU does not need an LLM. A question about whether the agent chooses inventory instead of order history does. A question about whether the inventory sandbox accepts the current request format belongs in integration. This allocation lowers diagnosis time because each suite has one reason to fail.

Tool selection correctness needs a dataset of user intents and expected choices. It should use a controlled or already-tested tool implementation so a tool bug does not contaminate selection scores. Tool schema evolution needs compatibility fixtures. It can reuse the schema exported by the same unit-tested object.

Avoid exact final prose assertions in agent-level tests unless the prose itself is a regulated contract. Prefer required facts, tool trajectory, policy decisions, and structured output. Keep exact equality for deterministic tool results where it is meaningful.

A release suite can therefore contain many local tool cases, a smaller set of dependency checks, and focused model-backed scenarios. No invented ratio is required. Choose coverage from risk, side effects, change frequency, and the cost of a missed failure.

How do you build a repeatable CrewAI tool test workflow?

Use the following workflow for each new or changed tool:

1. Write the input, output, failure, side-effect, and metadata contracts in concrete terms.
2. Identify the public CrewAI invocation surface and any framework-neutral helper.
3. Create semantic fixtures for valid, boundary, invalid, empty, and dependency-failure cases.
4. Inject or patch outbound HTTP, database, file, queue, and clock dependencies at their lookup point.
5. Assert the successful Python value and the exact dependency request.
6. Assert schema rejection occurs before side effects.
7. Assert each dependency failure becomes the documented exception or structured error.
8. Await async paths and verify cleanup or cancellation behavior where applicable.
9. Add one controlled provider integration check when external compatibility is material.
10. Add model-backed selection and use scenarios only for behavior that a direct call cannot prove.

Review failures by layer. If a direct tool case fails, fix the tool or its contract before rerunning a crew. If only the agent scenario fails, inspect names, descriptions, prompt context, argument generation, and trajectory. This order prevents model tuning from hiding ordinary software defects.

Preserve evidence that helps review: the case ID, normalized input, dependency fixture, structured result, and sanitized error category. Do not preserve secrets or unrelated payload content. The aim is a small record that lets another engineer reproduce the assertion.

FAQ

Can a CrewAI tool run without an agent?

Yes. Invoke the tool's supported Python interface with explicit arguments. This proves computation, validation, dependency handling, and returned shape without an Agent, Task, or Crew. Add an agent only when the expected behavior concerns selection, generated arguments, interpretation, retry, or stopping.

Should tests call _run or run?

Use run for the public CrewAI contract because it can include validation and formatting. Use _run only for a deliberately isolated BaseTool implementation test. A useful suite may contain both, but their names and assertions should show that they protect different layers.

How do you test an @tool decorated function?

Import the decorated object, call its documented run interface, and assert result, schema, description, and errors. If business logic lives in a plain helper, test that helper separately. Patch dependencies in the production module where the decorated function resolves them.

How should paid APIs be mocked?

Mock the adapter method that represents the provider and return reviewed fixtures for success and failure. Assert the outbound request as well as the tool result. Keep a separate integration test with test credentials for provider compatibility, and never place those credentials in unit fixtures.

Do async CrewAI tools need async pytest support?

Yes. Run them in an async test, await the supported path, and use async fakes for I/O. Include error and cancellation behavior when those are part of the application contract. A synchronous wrapper can conceal event-loop and cleanup defects, so it should not replace direct async coverage.

When is an LLM call necessary?

An LLM is necessary when the test asks what the model will choose, generate, infer, retry, or say. It is not necessary to prove that a tool validates a SKU, maps a provider response, denies an action, or emits a stable structured failure.

Conclusion: Keep tool code and agent behavior independently testable

Direct CrewAI tool tests establish a dependable base for agent testing. They prove Python behavior, schemas, dependency boundaries, typed results, errors, and side effects without provider variability. Agent tests can then focus on the genuinely model-dependent questions instead of rediscovering ordinary defects through an expensive execution path.

Start with one tool and write its success, rejection, and dependency-failure cases. Then add the smallest integration check and the smallest agent-use scenario that prove what the unit layer cannot. That separation makes the suite easier to interpret and makes every model-backed run carry a clear purpose.

Review the tool description and schema whenever behavior changes, not only when Python signatures change. An implementation that adds a side effect while retaining a read-only description creates an agent-facing contract defect. The cold suite is the earliest place to make that mismatch visible.