When your agent chooses almost the right tools

An agent completes a booking, then sends the confirmation twice. The end-to-end test sees a valid reservation and passes, but the customer receives two emails and the audit log records an action nobody requested. Ragas ToolCallF1 can compare the structured trace with expected calls through precision and recall. That score helps only when the suite also preserves which call was missed, added, or malformed, because it cannot rank a harmless lookup against a second payment or prove a safe order.

See exactly what the score counts

The unit under comparison is a structured tool call, not the assistant's prose and not the tool's returned value. The current Ragas documentation shows the modern import as ragas.metrics.collections.ToolCallF1. The scorer receives a conversation in user_input and the expected calls in reference_tool_calls. It extracts calls from the AI messages and compares each call's name and arguments with the reference.

Three counts explain the result:

• A true positive is an observed call whose name and arguments match an expected call.
• A false positive is an observed call that has no expected match.
• A false negative is an expected call that the agent did not make.

Precision answers, "Of the calls the agent made, how many belonged in the plan?" Recall answers, "Of the calls the plan required, how many did the agent make?" F1 combines those two rates. If either side is weak, the combined score falls.

The documented matching is unordered. That is a feature when independent lookups may run in either sequence. It is a blind spot when order carries meaning. A trace containing charge_card before confirm_inventory can receive the same unordered result as the safe sequence if the names and arguments otherwise match. Keep sequence validation outside this metric whenever the workflow has prerequisites.

Argument matching is equally important. lookup_customer({"email": "sam@example.com"}) and lookup_customer({"email": "pat@example.com"}) are not the same action. Normalizing away account identifiers to make scores look stable destroys the very evidence a QA engineer needs. Normalize only fields that the product contract declares irrelevant, such as a generated correlation identifier, and retain both raw and normalized artifacts.

Here is a complete use of the modern API, following the shape in the official Ragas agent-metrics documentation:

import asyncio

from ragas.messages import AIMessage, HumanMessage, ToolCall
from ragas.metrics.collections import ToolCallF1


async def main() -> None:
    conversation = [
        HumanMessage(content="Find order A-104 and email its receipt."),
        AIMessage(
            content="I will find the order first.",
            tool_calls=[
                ToolCall(name="get_order", args={"order_id": "A-104"}),
            ],
        ),
        AIMessage(
            content="The order exists, so I will send the receipt.",
            tool_calls=[
                ToolCall(name="send_receipt", args={"order_id": "A-104"}),
            ],
        ),
    ]
    expected = [
        ToolCall(name="get_order", args={"order_id": "A-104"}),
        ToolCall(name="send_receipt", args={"order_id": "A-104"}),
    ]

    result = await ToolCallF1().ascore(
        user_input=conversation,
        reference_tool_calls=expected,
    )
    print(result.value)


if __name__ == "__main__":
    asyncio.run(main())

This metric does not need an LLM judge for the comparison shown above. That removes grader drift, but it does not remove test-data errors. The reference list is an executable statement of what the agent was supposed to do. If that list is incomplete, the metric will confidently penalize a valid call. If it includes an unsafe call, the metric will reward the unsafe behavior.

Do not read a corpus average without its counts. Consider ten cases with one expected call each. Nine exact matches and one case with an extra destructive call can still produce a reassuring aggregate. The product risk lives in the failing row, not in the decimal printed at the top of the report. Store the case identifier, expected calls, observed calls, and mismatch category beside every score.

Work through failures that need different fixes

Suppose a support agent must fetch an invoice and send it to the email address already verified on the account. The expected plan contains get_invoice followed by send_invoice. A new system prompt encourages the model to be "helpful," and it also calls update_contact_email using an address copied from the chat.

The extra call is a precision failure. Improving recall will not help because the required calls were already present. The likely fixes are to narrow the tool catalog for this task, strengthen the policy around profile changes, or require explicit confirmation before exposing the mutation tool. The regression test should also fail directly on any unapproved update_contact_email call. F1 is useful evidence, but a forbidden-action assertion is the release gate.

Now take a refund workflow. The agent calls the correct issue_refund tool but passes {"amount_cents": 8900} when the approved amount is 9800. A dashboard that groups only by tool name reports success. ToolCallF1 should treat the call as unmatched when the arguments differ, producing both a missing expected call and an unexpected observed call in the conceptual confusion counts.

That second failure is not tool selection. It may come from entity extraction, stale context, currency conversion, or a mapping bug between the model's arguments and the executor payload. The quickest diagnostic is to compare the raw model call with the payload received by the tool. If the model emitted 9800 and the executor logged 8900, tuning the model is wasted effort. If both contain 8900, inspect the conversation turn where the amount was established.

A third case looks like poor recall but is really a bad oracle. A travel agent can find an airport either with search_airport_by_city or with resolve_iata_code. The reference contains only the first route, while the agent uses the second and reaches the correct airport. Unordered matching cannot infer that these plans are semantically equivalent. The row fails because the label permits one implementation, not because the product failed.

There are only three honest responses to that ambiguity. You can define one canonical tool contract and require every agent to follow it. You can maintain separate cases for accepted plans and evaluate the agent against the plan selected by an upstream router. Or you can stop using exact tool-call F1 as the release oracle for that scenario and judge the resulting state. Picking the highest score across several hand-written references is possible in a custom harness, but it changes the evaluation definition and must be documented as such. Do not imply that ToolCallF1 accepts alternative reference plans unless the installed version's API explicitly supports them.

Repeated calls deserve their own case. Retries can create two identical trace entries even when the executor deduplicates the side effect with an idempotency key. The customer outcome may be safe while latency and cost still regress. Conversely, a trace collector might collapse identical calls and hide a real duplicate execution. Because public metric descriptions do not fully define every duplicate-matching edge case, characterize the exact library version you run and assert call counts independently.

This diagnostic comparator is intentionally separate from Ragas. It preserves duplicate calls with a multiset, emits the exact missing and extra objects, and makes no claim about reproducing private implementation details:

from collections import Counter
import json
from typing import Any


def key(call: dict[str, Any]) -> str:
    return json.dumps(
        {"name": call["name"], "args": call["args"]},
        sort_keys=True,
        separators=(",", ":"),
    )


def compare_calls(
    expected: list[dict[str, Any]],
    observed: list[dict[str, Any]],
) -> dict[str, list[dict[str, Any]]]:
    expected_by_key = {key(call): call for call in expected}
    observed_by_key = {key(call): call for call in observed}
    expected_counts = Counter(map(key, expected))
    observed_counts = Counter(map(key, observed))

    missing = [
        expected_by_key[item]
        for item, count in (expected_counts - observed_counts).items()
        for _ in range(count)
    ]
    extra = [
        observed_by_key[item]
        for item, count in (observed_counts - expected_counts).items()
        for _ in range(count)
    ]
    return {"missing": missing, "extra": extra}


if __name__ == "__main__":
    reference = [
        {"name": "get_invoice", "args": {"invoice_id": "INV-7"}},
        {"name": "send_invoice", "args": {"invoice_id": "INV-7"}},
    ]
    actual = [
        {"name": "get_invoice", "args": {"invoice_id": "INV-7"}},
        {"name": "send_invoice", "args": {"invoice_id": "INV-7"}},
        {"name": "send_invoice", "args": {"invoice_id": "INV-7"}},
    ]
    print(json.dumps(compare_calls(reference, actual), indent=2))

The output identifies one extra send_invoice object. That is more actionable than "F1 below threshold." It tells the owner to inspect retry handling, response timeouts, or duplicate planning around that call. For the wrong refund amount, the same output displays one missing call with 9800 and one extra call with 8900, which points toward argument construction.

Prove it is a tool mismatch before changing the agent

Start with the raw trace at the boundary nearest the model. You need the assistant message that contains the structured tool call, not a natural-language summary produced later. Preserve the tool name, complete arguments, turn index, provider response identifier, and timestamp. Secrets should be redacted by field name, but redaction must be stable so expected and observed values remain comparable.

Then capture the executor boundary. A common pipeline parses the model response, validates it against a schema, applies defaults, and sends a new payload to the tool. Each step can change the evidence. If Ragas sees the model message while the production failure happens after schema coercion, the evaluation and the product are observing different objects.

Use three questions to locate the defect:

1. Did the model emit the expected structured call?
2. Did the adapter preserve its name and arguments?
3. Did the executor run it once and return the recorded result?

If the first answer is no, the problem belongs to prompting, model behavior, tool descriptions, or conversation state. If the first is yes and the second is no, fix the adapter. If the first two are yes but the executor ran twice, inspect retries and idempotency. A tool-call metric over model messages cannot diagnose an executor retry it never receives.

The console output for a useful failing case should name the row and the mismatch. This sample is illustrative, not a measurement from a real suite:

$ python -m pytest -q tests/evals/test_tool_contract.py::test_tool_contract
F                                                                        [100%]
=================================== FAILURES ===================================
________________________ test_tool_contract[refund-amount] __________________
missing: [{"name":"issue_refund","args":{"amount_cents":9800,"order_id":"A-104"}}]
extra:   [{"name":"issue_refund","args":{"amount_cents":8900,"order_id":"A-104"}}]
assert 8900 == 9800
1 failed in 0.08s

That shape separates an argument mismatch from a missing trace. When both lists are empty but the Ragas call fails, inspect message conversion and library versions. When the raw trace contains no tool calls, confirm that the agent actually ran with tools enabled and that the collector captured assistant tool-call messages. A blank observation is an instrumentation failure until proven otherwise.

Also check reference freshness. Tool schemas evolve. A field may have moved from customer_id to account_id, or a default may now be supplied by the executor. Keep the schema version and agent version with each evaluation artifact. Without them, a failed exact match can look like a model regression even though the contract changed.

Do not diagnose from the rounded F1 value. Two cases can share the same score while needing opposite fixes. Low precision suggests extras; low recall suggests omissions. Wrong arguments often create one of each. Report those components and mismatch objects. If the public result object in your pinned Ragas version exposes only value, calculate diagnostic counts in your harness from the saved calls rather than inventing undocumented result fields.

Separate a bad agent call from a bad trace join

One failure can produce the same missing-and-extra pair through two unrelated defects. Suppose case A-104 expects get_order with order A-104, but its report shows an extra get_order for B-209 and a missing call for A-104. The agent may have carried B-209 from stale conversation context. The evaluation collector may instead have attached another case's perfectly valid trace to A-104. The mismatch objects look identical after the join, so changing the prompt before checking provenance can make the incident harder to reproduce.

The separating evidence is the raw assistant response identifier paired with the generation request, not the final score row. Start from the request for A-104 and follow its case identifier through prompt assembly, provider response capture, message conversion, and scoring input. If the raw response associated with that request already contains B-209, the defect is upstream of collection. Inspect the conversation turns and context selection passed to the model. If the raw response contains A-104 while the scorer receives B-209, the collector or artifact join changed ownership. If the saved response containing B-209 belongs to a request whose prompt also names B-209, the response is healthy and the case-to-trace join is broken.

A useful diagnostic record makes those alternatives visible beside each other. Read the case identifier first, then the generation request identifier, response identifier, trace artifact digest, expected-call count, observed-call count, and the full mismatch objects. In a healthy row, one case flows through every identifier, the trace digest refers to the artifact opened by the report, and the observed call belongs to the request text. In the broken join, the score row says A-104 while one of those ownership fields points to B-209's generation. A misleading row can have matching timestamps and a plausible tool name because parallel cases ran within the same second and invoked the same tool. Time proximity and name equality are not ownership evidence.

This check also catches a subtler reporting defect. Some harnesses aggregate scores correctly but build drill-down links from array position after filtering failed generations. The corpus number is then computed from one set of rows while the UI opens another row's trace. Compare the case identifier inside the scored artifact with the identifier in the link target. A prompt change cannot repair a report that displays the wrong evidence.

Build references that survive real product changes

Good labels describe obligations, not the path one engineer happened to observe during test creation. For each case, write the user intent, required calls, forbidden calls, argument rules, sequence rules, and expected outcome as separate fields. ToolCallF1 consumes the required calls. Other assertions consume the rest.

Volatile values need explicit treatment. A server-generated request ID can be excluded from comparison if it is not chosen by the agent. A destination account, refund amount, permission scope, or file path usually cannot. Put normalization in named functions, review those functions like production code, and save normalized output next to raw output.

The following test demonstrates a layered contract. It uses pytest's documented parametrization feature, checks high-risk rules directly, and leaves the aggregate threshold visible in the case data. The F1 values are illustrative inputs to the gate, not claimed results from an experiment:

from collections import Counter

import pytest


CASES = [
    {
        "id": "invoice-happy-path",
        "f1": 1.0,
        "observed_names": ["get_invoice", "send_invoice"],
        "forbidden": {"update_contact_email"},
        "minimum_f1": 1.0,
    },
    {
        "id": "read-only-search-variation",
        "f1": 0.8,
        "observed_names": ["search_orders", "get_order", "search_help"],
        "forbidden": {"issue_refund", "delete_order"},
        "minimum_f1": 0.75,
    },
]


@pytest.mark.parametrize("case", CASES, ids=lambda case: case["id"])
def test_tool_contract(case: dict[str, object]) -> None:
    names = Counter(case["observed_names"])
    forbidden = set(case["forbidden"])

    assert forbidden.isdisjoint(names), (
        f"forbidden calls observed: {forbidden.intersection(names)}"
    )
    assert names["send_invoice"] <= 1, "receipt-like side effect repeated"
    assert float(case["f1"]) >= float(case["minimum_f1"])

This split matters during triage. A threshold failure in the read-only search slice can go to review. A forbidden refund call fails immediately even if the rest of the trace produces a high F1. Risk is not linear, so the gate should not pretend it is.

Reference review needs two people when side effects matter: someone who understands the user journey and someone who owns the tool contract. Product knowledge catches valid alternate plans. API knowledge catches stale fields and unsafe defaults. Record why a call is required, not just that it appeared in a successful trace.

Add negative cases before adding volume. A case where the agent must ask for confirmation is more revealing than twenty paraphrases of the happy path. Include missing identifiers, conflicting amounts, unavailable inventory, denied permissions, timeouts before execution, timeouts after execution, and ambiguous requests. Each one exercises a different boundary.

Defaulted arguments need a deliberate comparison rule. Imagine that send_invoice accepts an optional locale and the executor supplies en-US when the model omits it. A reference that contains {"locale": "en-US"} will not have the same raw shape as a model call that leaves the field out, even if execution produces the same customer-facing result. Decide whether the test is about model output or executor input. If it is about model output, omission may be the intended behavior. If it is about the executed request, capture the post-default payload and label against that boundary.

Apply the same care to null values, empty arrays, numeric strings, and object key order. JSON object key order should not carry business meaning, while list order often does. The string "10" may or may not be equivalent to the number 10 under the tool schema. Do not add a broad “make JSON similar” cleanup pass. Write field-specific canonicalization backed by the schema, and add unit cases that prove both accepted equivalence and rejected differences.

Cases with no expected tool calls are especially useful for refusal and clarification behavior. They are also an edge case for any precision and recall formula because a denominator can be zero. Do not assume how a Ragas release scores an empty reference and empty observation. Add a characterization test for the pinned version, then keep a direct assertion that protected tools were not called. The product decision is clear even if the aggregate metric's convention changes.

Protect the evaluation set from accidental training and prompt tuning. Engineers naturally fix visible failures, but repeated tuning against every release case turns the suite into a memorized checklist. Keep a small debugging set that developers can inspect, a broader regression set with controlled access, and fresh incident-derived cases. Report the provenance and last review date of each row. This costs curation time, yet it gives a passing score meaning beyond “the prompt learned these exact examples.”

Roll the gate into CI without hiding the bad row

Begin in report-only mode against a pinned model, prompt, tool schema, Ragas version, and dataset revision. Save per-case JSON for several runs. The purpose is not to manufacture a universal baseline. It is to find unstable cases, incorrect references, missing telemetry, and slices with different risk.

Next, make deterministic policy failures blocking. Examples include forbidden tools, malformed required arguments, repeated non-idempotent calls, and missing confirmation before a protected action. Keep F1 advisory until reviewers have resolved labeling defects. Once the dataset is trusted, gate critical slices separately and use a lower-severity trend check for broad exploratory cases.

Never gate only on the mean. At minimum, publish the number of evaluated rows, exact-match rows, false-positive calls, false-negative calls, and failures by tool family. A stable mean can conceal a transfer regression if easier search cases dominate the set. Weighting the mean does not solve invisibility; retain the row-level report.

This workflow is intentionally small. It installs a locked environment, runs the dedicated suite, writes JUnit output, and uploads the artifact directory even when a test fails:

name: agent-tool-evaluation

on:
  pull_request:
    paths:
      - "agent/**"
      - "evals/tool_calls/**"
      - "requirements.lock"

jobs:
  tool-contract:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements.lock
      - run: python -m pip install -r requirements.lock
      - run: mkdir -p artifacts
      - run: python -m pytest tests/evals/test_tool_contract.py --junitxml=artifacts/tool-calls.xml
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: tool-call-evidence
          path: artifacts/

The cache-dependency-path entry is load bearing when the lock file is named requirements.lock. Pip caching otherwise resolves **/requirements.txt with **/pyproject.toml as a backup, matches neither, and stops the job at setup before a single tool call is scored.

Model-backed fixture generation should not run silently inside every assertion. Capture the agent trace in an explicit step, then score the immutable artifact. That makes reruns cheaper and distinguishes "the agent changed its answer" from "the scorer changed its interpretation." Since ToolCallF1 is deterministic over supplied structured calls, repeated score changes with identical artifacts point to version or conversion changes.

Roll out by tool family. Read-only retrieval is a safer first slice than payments or deletion. Once the harness proves it preserves arguments and duplicates, add side-effecting tools with strict policy assertions. Finally, add scheduled runs for a larger, slower corpus. Pull requests need fast feedback; nightly jobs can cover model variability with repeated generations.

Every waiver needs an expiry and a row owner. A blanket xfail on an unstable dataset slowly converts a gate into decoration. If a case is genuinely nondeterministic, quarantine it from blocking totals while keeping its artifacts visible. Fix the label or the product, then restore it.

Migrate an existing suite without moving the goalposts

An established suite usually has loaders, snapshots, dashboards, and alert rules built around one scalar. Those consumers tend to break before the scorer does. Land the row-level result schema and provenance capture first, while continuing to publish the old scalar unchanged. Make missing provenance explicit for historical fixtures. Do not manufacture response identifiers for artifacts created before that evidence existed, and do not count those rows as provenance passes.

Next, teach report consumers to read both the scalar and the mismatch details. Run the old and new reporting paths over the same immutable traces. Their top-level values should agree because the metric input has not changed, while the new path should add counts and ownership fields. Resolve discrepancies in conversion or aggregation before changing any threshold. Changing capture, labels, and release policy in one pull request leaves no stable boundary for diagnosis.

Only after the readers are compatible should the generation step begin writing the new trace envelope. Backfill reference schema versions where the source is known, add characterization cases for duplicates and empty references, then freeze a dataset revision for the first compatibility comparison. Version the artifact envelope independently from the dataset so a reader can reject a newer shape instead of dropping unknown fields. Roll the writer through one evaluation slice at a time and require identical row coverage as well as numeric agreement. Remove the scalar-only path after saved artifacts from the compatibility window have expired or been migrated.

The extra evidence has a concrete price. Retaining both raw assistant messages and normalized call objects stores two representations of each evaluated turn, increases artifact upload time, and expands the redaction surface. Separating generation from scoring adds a pipeline stage, although rescoring an immutable trace avoids paying generation latency on every diagnostic rerun. The trace format and field-specific redaction rules also become maintained interfaces rather than incidental debug output.

Ownership follows the first divergent boundary. The agent team owns a wrong call already present in the raw response. The adapter team owns a correct raw response converted into the wrong structured call. The evaluation-platform team owns a trace joined to the wrong case. The tool owner approves schema equivalence and argument normalization, while the product owner decides whether an alternate plan is acceptable. A handoff should contain the case and dataset revisions, request and response identifiers, raw and normalized calls, tool schema version, the first boundary where values differ, and a one-case replay command or fixture. A screenshot of the F1 number is not a sufficient handoff.

Know when this metric is the wrong oracle

Do not use unordered F1 to prove a required sequence. Authentication before data access, inventory confirmation before charging, and approval before deletion need an ordered state-machine assertion. The same bag of calls can represent a safe workflow or an incident.

Avoid exact argument matching when the task intentionally allows many equivalent values and you cannot define a sound normalizer. Search queries, free-form summaries, and generated filenames often have several valid forms. A brittle reference will measure wording choices instead of product quality.

Skip a tool-path gate when only the achieved outcome matters. An agent may legitimately gain a better tool or consolidate two calls into one. In that case, validate the final state, permissions, side effects, and user-visible answer. Keep tool-call evaluation as diagnostic coverage, not as a permanent architecture lock.

Finally, do not let F1 certify safety. It says how closely a trace matches a labeled set of calls. It does not prove authorization, factual correctness, successful execution, idempotency, latency, or customer satisfaction. Pair it with those checks where the risk exists, and let each oracle answer one question well.

It also does not inspect what the agent says after the tools return. An agent can make every expected call with exact arguments, then expose a confidential field from the tool response in its final message. Tool-call F1 remains perfect because the leak occurs outside the compared objects. Add response-content and data-handling checks for that failure; no stricter call threshold will reveal it.