Test the retriever and the agent without blending their failures

An assistant retrieves the correct cancellation policy, then calls close_account instead of cancel_subscription. The answer sounds confident and cites the right document, but the action is wrong. A blended dashboard score can make that release look acceptable because strong retrieval masks a dangerous tool failure.

Treating a RAG pipeline and an agent as one evaluator target destroys the most useful clue: where the defect began. Retrieval metrics should judge the evidence selected for a response. Agent metrics should judge structured actions or the achieved outcome. A release may require both lanes to pass, but their scores should never pay off each other's failures.

Where to split the evaluation contract

A typical RAG request has observable stages: the user input becomes a retrieval query, candidate documents are filtered and ranked, selected contexts reach the model, and the model produces a response. An agent adds another control loop. It interprets the request, selects a tool, supplies arguments, receives a tool result, decides whether another action is needed, and eventually stops.

Those stages create different test oracles. For retrieval, a reference answer and retrieved contexts can support entity coverage checks such as ContextEntityRecall. For an agent, a message history containing actual structured calls can be compared with reference_tool_calls using ToolCallF1 or ToolCallAccuracy. If the business cares only about the final outcome, a goal metric may fit better than a fixed call sequence.

Tool names such as cancel_subscription in the worked cases are application-owned fixture names, not Ragas APIs. The Ragas classes, inputs, and result access shown in code follow the current official collections API.

Ragas documents these as separate metric families. ContextEntityRecall compares entities in reference with entities in retrieved_contexts. The current collections version uses an evaluator model and returns the score through result.value. ToolCallF1 compares actual tool call objects in the conversation with expected tool call objects. It uses unordered matching and reflects precision and recall over exact call names and arguments. ToolCallAccuracy also examines tool calls, with strict order enabled by default and an option for flexible order.

The input schemas tell you why the values should not be blended. Retrieval needs prose evidence and a reference. Tool evaluation needs structured calls and an expected action contract. A context score of 1 cannot prove the correct tool executed. A tool score of 1 cannot prove the agent received sufficient evidence or produced a grounded explanation.

Define two release lanes even when one request exercises both:

The outcome row is separate because tool-call conformity and user success are not identical. A workflow can reach the right state through an unauthorized tool. It can also reach the right state through a safe alternative path not listed in an overfitted reference. Decide which paths are requirements before choosing the metric.

Never publish a formula such as (retrieval_score + tool_score) / 2 as release quality. The units differ, the risks differ, and a severe zero can become a comfortable average. Store each metric by name, case, and lane. Apply policy after the facts are visible.

How to build cases that locate the first defect

Start with a case contract written before the run. It should name the user goal, permitted data scope, approved reference evidence, allowed or required tool behavior, expected final state, and the artifacts to retain. Avoid encoding implementation detail as truth unless the detail is itself a safety or business requirement.

Three paired examples expose mistakes that a single end-to-end score misses.

Correct evidence, wrong destructive tool

A user asks to cancel monthly renewal while retaining access through the paid period. Retrieval returns the correct policy: cancellation stops future renewal but does not close the account. The agent calls close_account with the user's identifier. The final natural-language response may still say the subscription was cancelled.

The retrieval lane should pass if the contexts contain the required policy entities. The tool lane must fail because the actual call name differs from cancel_subscription. The outcome lane should also check the durable state, since a tool message claiming success is not proof that the intended state exists.

This is a good use of an exact tool reference. The two tools have materially different consequences, so treating them as equivalent would weaken the contract. Keep the expected call arguments narrow enough to matter, including subscription ID or effective mode, but do not include volatile fields such as a generated trace ID unless the product contract requires them.

Weak evidence, correct action by accident

Now imagine the retriever returns a generic cancellation FAQ without the customer's plan or jurisdiction. The agent still calls cancel_subscription with the right ID because the model learned the common path or the application supplies defaults. Tool F1 can pass. The workflow might even reach the expected final state.

Retrieval should still fail if the decision required jurisdiction-specific evidence. A successful action does not make the missing evidence safe. The next jurisdiction may have a notice period, refund rule, or approval step that the generic path ignores.

This case is particularly important in regression suites because memorized or default behavior can conceal retrieval failures. Force the case to preserve retrieved document IDs and decision inputs. Do not infer that evidence was used merely because the action happened to match the reference.

Different tool path, valid outcome

A travel agent must obtain weather and UV information. The reference lists weather_lookup followed by uv_lookup. A new orchestration version calls travel_conditions, an approved composite tool that returns both. The user receives the correct result, and the final state is valid, but exact tool comparison fails.

That failure may belong to the dataset rather than the product. If the requirement is "use these two tools in this order," keep the strict reference. If the requirement is "obtain current weather and UV data through approved read-only tools," represent permitted alternatives or evaluate the goal. Do not repeatedly edit expected calls after every implementation change without reviewing the policy they encode.

The case record should make these ownership decisions explicit. A useful schema contains case_id, case_revision, retrieval_reference, retrieved_contexts, retrieved_document_ids, actual_messages, reference_tool_calls, expected_outcome, corpus_version, and the versions of both evaluators. Store raw lane results beside a disposition, not in place of them.

The following deterministic validator catches fixture defects before any metric runs. It does not call Ragas and does not decide semantic quality. It checks that a combined scenario has both independent contracts.

from typing import Any


def validate_case(case: dict[str, Any]) -> None:
    required = {
        "case_id",
        "case_revision",
        "retrieval_reference",
        "retrieved_contexts",
        "actual_messages",
        "reference_tool_calls",
        "expected_outcome",
        "corpus_version",
    }
    missing = sorted(required - case.keys())
    if missing:
        raise ValueError(f"case is missing required fields: {', '.join(missing)}")
    if not isinstance(case["retrieved_contexts"], list):
        raise TypeError("retrieved_contexts must be a list")
    if not isinstance(case["reference_tool_calls"], list):
        raise TypeError("reference_tool_calls must be a list")
    if not case["retrieval_reference"].strip():
        raise ValueError("retrieval_reference must not be empty")


validate_case(
    {
        "case_id": "subscription-cancel-001",
        "case_revision": "3",
        "retrieval_reference": "Cancel renewal but keep the account open.",
        "retrieved_contexts": ["Cancellation stops renewal and preserves access."],
        "actual_messages": [{"role": "user", "content": "Cancel my renewal"}],
        "reference_tool_calls": [
            {"name": "cancel_subscription", "args": {"subscription_id": "sub_42"}}
        ],
        "expected_outcome": {"renewal": "cancelled", "account": "open"},
        "corpus_version": "policies-2026-08-01",
    }
)

Fixture validation is unglamorous, but it prevents a missing reference from being recorded as a model failure. It also catches the common mistake of creating retrieval rows for some cases and tool rows for others, then averaging across different populations.

How to run the metrics without combining them

Use the collections imports for new code. The official Ragas pages mark the older metric imports as legacy and recommend migrating to ragas.metrics.collections. Direct ascore calls make the required fields visible and are easy to wrap with your own artifact capture.

The next example evaluates one retrieval lane and one tool lane. It deliberately returns a nested report. No overall_score field exists.

import asyncio
import os

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.messages import AIMessage, HumanMessage, ToolCall
from ragas.metrics.collections import ContextEntityRecall, ToolCallF1


async def evaluate_case() -> dict:
    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    evaluator = llm_factory("gpt-4o-mini", client=client)

    retrieval_metric = ContextEntityRecall(llm=evaluator)
    retrieval_result = await retrieval_metric.ascore(
        reference=(
            "Cancel subscription sub_42 without closing Anika Shah's account."
        ),
        retrieved_contexts=[
            "Subscription sub_42 belongs to Anika Shah.",
            "Cancellation stops renewal while the customer account remains open.",
        ],
    )

    messages = [
        HumanMessage(content="Cancel renewal for subscription sub_42."),
        AIMessage(
            content="I will close the account now.",
            tool_calls=[
                ToolCall(name="close_account", args={"subscription_id": "sub_42"})
            ],
        ),
    ]
    expected_calls = [
        ToolCall(name="cancel_subscription", args={"subscription_id": "sub_42"})
    ]
    tool_result = await ToolCallF1().ascore(
        user_input=messages,
        reference_tool_calls=expected_calls,
    )

    return {
        "case_id": "subscription-cancel-001",
        "retrieval": {
            "metric": "context_entity_recall",
            "score": retrieval_result.value,
        },
        "agent_tools": {
            "metric": "tool_call_f1",
            "score": tool_result.value,
        },
    }


if __name__ == "__main__":
    print(asyncio.run(evaluate_case()))

The exact tool mismatch in this fixture is intentional: the expected and actual names differ, so the tool lane should expose it. The retrieval value is evaluator-dependent, so do not copy a guessed number into an assertion. Capture the actual result and compare it through a reviewed policy.

ToolCallF1 is unordered. That makes it useful when presence and correctness of calls matter more than sequence. It also means it cannot enforce "authorize before transfer" or "search before purchase." For required ordering, the Ragas ToolCallAccuracy metric uses strict order by default. Set strict_order=False only when your workflow truly permits reordering, such as independent lookups.

Exact argument matching has a trade-off. It catches a wrong location, account ID, or amount, but it can also penalize harmless representation differences. Normalize upstream representations only when the product contract defines them as equivalent. For example, converting a canonical ISO date to another accepted canonical form may be reasonable. Lowercasing every string could hide case-sensitive identifiers.

Do not reconstruct actual tool calls from assistant prose. The message history should contain structured ToolCall objects produced by the runtime trace. If the agent says "I checked the weather" but no call exists, that is not a weather call. Likewise, a tool request is not evidence the tool succeeded. Keep tool results and final state for outcome evaluation.

For a dataset, Ragas supports collections of homogeneous single-turn or multi-turn samples. In your own combined test harness, keep retrieval rows and agent conversation rows linked by case ID while preserving their native shapes. Forcing every signal into one flat row tends to create nullable fields and accidental coercions.

What to inspect when a lane fails

For retrieval failures, start with the reviewed reference and raw retrieved contexts. Identify the missing evidence, then inspect candidate document IDs before filtering, after filtering, and after reranking. Record query text, tenant or user scope, metadata filters, corpus version, and ranks. If the eligible corpus lacks the evidence, route the issue to ingestion or reference review instead of tuning the retriever.

For tool failures, print the actual and expected structured calls in a stable form. Compare call count, name, and arguments. Then inspect the order if the chosen metric cares about order. Finally, verify tool results and durable state. A matched call can still fail at execution, and a successful tool response can still leave the wrong final state after later actions.

This diagnostic script creates a human-readable diff for tool calls. It uses only recorded artifacts, so it remains useful even when the metric provider is unavailable.

import json
from collections import Counter


def key(call: dict) -> str:
    args = json.dumps(call["args"], sort_keys=True, separators=(",", ":"))
    return f"{call['name']}({args})"


expected = [
    {"name": "cancel_subscription", "args": {"subscription_id": "sub_42"}}
]
actual = [
    {"name": "close_account", "args": {"subscription_id": "sub_42"}}
]

expected_counts = Counter(map(key, expected))
actual_counts = Counter(map(key, actual))

missing = list((expected_counts - actual_counts).elements())
unexpected = list((actual_counts - expected_counts).elements())

print(json.dumps({"missing_calls": missing, "unexpected_calls": unexpected}, indent=2))
assert missing == ['cancel_subscription({"subscription_id":"sub_42"})']
assert unexpected == ['close_account({"subscription_id":"sub_42"})']

That diff localizes an exact mismatch. It does not calculate Ragas F1, and it should not claim to. Duplicate calls remain visible because Counter preserves multiplicity. This matters when an agent retries a destructive action and the set of unique names looks correct.

The same missing and unexpected call lines can come from a trace adapter defect rather than agent planning. The runtime may dispatch cancel_subscription correctly, while the adapter drops the call, reads an earlier assistant message, or serializes an application alias as close_account when it constructs the evaluation conversation. Tool evaluation then correctly rejects the structured calls it received. Changing the agent prompt would be a fix to the wrong component.

Preserve a dispatch-boundary record separately from the Ragas message artifact. Compare call count, canonical tool identity, argument names and values, attempt identity, and ordering at both boundaries. If both runtime dispatch and evaluation input contain close_account, planning or tool selection is implicated. If runtime dispatch contains cancel_subscription but evaluation input contains close_account or nothing, the instrumentation adapter is the first divergence. If both contain the expected call and the durable state is wrong, move to tool execution or outcome handling.

Read the diagnostic fields before the lane score. Start with case revision and tool-schema version, then evidence-completeness status, runtime call count, evaluation call count, missing calls, unexpected calls, argument differences, tool results, and final state. A healthy lane has complete evidence and matching structured calls before policy declares it passed. A broken product lane has complete evidence plus the same wrong call at runtime and in the evaluator. A broken adapter lane has unequal boundary records. A misleading perfect score can occur if a fixture accidentally supplies the reference calls as both expected and actual input, so provenance for each side matters as much as the numeric result.

This comparison adds storage and schema maintenance. Dispatch records can contain sensitive arguments, while aggressively redacted arguments may remove the value that caused the failure. Store approved synthetic or tokenized values where possible, restrict raw evidence, and make an explicit unsupported status available when safe comparison is impossible. Do not silently compare only tool names and claim full argument coverage.

Several status patterns are especially useful:

• Retrieval fails, tool calls pass: the agent acted with incomplete evidence, matched a default path, or used knowledge outside the captured contexts.
• Retrieval passes, tool calls fail: evidence was available, but planning, tool selection, argument construction, or policy enforcement failed.
• Both pass, outcome fails: the tool returned an error, state verification is missing, a later step reversed the action, or the expected outcome is wrong.
• Both fail: investigate independently before assigning one root cause. A malformed query may hurt retrieval and change planning, but the artifacts must demonstrate that link.
• Evaluation is incomplete: timeout, authentication, schema, or provider errors belong to the harness. Do not convert them to a quality zero or a pass.

Compare case results, not just suite means. One critical account closure must not disappear inside many read-only weather cases. Label criticality before the run. Segment tool failures by tool risk, workflow, and permission boundary. Segment retrieval by intent, corpus, language, and access scope when those dimensions have enough reviewed cases to support a decision.

Retries need care. Retrying an evaluator call can resolve a transient provider error. Retrying the product agent until it takes the expected action measures best-of-many behavior, not first-attempt reliability. If production retries, model that policy explicitly and retain every attempt. Never report only the successful retry.

Which near-misses can fool the report

One common near-miss is an overfitted reference_tool_calls list. A reviewer captures the first successful trace and treats it as the only valid implementation. A later agent uses an approved equivalent tool, changes the order of independent reads, or omits a redundant lookup. F1 or strict accuracy falls even though the user goal and safety constraints hold.

The evidence is a valid final state plus a policy showing alternative paths are allowed. The fix is to improve the oracle, not waive every tool failure. Use goal evaluation when only the outcome matters, or maintain explicit allowed paths when tool choice carries some constraints. Keep exact references for regulated, destructive, or security-sensitive sequences.

Another near-miss is a perfect call trace with wrong execution. The agent emits transfer_funds with the expected arguments, but the tool returns an authorization error. Tool F1 evaluates the call against the reference, not the transaction result. The outcome lane must fail. Persist tool messages and query the authoritative final state when the test environment permits it.

A third is high entity recall from entity stuffing. The retriever returns a passage containing every name and identifier but the wrong policy relation. Retrieval entity coverage may pass while the source contradicts the reference. Inspect propositions and use an appropriate correctness or faithfulness check. Do not call it an agent planning defect until the evidence supplied to planning is known to be correct.

A fourth is state leakage between cases. Case B passes because case A already created a customer record or cached a document. The trace looks valid in isolation. Replay each case with isolated identifiers and known initial state. Record setup state and cleanup outcome. If isolation changes the result, the earlier score described a contaminated fixture.

A fifth is authorization mismatch. The reference expects a tool or document unavailable to the test identity. Both lanes may fail correctly. Capture permissions and tool availability at run time, then align the case with the intended persona. Granting broader privileges to make the metric green would invalidate the safety test.

These lookalikes all produce red cells, but their owners differ: dataset design, tool execution, source correctness, fixture isolation, or authorization. A lane name alone is not enough. Each failure needs the first divergent artifact.

How to roll the checks into CI

Run the two lanes in separate jobs or clearly separated steps. Independent jobs make ownership and reruns cleaner. A final policy job can require both artifacts and decide whether the release proceeds. It should consume statuses, not average raw scores.

The configuration below shows the wiring only. Keep retrieval floors and tool rules in their respective test modules, with values calibrated from reviewed cases rather than copied from an example.

name: rag-and-agent-evals

on:
  pull_request:
    paths:
      - ".github/workflows/rag-and-agent-evals.yml"
      - "src/retrieval/**"
      - "src/agent/**"
      - "evals/**"

jobs:
  retrieval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: python -m pip install -r evals/requirements.lock
      - run: mkdir -p artifacts
      - run: python -m pytest evals/retrieval -q --junitxml=artifacts/retrieval.xml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: retrieval-results
          path: artifacts/retrieval.xml

  agent-tools:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: python -m pip install -r evals/requirements.lock
      - run: mkdir -p artifacts
      - run: python -m pytest evals/agent_tools -q --junitxml=artifacts/agent-tools.xml
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: agent-tool-results
          path: artifacts/agent-tools.xml

Add blocking in stages. First validate schemas and collect artifacts without gating. Next block missing or malformed evidence, since no quality conclusion is possible. Then block named critical tool cases with reviewed exact references. Finally add retrieval slice policies after reviewers understand evaluator variance and corpus drift.

For an established combined suite, land the independent result schema and readers before splitting execution jobs. Existing dashboards may assume every case has one status and one rerun button. During migration, emit the old presentation from the two underlying lane statuses, but keep raw lane artifacts authoritative and mark any incomplete lane. Then separate job ownership, add per-lane canaries, and remove the compatibility view only after release tooling consumes both statuses. Otherwise the first failure is often a reporting regression that hides one lane rather than a quality regression.

The split is working when each case produces retrieval, tool, and outcome evidence from the same revision; known controls fail only their intended lane; incomplete evidence never becomes a quality value; and the release policy names the exact blocking case and owner. Track artifact completeness and case population alongside pass rates. A higher tool score after half the traces stopped joining is not progress.

Ownership should follow the first divergent artifact. Retrieval engineering owns query construction through the final context bundle. The agent team owns planning and runtime tool dispatch. Each tool service owns execution semantics and durable state. The evaluation team owns adapters, metric configuration, and result classification. Dataset reviewers own references, permitted alternatives, and criticality. A handoff should contain the case and revision, user scope, corpus and tool-schema versions, runtime and evaluator boundary records, expected calls or allowed paths, tool results, final state, lane statuses, and the first difference. This prevents a retrieval ticket from arriving with only an agent score, or a tool-service ticket from arriving without evidence that the call actually ran.

Version references, corpus snapshots, tool schemas, evaluator models, prompts where configurable, and dependency locks. When upgrading Ragas or an evaluator model, run old and new configurations over the same frozen cases. Review changed dispositions before moving the baseline. Pinning supports comparison, but it also retains old behavior, so schedule upgrades rather than avoiding them.

The split has costs. Two lanes create more fixtures, artifacts, and ownership rules. LLM-based retrieval evaluation adds latency and provider cost. Exact tool references require maintenance when APIs evolve. Outcome checks may need isolated test accounts and cleanup. The benefit is not a magical number. It is the ability to send a failure to the engineer who can fix it with evidence already attached.

Do not run the full expensive suite on every text-only change if risk does not justify it. Maintain a small critical pull-request set and a broader scheduled suite. Tag cases by affected retriever, tool, corpus, and workflow so targeted runs remain defensible. Sampling saves cost but reduces coverage, so keep a periodic full run to catch incorrect tags and cross-component effects.

When these gates do not fit the workflow

Skip exact tool-call gating when the agent is intentionally free to choose among many safe paths and the user outcome is the contract. Prefer an outcome check plus safety invariants. Skip RAG metrics when the workflow did not retrieve external context. Do not invent empty contexts merely to keep the dashboard schema uniform.

Avoid ToolCallF1 when order is the risk. A payment workflow that must authorize before capture needs an order-sensitive contract. Conversely, do not use strict order for independent read-only lookups merely because the first recorded trace happened to choose one order. The metric should encode a requirement, not preserve an accident.

Do not use entity coverage as a proxy for retrieval quality when document identity is the real requirement. If a compliance answer must cite policy revision P-104, test that stable document ID directly. Names found in an older revision do not satisfy the requirement even if entity recall is high.

Outcome-only evaluation is also insufficient for destructive or regulated flows. A test account may reach the expected state after an unauthorized intermediate action, a duplicate charge followed by a refund, or a forbidden data lookup. Preserve tool and state-transition checks when the path carries risk.

Even three passing lanes do not prove that retrieved evidence caused the agent's decision. The retriever can return the right policy and the agent can choose the right tool from a memorized default while ignoring that policy. This technique does not catch that causal disconnect. Add a paired counterfactual case when evidence use matters: keep the user goal and tool availability fixed, change one decision-bearing fact in the approved context, and require the action or refusal to change accordingly. The pair costs roughly twice the executions and needs careful review so only the intended fact differs, but it detects a class of accidental passes that independent lane scores cannot.

Finally, keep incomplete runs out of quality comparisons. Missing traces, provider timeouts, and unavailable test tools say nothing about retrieval or planning quality. The release policy may still block because required evidence is absent, but the report must call it an evaluation failure and name the rerun owner.