Evaluate Multi-Turn Agent Goals with Ragas

Ragas multi turn agent goal accuracy evaluates whether an agent actually achieved the user's conversational objective, with or without a reference outcome. Capture the complete message and tool trace, verify durable side effects first, score through the current collections API, calibrate against humans, and gate CI by critical case failures plus an explicit evaluator budget.

Goal achievement is not the same as a convincing final sentence. An assistant may say a reservation succeeded after the provider rejected it, or use an unexpected path and still produce the correct durable outcome. The Ragas evaluation guide helps position this binary outcome metric beside component and business checks; this article focuses on multi-turn agent evidence.

Define the User Goal and Observable End State

Write the goal as a state transition, not a topic. "Discuss restaurant options" describes conversation content. "Reserve an approved restaurant for the user's chosen time and party size, then return its confirmation" describes an outcome. The latter can be checked against transcript evidence and a durable booking record.

The current Ragas agentic metrics documentation describes agent goal accuracy as binary, with one for achieved and zero for not achieved. AgentGoalAccuracyWithReference compares the achieved workflow end state with an expected ideal outcome. AgentGoalAccuracyWithoutReference infers both the user's intended goal and achieved outcome from the conversation before comparing them.

For every case, define:

• The initial user intent and any constraints already known.
• How later user turns refine, replace, or cancel that intent.
• The durable success state and authoritative system that owns it.
• Partial completion states that still count as failure under the release contract.
• Prohibited side effects, such as booking twice or acting without confirmation.
• Whether an explicit reference outcome is available and approved.

The first observable assertion should be deterministic. Confirm that the trace belongs to the expected session, messages are complete and ordered, tool results correlate to calls, and the final domain receipt exists or is absent as expected. Agent goal accuracy then supplies a model-based interpretation of the workflow, not primary authority over money, permissions, or external state.

The broader LLM judge evaluation guide explains evaluator bias and calibration. Apply it here because the metric must infer goals and outcomes from natural language, especially without a reference.

Build Multi-Turn Cases with Transcript and Receipt Evidence

Represent the exact interaction the agent saw and produced. Preserve human messages, AI messages, tool calls, tool results, ordering, and the final assistant response. Do not reconstruct the transcript from summaries after the test; a missing failure tool message can make a claimed success look valid.

Add a case ID, session ID, scenario slice, application version, system prompt hash, agent model, tool registry version, tool-call IDs where the application has them, external operation IDs, reference version, human outcome label, and evaluator metadata. Store secrets and personal data outside the fixture. Use synthetic identities while retaining the fields that affect behavior.

{
  "case_id": "support-ticket-failed-create-021",
  "slice": "ticket-creation-critical",
  "reference_version": "ticket-goal-reviewed",
  "reference": "One support ticket is created for the reported billing issue and its confirmed ticket ID is returned.",
  "transcript": [
    {"role": "human", "content": "Create a support ticket for the duplicate charge."},
    {
      "role": "ai",
      "content": "I will create the ticket.",
      "tool_calls": [{"name": "create_ticket", "args": {"category": "billing"}, "call_id": "call-1"}]
    },
    {"role": "tool", "call_id": "call-1", "content": "Creation failed: service unavailable."},
    {"role": "ai", "content": "Your ticket was created successfully."}
  ],
  "domain_evidence": {"ticket_id": null, "operation_id": "operation-reviewed"},
  "human_adjudication": {
    "goal_achieved": false,
    "reason": "No ticket exists and the final response contradicts the tool result."
  },
  "control_role": "negative_false_claim"
}

The negative case matters because the surface conversation ends with success language. The authoritative tool and domain evidence show failure. Keep that evidence available to deterministic assertions even if the Ragas metric receives only its documented message list and optional reference.

Build cases from normal journeys, production defects, critical side effects, user corrections, abandoned goals, partial success, tool retries, duplicate prevention, ambiguous requests, and safe refusals. Ragas testset generation from real documents can suggest scenarios from source material, but multi-turn trajectories and final outcomes still need human review.

Partition by goal family and workflow template so near-identical traces do not leak across development and held-out sets. Publish a dataset card with scope, traffic window, reference-writing policy, tool and domain evidence requirements, privacy handling, known gaps, and human adjudication process.

Use the Current Collections APIs in Python

The modern classes live in ragas.metrics.collections. This is a critical version boundary. Do not copy the older from ragas.metrics import AgentGoalAccuracyWithReference pattern into new code. Construct the selected metric with an evaluator LLM and call ascore directly with user_input; add reference only for the with-reference class.

Ragas messages model the conversation using HumanMessage, AIMessage, ToolCall, and ToolMessage. The following example runs both modes on the same controlled transcript so their difference can be reviewed.

import asyncio
import os

from openai import AsyncOpenAI
from ragas.llms.base import llm_factory
from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage
from ragas.metrics.collections import (
    AgentGoalAccuracyWithReference,
    AgentGoalAccuracyWithoutReference,
)


def controlled_transcript() -> list:
    return [
        HumanMessage(content="Create a billing support ticket for the duplicate charge."),
        AIMessage(
            content="I will create the ticket.",
            tool_calls=[ToolCall(name="create_ticket", args={"category": "billing"})],
        ),
        ToolMessage(content="Ticket TICKET-TEST was created for the billing issue."),
        AIMessage(content="Ticket TICKET-TEST was created for your duplicate charge."),
        HumanMessage(content="Thanks, that solves it."),
    ]


async def score_goal() -> dict:
    client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
    llm = llm_factory(os.environ["RAGAS_EVALUATOR_MODEL"], client=client)
    messages = controlled_transcript()

    with_reference = AgentGoalAccuracyWithReference(llm=llm)
    referenced = await with_reference.ascore(
        user_input=messages,
        reference="One billing support ticket is created and its confirmed ID is returned.",
    )

    without_reference = AgentGoalAccuracyWithoutReference(llm=llm)
    inferred = await without_reference.ascore(user_input=messages)

    return {
        "with_reference": referenced.value,
        "without_reference": inferred.value,
        "evaluator_model": os.environ["RAGAS_EVALUATOR_MODEL"],
    }


if __name__ == "__main__":
    print(asyncio.run(score_goal()))

Both ascore calls are model-based. Running both is not a free consistency check; account for each evaluator call and preserve each result separately. In Ragas 0.4.3, these built-in goal metrics return the binary MetricResult.value and do not provide a rationale field. Preserve human rationale and decisive domain evidence separately; use a reviewed custom metric if a model-generated explanation is itself a required output. The with-reference result should usually drive a release check when an authoritative outcome can be defined. The without-reference result is valuable for discovering whether the evaluator inferred the same goal a human saw.

Pin the Ragas package, native provider client, evaluator model, metric class, prompt configuration if customized, message renderer, and result schema. The agentic docs use ragas.llms.base.llm_factory; the public ragas.llms re-export appears in other current guides. Choose the import supported by the pinned version and test it in CI rather than mixing examples.

Put the application-to-Ragas message adapter under direct tests. Feed it one example of every supported application message shape and assert the resulting Ragas class, content, tool name, arguments, and order. Add negative fixtures for an unknown role, missing tool result, duplicate call identifier, non-serializable argument, and redacted content that removes the decisive outcome. The adapter should reject an unsupported event rather than dropping it. Silent omission is especially dangerous because the metric can still return a plausible binary judgment over an incomplete conversation.

Keep transcript normalization conservative. Normalize transport-only fields through an explicit versioned rule, but do not merge adjacent human turns, rewrite tool error text, infer a missing result, or replace a provider receipt with the assistant's summary. Hash both the source trace and rendered message list. When those hashes change after an adapter migration, require a paired review even if the high-level case ID remains the same.

Create Positive, Negative, and Goal-Shift Controls

Controls should make a change in outcome visible while holding conversational style as constant as practical.

Positive control: The user requests one action, the tool confirms it, durable domain state contains one matching record, and the assistant returns the confirmation. The reference states that same ideal outcome.

Negative failed-tool control: The tool reports failure, no domain receipt exists, and the assistant truthfully says the action was not completed. The goal remains unachieved even though the response is honest.

Negative false-claim control: Use the same failed tool result, but make the assistant claim success. This tests whether the evaluator reads tool evidence instead of accepting the final sentence.

Partial-goal control: The user requests two required changes and only one succeeds. The reference must make all-or-nothing semantics explicit if partial completion is a failure.

Goal-shift control: The user changes the requested time, recipient, quantity, or action after the agent begins. The final reference should reflect the accepted latest intent. Keep the old and new tool calls visible so the evaluator can distinguish obsolete work from final success.

Cancellation control: The user withdraws consent before execution. Achieving the new goal may mean no side effect occurs. A metric that equates action with success can misclassify safe cancellation.

Duplicate-side-effect control: A retry produces two domain records even though the conversation reports one success. Goal accuracy alone may see the requested outcome, so a deterministic uniqueness assertion must block release.

Run this numbered workflow:

1. Freeze goal definition, reference version, transcript schema, domain-receipt rules, evaluator model, Ragas version, and gate policy.
2. Validate message ordering, tool-call/result correlation, session identity, side-effect records, reference presence, and human adjudication.
3. Execute deterministic authorization, idempotency, record-count, and final-state checks before any model metric.
4. Run positive, failed-tool, false-claim, partial-goal, goal-shift, cancellation, and duplicate-side-effect controls.
5. Stop the evaluation if controls fail, required receipts are absent, result values are invalid, or the case manifest is incomplete.
6. Score with reference for cases with approved outcomes; score without reference only for its declared exploratory or calibrated use.
7. Compare per-case results with human labels and inspect disagreements by goal type, trace length, and failure boundary.
8. Apply the CI and cost policy, preserving unrun cases and routing ambiguous interpretation to human review.

Compare With-Reference, Without-Reference, and Tool Metrics

Choose the metric from the release question. Do not stack metrics simply because they are available. The official Ragas page distinguishes tool-call accuracy from agent goal accuracy: one cares about calls and arguments, while the other cares about outcome.

Use a reference whenever the ideal outcome can be stated without overconstraining implementation. It should specify what must be true, not the exact tool path. A reference such as "one approved ticket exists and its ID is returned" permits alternate valid routing while protecting the durable result.

Govern references like test data. Assign a source owner, effective version, review date, and applicability scope. Separate required outcome, forbidden outcome, and acceptable alternatives instead of hiding them in one long paragraph. When product semantics change, issue a new reference version and evaluate baseline and candidate agents against the same current reference. Do not compare a historical score produced under the old success definition with a new run as though only the agent changed.

The Ragas metric overview distinguishes multi-turn evaluation and metric output types. Preserve the raw binary result, transcript hash, reference version, durable receipts, and human case rationale rather than treating a mean as the only artifact. The built-in 0.4.3 goal result does not supply a reason. Report completed, errored, and unrun cases beside any aggregate, then inspect critical failures individually.

Without-reference scoring is useful when goals emerge conversationally or references do not exist yet. Treat disagreements with the referenced variant as calibration evidence. The inferred metric may identify a user goal omitted by the reference, or it may misread politeness and side discussion as a new objective.

Faithfulness and context recall scenarios help separate evidence use from outcome correctness. RAG retrieval testing applies when the agent's goal depends on finding authorized source material. Keep those component signals joined to the same case without allowing them to override a failed durable outcome.

Calibrate Goal Judgments with Human Outcome Labels

Ask independent reviewers to identify the final accepted user goal, whether it was achieved, the decisive transcript and domain evidence, and any prohibited side effect. Hide metric results and model identity. Adjudicate disagreements, preserving preliminary labels and rationale.

Build calibration slices for short and long conversations, multiple tool calls, user corrections, goal abandonment, partial success, implicit confirmation, failed tools, deceptive final claims, alternate valid paths, and critical side effects. A binary aggregate can conceal systematic confusion around one of these patterns.

Compare with-reference and without-reference verdicts separately to the human outcome. Report false achievements, false failures, evaluator errors, and human disagreement by slice. Do not treat agreement between the two model-based metrics as human validation; both can share an evaluator bias.

The human-adjudicated calibration guide describes independent review and held-out design. Follow it here with goal-specific evidence. Use development cases to improve references or evaluator configuration, calibration cases to choose policy, and untouched cases to certify the final setup.

Run migration calibration whenever the Ragas version, evaluator model, message renderer, agent trace schema, reference style, language mix, or product goal changes. Retain stable sentinels for drift while adding newly adjudicated production failures to a future dataset release.

For conversational coverage beyond a single metric, how to test a RAG chatbot adds memory, citation, refusal, recovery, and user experience. Goal accuracy should answer its narrow question rather than absorb all of those criteria.

Test the Legacy-to-Collections Migration

Current documentation marks AgentGoalAccuracyWithReference and AgentGoalAccuracyWithoutReference imports from ragas.metrics as legacy and deprecated for removal in v1.0. Those examples build a MultiTurnSample and call multi_turn_ascore. New code should import both classes from ragas.metrics.collections and call modern ascore(...) with the message list directly.

Migration tests should cover import path, constructor, message object types, tool-call arguments, ascore keyword names, reference presence, binary result value, serialized run record, and async timeout behavior. Add a contract test that rejects code expecting a built-in reason field, plus a source scan so a copied legacy path fails review before runtime.

Run old and new harnesses on the same frozen controls during migration, but do not assume their result prompts or semantics are byte-identical. Review per-case transitions and record both configurations. The Ragas v0.3 to v0.4 migration guide also recommends native provider clients through llm_factory instead of deprecated LangChain or LlamaIndex wrappers.

Use a shadow migration before replacing release evidence. The existing certified job and modern collections job should consume one immutable case manifest independently, then publish joined records by case ID. Classify missing rows, import failures, rendering differences, result-shape changes, and verdict transitions separately. A transition is not automatically a regression because the metric implementation may have changed, but every critical transition needs human outcome evidence before the new job receives authority.

Do not keep both APIs active indefinitely. Set an owner and drain date for legacy records, update dashboards to recognize the modern result schema, and block newly added legacy imports. Retain old artifacts for audit, but make the current import path visible in run metadata so later analysis cannot combine unlike measurements.

Test transcript conversion independently. Ensure each application role maps to the correct Ragas message type, tool calls remain attached to the correct AI message, tool results stay ordered, and text is not truncated or redacted into a different meaning. A correct metric cannot repair a malformed conversation.

Preserve rollback inputs: lockfile reference, metric import mode, message renderer version, evaluator configuration, and result schema. Rollback means restoring a certified matching configuration, not comparing new-format results with an old baseline that used different inputs.

Debug Goal Failures from Durable State Backward

Start with authoritative final state. Did the ticket, reservation, payment, or update exist with the right owner and parameters? Was it created once? Did authorization and approval hold? Then correlate the domain operation to tool result, tool call, AI message, session, and user goal.

Apply deterministic blockers before interpreting the metric:

• Missing, duplicated, or out-of-order message records.
• Tool result without a matching call, or call without a required result.
• Session, tenant, user, or operation ID mismatch.
• Claimed success without the authoritative domain receipt.
• Duplicate or forbidden side effect despite a successful outcome.
• Missing, stale, or unapproved reference for a with-reference case.
• Undeclared evaluator model, package, message renderer, or metric class.
• Result outside the documented binary values, absent result, or swallowed provider error.
• Failed control ordering or missing case from the run manifest.
• Cost boundary reached before required critical cases completed.

If durable state is correct but the metric says the goal failed, inspect the reference and transcript rendering. The reference may overconstrain a valid alternate path, omit a user correction, or require wording rather than outcome. If state is wrong but the metric passes, check whether the final assistant claim outweighed failed tool evidence in the evaluator's interpretation.

Keep retries visible. A later successful retry may achieve the goal, but duplicate attempts can still violate idempotency or cost policy. Record each operation and final state. The binary goal score belongs beside attempt count and prohibited-side-effect checks, not in place of them.

Do not retry the evaluator until it returns one. Preserve repeated judgments only during a declared stability study. If the same frozen case crosses the release boundary, narrow the decision, improve evidence, change evaluator configuration through calibration, or require human review.

Measuring RAG accuracy provides a wider failure-attribution model when retrieval and answer evidence sit inside the agent workflow. Use the earliest changed artifact to assign ownership instead of blaming the final metric.

Add a CI Gate and Evaluator Cost Boundary

Use a layered pipeline. Run transcript schema, receipt, authorization, idempotency, and reference checks on every affected change. Run a compact frozen goal-metric sentinel for evaluator-facing changes. Run representative and expensive multi-turn cases before release or on a controlled schedule.

Policy should specify required cases, critical slices, maximum failed goals by slice, allowed evaluator errors, metric mode, and spend. Values come from human-adjudicated baselines and business risk, not generic examples.

from collections import Counter
from dataclasses import dataclass
from decimal import Decimal


@dataclass(frozen=True)
class GoalPolicy:
    max_failures_by_slice: dict[str, int]
    max_evaluator_errors: int
    max_total_cost: Decimal


def goal_gate(rows: list[dict], policy: GoalPolicy) -> list[str]:
    blockers: list[str] = []
    failures: Counter[str] = Counter()
    errors = 0
    cost = Decimal("0")
    allowed_slices = set(policy.max_failures_by_slice)

    for row in rows:
        required = {"case_id", "slice", "score", "cost", "receipt_verified"}
        if not required.issubset(row):
            blockers.append("goal result is missing required evidence")
            continue
        if row["slice"] not in allowed_slices:
            blockers.append(f'{row["slice"]}: no reviewed goal policy')
        if not row["receipt_verified"]:
            blockers.append(f'{row["case_id"]}: durable receipt not verified')
        if row.get("evaluator_error"):
            errors += 1
        elif row["score"] not in {0, 1}:
            blockers.append(f'{row["case_id"]}: invalid binary goal result')
        elif row["score"] == 0:
            failures[row["slice"]] += 1
        cost += Decimal(str(row["cost"]))

    for slice_name, count in failures.items():
        limit = policy.max_failures_by_slice.get(slice_name)
        if limit is None:
            blockers.append(f"{slice_name}: no reviewed goal policy")
        elif count > limit:
            blockers.append(f"{slice_name}: goal-failure limit exceeded")
    if errors > policy.max_evaluator_errors:
        blockers.append("evaluator error limit exceeded")
    if cost > policy.max_total_cost:
        blockers.append("evaluator cost boundary exceeded")
    return sorted(set(blockers))

Account separately for with-reference and without-reference evaluator calls. Declare concurrency, retries, timeout, token or usage source, cache key, maximum calls, and monetary allowance. Stop new work at the boundary, list unrun cases, and mark the evaluation incomplete. Never drop long critical traces silently to make the job cheaper.

Cache only when the full semantic input matches: metric class, Ragas version, evaluator model and settings, message-renderer version, transcript hash, reference hash for referenced runs, and any customized prompt hash. A without-reference result cannot satisfy a with-reference request, even for the same transcript. Record cache provenance and validate that cached values still use the documented binary domain before they enter the gate.

Estimate capacity from the declared manifest before starting a release run and reserve room for critical long traces. If both metric modes are scheduled, count them as separate evaluations. Provider retries consume the same boundary unless policy says otherwise. A cheaper partial run that excludes complex goal shifts changes the evidence scope and must not inherit the prior certification.

Publish passed, quality failed, incomplete, and review-required states. A provider outage is not a product goal failure, and unresolved human ambiguity is not a pass. Overrides require named approver, affected cases, domain evidence, expiry, and follow-up.

Sample production goal disagreements through privacy review and human adjudication before adding them to future test releases. Keep the certified sentinel set stable enough to detect evaluator drift. Revisit the Ragas evaluation architecture when combining goal accuracy with retrieval, grounding, tool, latency, and business metrics.

Use the wider RAG chatbot testing workflow to source recovery, clarification, and cancellation journeys that a happy-path agent dataset tends to miss. Convert each selected journey into an explicit final-goal contract before adding it to the metric suite.

Frequently Asked Questions

What does Ragas agent goal accuracy measure?

Agent goal accuracy judges whether a multi-turn workflow identified and achieved the user's goal. The current metric is binary: one indicates achieved and zero indicates not achieved. It evaluates the conversation outcome, not whether every intermediate tool call followed a preferred path, so pair it with deterministic side-effect and policy evidence.

When should I use AgentGoalAccuracyWithReference?

Use AgentGoalAccuracyWithReference when an approved ideal outcome can be written for each case. The metric compares the workflow's achieved end state with that reference, making release intent explicit. Version the reference and retain durable receipts because a conversational claim of success is weaker than a booking, payment, ticket, or database record.

When should I use AgentGoalAccuracyWithoutReference?

Use AgentGoalAccuracyWithoutReference for exploratory evaluation or cases where the user's goal must be inferred from the transcript and no reliable ideal outcome exists. It infers both intended goal and achieved outcome, which adds ambiguity. Calibrate it with humans and avoid giving it unattended release authority on critical side effects.

What is the current import path for agent goal accuracy?

Import AgentGoalAccuracyWithReference and AgentGoalAccuracyWithoutReference from ragas.metrics.collections, construct them with the evaluator LLM, and call ascore with the message-list user_input plus reference when required. The ragas.metrics imports and MultiTurnSample multi_turn_ascore examples are legacy, deprecated, and documented for removal in Ragas v1.0.

Can goal accuracy prove that a tool side effect happened?

No. It is an LLM-based judgment over the supplied conversation and optional reference. An assistant can claim success after a failed tool call, or a transcript can omit the authoritative provider state. Assert durable receipts, idempotency keys, authorization, and final domain state deterministically, then use goal accuracy as an outcome-level diagnostic.

How should agent goal accuracy run in CI?

Validate transcript schema, message ordering, references, side-effect receipts, versions, and case manifests first. Run frozen controls and risk-selected cases with a pinned evaluator under call and spend limits. Block on critical goal failures, invalid results, failed controls, missing receipts, or incomplete evidence, while routing ambiguous judge disagreements to human review.

Practice Multi-Turn Goal Triage in QABattle

Take one transcript where the final assistant claims success. Trace the accepted user goal, tool result, durable receipt, and reference before looking at the metric. Decide whether the failure belongs to the agent, trace, reference, evaluator, or gate. Then practice that evidence-first outcome decision in the QABattle battles arena, where confident language cannot create a missing side effect.