How to Evaluate an AI Agent's Tool Use

A support agent can sound perfect while issuing a double refund, emailing the wrong customer, or inventing a tool result that never ran. Evaluating an AI agent's tool use means scoring multi-step trajectories: tool selection across a task, argument quality, sequencing, sandbox side effects, recovery, and cost.

This is the multi-step guide. For single-call schema adherence, wrong-tool selection on one model turn, and argument validation without a full agent loop, use testing function-calling LLMs. Also pair with broader testing AI agents, MCP server testing, and LangChain testing.

What "Tool Use" Means in Agent Evaluation

Tool use covers the full loop:

1. Decide whether a tool is needed.
2. Choose which tool.
3. Produce arguments.
4. Execute (or propose) the call.
5. Observe the result or error.
6. Possibly call more tools.
7. Produce a final user-facing answer or side effect.

Evaluation must cover selection, arguments, execution safety, interpretation, and final consistency.

Write a Tool Contract Before Scoring Anything

For each tool, document:

Field	Example
Name	create_refund
Purpose	Issue refund for an eligible order
Auth / role	Agent role + customer match
Input schema	order_id string, amount number, reason enum
Business rules	amount <= captured total, not already refunded
Side effects	Writes refund record, may email customer
Idempotency	Keyed by order_id + reason code
Failure modes	404 order, 409 already refunded, 503 downstream
Confirmation	Required above $50
Forbidden uses	Never invent order_id; never refund other customers

Without contracts, evals become opinions about "reasonableness."

Evaluation Dimensions for Tool Use

1. Tool selection accuracy

Did the agent pick an appropriate tool for the goal?

Cases:

• Needs tool vs can answer from dialogue only.
• Correct tool among near duplicates (get_order vs search_orders).
• Should refuse when no safe tool exists.

Metrics:

• Selection accuracy on labeled cases.
• False tool rate (called a tool when none needed).
• Missed tool rate (answered without calling a required tool).

2. Argument validity across the trajectory

Even the right tool fails if arguments are wrong. Deep schema-layer validators and field-level argument suites live in testing function-calling LLMs. On agent paths, assert arguments in context of prior steps:

• JSON schema validity on every hop.
• Enum and type correctness.
• Referential integrity (IDs exist in sandbox and match earlier lookups).
• Goal alignment (refund amount matches user request, policy, and prior get_order result).
• No leaked secrets in arguments or tool-to-tool handoffs.

3. Trajectory quality

Sequence matters:

• Lookup before mutate.
• Confirm before irreversible action.
• No infinite retries.
• No contradictory double submits.

Score paths, not only endpoints.

4. Side effect correctness

In a sandbox, assert world state:

• Record created once.
• Email sent once with correct template.
• No delete when user asked for archive.
• Transactions rolled back on failure when required.

5. Result grounding

Final answer must reflect tool output:

• Must not invent fields missing from the tool response.
• Must surface errors honestly.
• Must not claim success if tool failed.

6. Recovery and resilience

When tools fail, does the agent:

• Retry with backoff when appropriate?
• Switch strategy?
• Ask the user?
• Stop safely without loops?

7. Cost and latency

Tools per success, tokens, wall time, dollars. An agent that succeeds after 40 calls may lose to one that succeeds in 3.

How to Evaluate an AI Agent's Tool Use: Practical Process

Step 1: Inventory tools and risk tiers

Tier	Examples	Eval depth
Read-only low risk	weather, public docs	schema + selection
Read PII	get_customer, get_order	ACL + grounding
Write reversible	draft_email, create_ticket	side effect + content
Write irreversible	refund, delete, wire transfer	confirmation + strict sandbox
Code / shell	run_command	isolation + allowlist

Spend most eval energy on high tiers.

Step 2: Build scenario packs

Scenario types:

1. Happy path single tool.
2. Multi-tool dependency chain.
3. Ambiguous user goal (should clarify).
4. Near-miss tool confusion.
5. Invalid user request (should refuse).
6. Tool 4xx/5xx errors.
7. Empty result sets.
8. Partial success (one of two writes fails).
9. Idempotent retry (user double-clicks intent).
10. Injection via tool result text.
11. Permission denied.
12. Stale data / eventual consistency.

Step 3: Define success per scenario

Bad success definition:

Agent responds helpfully.

Good success definition:

Scenario: Refund eligible order under $50
Given order ORD-100 is delivered and eligible
When user says "Refund my headphones order ORD-100, full amount"
Then:
  - get_order called with order_id=ORD-100 before refund
  - create_refund called once with amount=79.00 and valid reason
  - sandbox.refunds for ORD-100 length == 1
  - final message states refund submitted and amount 79.00
  - no other customer tools called

Step 4: Instrument traces

Your harness must capture:

• Full message history.
• Each tool name, args, raw result, latency.
• Policy decisions / confirmations.
• Final answer.
• Sandbox state diff.

Without traces, you are evaluating theater.

Step 5: Score automatically where possible

def score_tool_use(trace, scenario, sandbox_before, sandbox_after):
    scores = {}
    scores["schema_ok"] = all(validate_schema(c) for c in trace.calls)
    scores["selection_ok"] = set(trace.tool_names) == set(scenario.expected_tools) \
        or selection_allows_superset(trace, scenario)
    scores["args_ok"] = args_match(trace, scenario.expected_args)
    scores["side_effects_ok"] = sandbox_after == apply_expected(sandbox_before, scenario)
    scores["grounded"] = final_consistent_with_tools(trace)
    scores["no_extra_high_risk"] = not unexpected_high_risk(trace, scenario)
    scores["pass"] = all(scores[k] for k in scenario.blocker_keys)
    return scores

Use LLM judges only for soft qualities (clarity of user message) after hard blockers pass.

Step 6: Run multi-sample evaluations

For non-determinism:

• n=5 on critical money paths.
• Report success rate, not single green.
• Investigate high variance scenarios (often underspecified prompts or flaky tools).

Step 7: Compare versions fairly

When comparing prompts or models:

• Same scenarios.
• Same tool mocks or sandbox seed.
• Same model pins within a candidate.
• Same temperature policy.
• Same max-step budget.

Trajectory Patterns That Should Fail

Pattern	Why it is bad	Detection
Mutate without read	Wrong entity risk	Order of calls
Repeated identical write	Double charge / spam	Duplicate args window
Ignore tool error	False success	Final claim vs error
Invent tool result	Hallucinated state	Call log empty but claims data
Tool thrash loop	Cost / DoS	call count > budget
Wrong tenant ID	Data breach	ACL asserts
Secrets in args	Leakage	Secret scanner on args

Mocking Strategy

When to mock

• Fast CI selection tests.
• Rare error injection (500, timeout, malformed JSON).
• Deterministic trajectory grading.

When to use real sandboxes

• OAuth and authz behavior.
• Pagination quirks.
• Real schema drift detection.
• Latency budgets under realistic responses.

Hybrid pattern: contract tests with mocks + nightly integration against sandbox.

Worked Example: Calendar Scheduling Agent

Tools:

• list_events(date_range, calendar_id)
• create_event(title, start, end, attendees)
• send_invite(event_id)

Scenario: "Schedule 30 minutes with Alex tomorrow afternoon if free."

Expected trajectory (one acceptable path)

1. Resolve "tomorrow afternoon" into a concrete window using clock fixture.
2. list_events for that window.
3. Choose a free slot.
4. create_event with correct duration and attendee.
5. Optionally send_invite if product requires it.
6. Final message reports time in user timezone.

Fail examples

• Creates event without listing (collision).
• Duration 60 minutes despite request.
• Attendee wrong email hallucinated.
• Claims "invite sent" without send_invite or provider success.
• Loops list_events 20 times.

Scoring table

Check	Weight	Blocker
Free slot respected	high	yes
Duration 30 min	high	yes
Attendee correct	high	yes
<= 4 tool calls	medium	no
Clear confirmation text	medium	no

Evaluating Parallel and Multi-Tool Plans

Some agents call tools in parallel. Evaluate:

• Independence: parallel only when no data dependency.
• Merge correctness: final answer reconciles both results.
• Partial failure: one parallel call fails, agent handles it.

Label scenarios as serial_required vs parallel_ok so scorers do not unfairly demand a single order when multiple orders are valid. Prefer constraint-based trajectory scoring over one gold path when multiple paths succeed.

Constraint examples:

• get_order before create_refund.
• At most one create_refund.
• No delete_* tools.
• Must call search if user did not provide ID.

Permission and Confirmation Evaluation

High-risk tools need product gates beyond model judgment.

Tests:

1. Low-privilege user cannot cause privileged tool calls (backend enforced).
2. Agent proposes refund; without UI confirmation, no side effect.
3. Injection text tries to skip confirmation and fails.
4. Agent cannot self-approve by calling a fake confirm tool if that tool is not authorized.

Evaluate control plane and model together.

Metrics Dashboard for Tool Use

Track over time:

• Task success rate by scenario pack.
• Mean tools per successful task.
• Argument invalid rate.
• Recovery success after injected errors.
• Hallucinated success rate (claimed success without successful tool).
• Cost per successful task.
• Human escalation rate.

Cost per success is more honest than cost per run when many runs fail early.

Common Mistakes When Evaluating Tool Use

Mistake 1: Scoring only the final sentence

You will ship agents that sound right while doing wrong things.

Mistake 2: One golden trajectory only

Brittle scoring fails valid alternate plans. Use constraints.

Mistake 3: No sandbox state asserts

Logs can be incomplete. World state is ground truth for writes.

Mistake 4: Trusting the model summary of tool output

Always parse raw tool results in the harness.

Mistake 5: Ignoring idempotency

Retries and double submits are normal. Evaluate them.

Mistake 6: Unlimited step budgets in tests

Production will have limits. Enforce max steps in eval too.

Mistake 7: Never testing tool result injection

Hostile or weird tool payloads can re-enter the prompt as instructions.

Mistake 8: Averaging away critical fails

A 95% success rate with 5% wrong refunds is not "pretty good." Slice by severity.

Mistake 9: Evaluating tools without schemas frozen

If schemas drift silently, argument scores become noise. Version tool contracts.

Mistake 10: No multi-sample runs

Declaring pass from one lucky temperature sample is not evaluation.

Checklist: Tool-Use Eval Readiness

• Every tool has a written contract and risk tier.
• Scenarios cover happy, refuse, error, multi-tool, injection.
• Trace capture includes args and raw results.
• Schema validation is automated.
• Sandbox state diffs for write tools.
• Constraint-based trajectory scoring (not only one path).
• Confirmation and ACL tests for high-risk tools.
• Multi-sample success rates on critical packs.
• Cost and step budgets measured.
• Incident trajectories converted into regression cases.
• Release blockers defined for money, PII, and delete classes.

How This Fits a Full Agent Test Strategy

Tool evaluation is one pillar next to:

• Planning quality.
• Memory correctness.
• RAG grounding.
• User experience copy.
• Online monitoring.

See testing AI agents for the full stack. Use QABattle or sign up to practice structured scenario thinking under constraints, then encode the same precision into tool scenarios.

Schema Evolution and Tool Versioning

Tools change. Evaluation must version with them.

Practices:

• Store JSON schemas next to the agent release.
• When a required field is added, add cases that fail closed if the agent omits it.
• When a tool is deprecated, add cases that fail if the agent still calls it.
• Keep a compatibility window in staging where old and new schemas are tested during migration.

Example regression:

Tool create_refund v2 requires reason_code enum.
Case: user asks for refund without stating reason.
Expected: agent asks clarifying question OR supplies allowed default per policy.
Fail: agent calls create_refund without reason_code or with free-text in enum field.

Evaluating Read Tools vs Write Tools Differently

Read tools

Focus on:

• Selection when data is required.
• Correct filters (date ranges, user scope).
• Grounding of the final answer in results.
• Empty result honesty.

Write tools

Focus on:

• Authorization and confirmation.
• Idempotency.
• Exactly-once side effects under retry.
• Compensating actions if partial failure is possible.
• Audit log completeness.

A suite that treats get_order and create_refund with the same depth under-tests money movement.

Human Handoff as a Successful Tool Outcome

Sometimes the best tool use is stopping and escalating.

Scenarios:

• Fraud suspicion language from user.
• Policy exception not representable in tools.
• Repeated tool 503s.
• User explicitly asks for a human.

Expected:

• No unsafe write attempts to "force" completion.
• Handoff tool or ticket created with accurate summary.
• Final message sets correct expectations.

Score handoff quality: summary includes order id, user intent, and what already failed. This is part of how to evaluate an AI agent's tool use in real support products.

Sample Rubric Card for a Single Scenario

Scenario ID: TOOL-REFUND-009
Blockers (pass/fail):
  [ ] Lookup before refund
  [ ] Refund amount matches eligible total
  [ ] Single create_refund call
  [ ] Sandbox shows one refund record
  [ ] Final text does not claim email if email tool not called
  [ ] No cross-customer IDs

Efficiency (graded):
  Tools used: _ / budget 4
  Clarifying questions: appropriate / excessive / missing

UX (graded 1-3):
  Clarity of confirmation message: _

Run this card automatically for blockers and optionally with a human or judge for UX grades.

Debugging Playbook When Tool Evals Fail

1. Was the tool schema available to the model in the prompt?
2. Did the user utterance actually specify enough data?
3. Did mocks return realistic error shapes?
4. Is the grader too strict on call order when parallel was valid?
5. Did a prompt change remove few-shot tool examples that the model relied on?
6. Is max-step budget too low for the task difficulty?

Log these answers in the triage ticket. Over time you will see whether failures are model limits, harness bugs, or product policy gaps.

Logging and Privacy in Tool Traces

Traces are gold for evaluation and a risk for privacy. Rules of thumb:

• Redact secrets and raw PII in stored args when possible, while keeping enough structure to grade.
• Retain full raw traces only in secured environments for failed high-severity cases.
• Never paste production customer payloads into public issue trackers.
• Hash or synthesize identifiers in shared benchmark datasets.

Your harness should make the secure path the easy path, or engineers will dump everything into Slack.

Release Decision Example

Imagine two candidates on the same tool-use suite:

Config	Task success	Invalid args	Unauthorized writes	Cost / success
Prompt v7	0.88	0.04	0	$0.07
Prompt v8	0.91	0.06	1 case	$0.08

v8 is "smarter" on success rate but fails a zero-tolerance unauthorized write. You block v8, file a bug, add a regression scenario, and either ship v7 or fix v8. That decision pattern is the point of rigorous tool evaluation: product risk outranks a slightly higher average.

Final Takeaway

Evaluating an AI agent's tool use comes down to contracts, traces, and world state across a multi-step task. Check selection along the path, arguments in context, sequence constraints, side effects, grounding, recovery, and cost. Automate hard asserts first. Use graded judges last. Run critical scenarios more than once. Keep single-call schema and wrong-tool packs in testing function-calling LLMs, and use this guide for the full trajectory. If you only remember one rule: never trust a success message without a matching successful tool trace and sandbox proof.