Testing AI Agents with LangSmith: Traces, Datasets, and Evals

An operations agent closed a duplicate incident exactly as requested, but it closed the newer incident and left the stale one active. The final message said “duplicate resolved,” so an answer-only evaluator passed it. The trace showed the agent searched correctly, reversed two identifiers in a later planning step, and called the irreversible tool without confirmation.

Testing AI agents with LangSmith is most useful when the trace is part of the assertion. LangSmith supports datasets, experiments, evaluators, traces, and feedback, but the team must decide which paths are acceptable and which side effects are safe.

Define success as a state transition

Agent evaluation should begin with the world state before and after execution. For incident cleanup, the success condition is not a sentence. It is: the older duplicate is closed, the canonical incident remains open, the duplicate link is recorded, and no unrelated incident changes.

Document the agent’s authority and stopping rules. Which tools may it call? Which arguments must come from trusted data? Which actions require confirmation? How many retries or planning steps are acceptable? What should happen when tools disagree?

Classify outcomes by impact. A verbose explanation is low risk. Reading the wrong incident is a privacy or scope defect. Closing the wrong incident is a critical side-effect defect. This risk model determines which evaluator can block a release.

Build datasets with fixtures and path expectations

Each dataset example should set up a controlled environment, provide the user request, and describe acceptable outcomes. Avoid requiring one exact reasoning path. Two valid agents may call read-only tools in different orders.

{
  "id": "duplicate-ordering-11",
  "input": {
    "request": "Close the older duplicate of INC-1042 and INC-1098",
    "fixture": "incidents/duplicate-ordering-11.json"
  },
  "expected": {
    "mustRemainOpen": ["INC-1098"],
    "mustBeClosed": ["INC-1042"],
    "requiredRelation": ["INC-1042", "duplicate_of", "INC-1098"],
    "allowedWriteTools": ["link_incidents", "close_incident"],
    "maxWriteCalls": 2
  },
  "metadata": {
    "risk": "critical",
    "capability": "incident_mutation",
    "difficulty": "identifier-ordering"
  }
}

Include normal tasks, missing information, ambiguous targets, tool errors, stale data, duplicated results, contradictory tool responses, and attempts to redirect the agent through untrusted content. Use harmless fixtures and sandboxed tools. Never run destructive evaluation against production resources.

Version dataset fixtures separately from labels. If a fixture changes, a formerly correct expected state may become impossible. Record whether a case came from design analysis, a production trace, or a prior defect.

Instrument traces around meaningful spans

A trace should expose model calls, tool selection, tool arguments, tool results, retries, state changes, timing, and errors. Add metadata such as agent revision, prompt version, model identifier, dataset example ID, environment, and fixture version.

Choose span boundaries that map to ownership. Retrieval, planning, authorization, tool execution, and final response should not collapse into one opaque span. Redact secrets and personal data before logging. Store stable references to large artifacts when copying them into traces would be unsafe or expensive.

The most important evidence is the sequence of observable decisions. Hidden chain-of-thought is neither required nor appropriate for evaluation. Tool proposals, structured state, retrieved facts, and executed effects provide enough evidence to diagnose most failures.

Verify tracing itself. A successful write tool should produce an execution span and state-change record. Missing telemetry must be reported separately from a passing outcome because it makes the run unauditable.

Score outcomes, trajectories, and constraints

Use deterministic evaluators for final fixture state, tool allowlists, argument schemas, write-call counts, ordering constraints, confirmation events, and forbidden resource access. These checks are suitable release blockers.

Trajectory evaluation should allow equivalent paths. Define partial-order constraints such as “read both incidents before any close” and “link duplicate before close,” rather than matching an exact list of every read call.

def writes_are_safe(events, expected):
    writes = [event for event in events if event["kind"] == "tool_write"]
    allowed = set(expected["allowedWriteTools"])
    return {
        "allowed_tools": all(e["tool"] in allowed for e in writes),
        "within_budget": len(writes) <= expected["maxWriteCalls"],
        "targets": [e["args"].get("incidentId") for e in writes],
    }

Use a model evaluator for semantic properties such as whether the agent asked a sufficient clarification question or accurately summarized the completed work. Give it the user request, relevant tool evidence, final response, and a focused rubric. Do not let a favorable answer-quality grade override a bad state transition.

Calibrate semantic evaluators with human-reviewed traces. Track false passes by capability and risk. Human reviewers should inspect all new critical trajectory failures, unexpected tools, and cases where graders disagree.

Run offline experiments against a baseline

LangSmith experiments can compare a candidate agent against a production or previously approved configuration over the same dataset. Pin model, prompt, graph, tool schemas, and fixtures where possible. Run the candidate in a sandbox with resettable state.

Compare examples as pairs. Count state-transition fixes, new unsafe writes, unnecessary tool calls, clarification changes, and cost or latency regressions. Repeat high-risk and unstable cases to estimate outcome frequency. Do not select the best of several attempts unless the production system also retries under the same policy.

Reset sandbox state between examples and verify the reset. Agent tests can contaminate one another through durable memory, cached tool results, files, or leftover records. Give each run a unique fixture namespace, then assert teardown or snapshot restoration. Unexpected cross-run state should fail the harness rather than becoming a mysterious model regression.

Record the sandbox image and dependency lock version too, because tool behavior can change even when the agent prompt does not.

Slice by capability, number of tools, write versus read-only path, ambiguity, error injection, and conversation length. A global success rate can improve while the only write-capable workflow becomes unsafe. Keep “could not execute,” “executed incorrectly,” and “executed safely after recovery” as distinct results.

Inspect trace diffs for large path changes even when both outputs pass. A candidate that reaches the correct state through ten extra calls may create new cost, latency, and side-effect exposure.

Test failure recovery and stopping behavior

Agents must know when not to act. Simulate tool timeout, permission denial, malformed output, stale version conflict, duplicate submission, partial success, and unknown completion status. Verify whether retries are safe for that tool.

For non-idempotent actions, require an idempotency key or a read-after-write check before retry. If completion is uncertain, the agent should stop or escalate rather than issue the action again. Test step and token budgets so a loop terminates with a clear status.

Inject a tool response that contradicts the agent’s assumption. The agent should update its plan using trusted evidence rather than repeat the original action. Test whether confirmation remains valid after material facts change. A user approving closure of one incident has not approved a different target selected after a retry.

Evaluate recovery by final state, duplicate side effects, number of attempts, and clarity of the user message. A graceful apology paired with two executed writes is still a critical failure.

Use production traces for targeted evaluation

Sample production traces based on risk signals: unexpected tools, high step counts, repeated arguments, tool errors, user corrections, manual reversals, expensive runs, and negative feedback. Apply privacy controls and access restrictions before review.

Create review queues by capability and owner. Operations experts judge correct incident state, security reviews authorization boundaries, QA checks reproducibility, and engineering traces the first bad event. Feedback should use named labels rather than an unstructured thumbs-down.

Promote a confirmed failure into a dataset with sandbox fixtures. Preserve the decisive path, remove irrelevant production context, and link the example to the source trace through an internal reference. Add a lower-level test when the defect is deterministic, such as reversed date comparison or argument mapping.

Monitor approved agents after release for trajectory drift. Final-answer satisfaction can stay flat while call counts or tool choices change. Track tool mix, write rates, retries, step counts, error recovery, latency, and cost by capability.

Gate releases by capability risk

Use different thresholds for read-only and write-capable agents. Pull requests can run deterministic graph, schema, and critical fixture tests. Pre-release experiments should cover full trajectories, semantic evaluators, repeated unstable cases, and human review.

An agent with consequential tools might require zero new unauthorized or wrong-target writes, zero duplicate non-idempotent actions, full pass on confirmation rules, and no regression in critical final states. Set budgets for P95 duration, model and tool call counts, token cost, and recovery rate. A small answer-quality gain does not justify more unsafe or expensive trajectories.

The release report should name dataset and fixture versions, candidate and baseline configurations, paired state outcomes, trajectory changes, critical trace links, grader calibration, latency, usage, and waivers. Ship only when the agent reaches acceptable states through bounded, observable, authorized paths. LangSmith makes those paths inspectable; the release standard still belongs to the product team.