When an AI agent gets two different answers from its tools

The checkout agent says an order was refunded, but the payment ledger still shows a settled charge. Both tool calls returned successfully. If the test checks only the agent's polished final answer, it can pass while the workflow acts on whichever result happened to arrive last.

This is the hard part of conflicting agent tool observation testing: an HTTP 200 is not an agreement, and a fluent explanation is not evidence that the disagreement was handled. The useful test target is the decision path between the two raw observations and the next side effect.

Why one tool call can produce two truths

Agents often read the same business fact through different paths. A refund service owns the write. A search index exposes a denormalized copy. A support system caches the order summary. Each path can be healthy while its answer reflects a different point in the update sequence.

That distinction matters because transport success answers only one question: did the call complete according to its protocol? It does not prove freshness, completeness, or authority. A JSON body can be syntactically valid and still contain yesterday's state. A GraphQL-style response can carry useful data and an error in the same envelope. A timeout wrapper can return a cached fallback without making that fallback current.

The first engineering mistake is flattening every result into a short sentence before recording it. Consider these two observations:

• The refund service returned status refunded at source revision 42.
• The order-search tool returned status paid from index revision 39.

If the trace stores only "order status is refunded" followed by "order status is paid," the investigator has lost the fact that the values came from different stores with different revisions. The agent cannot apply a precedence rule reliably, and the QA engineer cannot tell whether the product ignored a rule or lacked the evidence needed to use it.

The second mistake is treating arrival order as truth order. Network scheduling decides which promise settles first. It says nothing about which source owns the field. Even timestamps need care. A server's updatedAt value, a proxy's Date header, and a caller's completedAt time describe different events, and clocks can disagree. A source-managed revision is stronger evidence when the source promises that revisions increase for that record. Without such a contract, keep the conflict open.

The third mistake is comparing whole payloads when only one field controls the action. Two address responses may differ in formatting while referring to the same destination. Conversely, two large payloads may look almost identical while disagreeing on canRefund, currency, or accountId. The comparison needs a named decision field and a normalization rule that is narrow enough to review.

A useful observation record has separate fields for identity, provenance, content, and execution:

• observationId identifies the recorded result.
• toolCallId joins the result to exactly one invocation.
• source names the system that produced the business value.
• arguments preserve the query that was actually sent.
• startedAt and completedAt describe caller-side execution.
• sourceRevision carries a version only when the source supplies one.
• value contains the unmodified response data needed by the oracle.
• errors preserves warnings, partial failures, and fallback markers.

Do not ask the model to reconstruct those fields from its own prose. Capture them in the tool boundary before the result becomes prompt text. The trace can later include a redacted view, but redaction must not erase the fields used by the test.

There are three valid outcomes after comparison. Confirmed means the relevant values agree under the approved normalization rule. Resolved means they disagree, but a documented authority or revision rule selects one and records why. Unresolved means the conflict affects the proposed action and no approved rule selects a winner. Unresolved is not a tool failure. It is a decision state, and the agent needs an explicit path for it.

Build an oracle that keeps the evidence

Start with a small domain function. It should not call a model, a database, or a remote tool. Given frozen observations, it returns a classification that a normal test runner can assert.

The example below handles order status. The policy is deliberately conservative. Agreement is accepted. A higher revision is accepted only when both observations come from the same revision domain. A configured authoritative source can win for one named field. Everything else remains a conflict. Save it as src/resolve-order-status.ts.

export type OrderStatus = "paid" | "refunded" | "chargeback";

export type StatusObservation = {
  observationId: string;
  toolCallId: string;
  source: "refund-service" | "order-ledger" | "search-index";
  revisionDomain: "payments" | "search";
  sourceRevision?: number;
  status: OrderStatus;
  completedAt: string;
  errors: string[];
};

export type StatusResolution =
  | { kind: "confirmed"; status: OrderStatus; evidence: string[] }
  | {
      kind: "resolved";
      status: OrderStatus;
      rule: "higher-source-revision" | "authoritative-source";
      evidence: string[];
    }
  | {
      kind: "conflict";
      reason: "partial-result" | "no-approved-precedence";
      evidence: string[];
    };

export function resolveOrderStatus(
  observations: readonly StatusObservation[],
): StatusResolution {
  if (observations.length < 2) {
    throw new Error("At least two observations are required");
  }

  const evidence = observations.map((item) => item.observationId);

  if (observations.some((item) => item.errors.length > 0)) {
    return { kind: "conflict", reason: "partial-result", evidence };
  }

  const statuses = new Set(observations.map((item) => item.status));
  if (statuses.size === 1) {
    return {
      kind: "confirmed",
      status: observations[0].status,
      evidence,
    };
  }

  const sameRevisionDomain = observations.every(
    (item) => item.revisionDomain === observations[0].revisionDomain,
  );
  const allHaveRevision = observations.every(
    (item) => item.sourceRevision !== undefined,
  );

  if (sameRevisionDomain && allHaveRevision) {
    const ordered = [...observations].sort(
      (left, right) => right.sourceRevision! - left.sourceRevision!,
    );
    if (ordered[0].sourceRevision !== ordered[1].sourceRevision) {
      return {
        kind: "resolved",
        status: ordered[0].status,
        rule: "higher-source-revision",
        evidence,
      };
    }
  }

  const refundOwner = observations.find(
    (item) => item.source === "refund-service",
  );
  if (refundOwner && observations.every((item) => item.status !== "chargeback")) {
    return {
      kind: "resolved",
      status: refundOwner.status,
      rule: "authoritative-source",
      evidence,
    };
  }

  return {
    kind: "conflict",
    reason: "no-approved-precedence",
    evidence,
  };
}

Every rule in that function is a product contract, not a universal truth. The refund service is authoritative only because the hypothetical system owner declared it so for refund state. It does not automatically own chargeback state. A different product may assign authority to the ledger, or require a manual review for every disagreement.

This is why a generic "take the latest" utility is dangerous. It quietly turns deployment topology into business policy. The oracle should make precedence visible in code and in its returned reason.

Now exercise the function with distinct paths. Save the listing as tests/resolve-order-status.spec.ts. Playwright Test supplies the runner and assertions without launching a browser, so these tests do not need a live model.

import { test, expect } from "@playwright/test";
import {
  resolveOrderStatus,
  type StatusObservation,
} from "../src/resolve-order-status";

const base: StatusObservation = {
  observationId: "obs-refund-1",
  toolCallId: "call-refund-1",
  source: "refund-service",
  revisionDomain: "payments",
  sourceRevision: 42,
  status: "refunded",
  completedAt: "2026-08-04T10:00:01.200Z",
  errors: [],
};

test("accepts agreement without using completion order", () => {
  const ledger: StatusObservation = {
    ...base,
    observationId: "obs-ledger-1",
    toolCallId: "call-ledger-1",
    source: "order-ledger",
    sourceRevision: 42,
    completedAt: "2026-08-04T10:00:00.900Z",
  };

  expect(resolveOrderStatus([base, ledger])).toEqual({
    kind: "confirmed",
    status: "refunded",
    evidence: ["obs-refund-1", "obs-ledger-1"],
  });
});

test("uses the source owner instead of a later search response", () => {
  const staleSearch: StatusObservation = {
    ...base,
    observationId: "obs-search-1",
    toolCallId: "call-search-1",
    source: "search-index",
    revisionDomain: "search",
    sourceRevision: 105,
    status: "paid",
    completedAt: "2026-08-04T10:00:02.400Z",
  };

  expect(resolveOrderStatus([base, staleSearch])).toEqual({
    kind: "resolved",
    status: "refunded",
    rule: "authoritative-source",
    evidence: ["obs-refund-1", "obs-search-1"],
  });
});

test("does not hide a partial response behind its data", () => {
  const partialLedger: StatusObservation = {
    ...base,
    observationId: "obs-ledger-2",
    toolCallId: "call-ledger-2",
    source: "order-ledger",
    errors: ["PAYMENT_EVENTS_UNAVAILABLE"],
  };

  expect(resolveOrderStatus([base, partialLedger])).toEqual({
    kind: "conflict",
    reason: "partial-result",
    evidence: ["obs-refund-1", "obs-ledger-2"],
  });
});

test("leaves chargeback authority unresolved", () => {
  const ledgerChargeback: StatusObservation = {
    ...base,
    observationId: "obs-ledger-3",
    toolCallId: "call-ledger-3",
    source: "order-ledger",
    status: "chargeback",
  };

  expect(resolveOrderStatus([base, ledgerChargeback])).toEqual({
    kind: "conflict",
    reason: "no-approved-precedence",
    evidence: ["obs-refund-1", "obs-ledger-3"],
  });
});

These cases assert the resolution and the evidence identifiers. That second check is important. A resolver that reaches the right status after silently dropping one observation is still defective because the audit trail cannot explain the decision.

Keep an end-to-end layer, but give it a different job. It should prove that production wiring preserves call IDs, raw envelopes, and the resolution reason. Do not make every pull request depend on a live model producing a particular phrase. The deterministic function owns the semantic contract; the end-to-end run checks transport and integration.

Work through failures that look the same at first

The final transcript "tool A says refunded, tool B says paid" can come from several mechanisms. One recovery does not fit all of them.

A replica is behind the source of truth

Suppose an agent calls the refund service immediately after submitting a refund, then calls an order search endpoint. The first returns refunded at payment revision 42. The search document still contains paid at search revision 105.

Revision 105 is numerically larger, but it belongs to another revision domain. Comparing 105 with 42 would be meaningless. Search revisions count index updates; payment revisions count payment events. The evidence that identifies this case is the combination of different source names, different revision domains, and a known ownership rule for refund status.

The correct fix is usually not another model instruction. Preserve the domains, then either trust the documented owner or wait for an explicit convergence condition before a read that depends on the index. Waiting costs latency. Trusting the owner reduces cross-check coverage. Choose based on the next action.

For a customer-facing status message, showing the refund service's accepted state may be appropriate. For an operation that ships goods, the workflow may need a settled business event rather than either read. The same pair of observations can therefore resolve differently for two actions. Put the action name in the policy input instead of pretending that one source is authoritative for everything.

A partial result was flattened into a success

Some APIs return a useful data object alongside warnings or field-level errors. The transport completes, the JSON parses, and the agent sees a plausible status. A wrapper that extracts only data can erase the warning that makes the status unsafe.

The signature here is not an older revision. It is an error or completeness marker in the same raw envelope as the value. Retrying may help if the missing dependency recovers, but a retry must be recorded as a new attempt. It must not rewrite the first observation in place.

The test should inject data plus an error and assert three things: the wrapper retains both, the resolver classifies the result as partial, and no protected side effect occurs. A final-answer assertion catches only the third condition, and only if the answer reliably reveals the action. Directly inspect the action log or fake side-effect adapter.

The trade-off is stricter availability. If any warning blocks every read, harmless optional-field errors can stop the agent. Classify completeness by the fields needed for the proposed action. Missing avatar data should not block a refund decision. Missing currency or payment status should.

Two deliveries describe one event differently

Webhook replay creates another similar trace. A tool may read an event store and see the original delivery, while another reads a projection that already processed a corrected delivery. If the harness compares only status strings, it reports a source conflict. The deeper problem may be duplicate or out-of-order event handling.

Look for event IDs, delivery IDs, causation IDs, and the projection checkpoint. When two records share an event ID but carry different bodies, that is an integrity problem, not routine eventual consistency. When delivery IDs differ and a later event explicitly supersedes the earlier one, the projection may be correct.

The fix belongs in event processing or idempotency handling. Teaching the agent to pick one payload merely hides corruption. This is a good example of when the agent test should fail and hand the defect to another component owner.

A normalization bug manufactures a disagreement

Not every difference is substantive. "PAID", "paid", and a localized display label may map to one canonical state. Address lines can differ in whitespace. Monetary values can use major or minor units.

Normalization must be field-specific and lossless enough for the decision. Lowercasing an enum is reasonable when the contract says values are case-insensitive. Removing punctuation from an account number is not automatically safe. Converting money requires the currency and unit, not a guess based on the number's size.

Evidence for this near-miss is stable provenance plus values that become equal under an approved canonicalizer. Add a separate canonicalizer test. Do not change raw observations before storage, or later investigators will be unable to see what arrived.

A response was joined to the wrong invocation

Concurrent lookups create a failure that resembles replica lag almost perfectly. The transcript shows one tool saying refunded and another saying paid, both calls are successful, and both observations carry plausible revisions. The difference is that one response belongs to another order. Waiting for convergence or preferring the refund service will not repair that defect. It can make the agent act confidently on a stranger's record.

Read identity before freshness. For each observation, compare the entity named in the captured arguments with the source-owned entity identity in the raw response. If the source does not echo an identity, the adapter must retain the immutable request context used to join the response. A mutable loop variable or a position in a completed-promise array is not durable evidence of that join.

Illustrative diagnostic output for a healthy observation would describe observation obs-31, tool call call-31, an argument for order fixture-204, a response for that same order, and source revision 44. A broken observation would show call-32 requested fixture-205 while its attached response identifies fixture-204. A misleading observation can still show an empty error list, the newest completion time, and revision 45. Those values prove successful transport and a fresh source record. They do not prove that the record answers this invocation.

Add this check to an existing suite in a deliberate order. Land capture of request identity and response identity in every relevant adapter first. Update frozen fixtures next, because older fixtures will otherwise fail for missing evidence rather than a real mismatch. Run the entity validator in report-only mode while concurrent cases and retries exercise it. After every adapter produces stable identities, make mismatched entities and responses with no provable request identity blocking failures. Only then enable the assertion in broad agent scenarios. The first breakage is usually in shared test doubles that return a valid payload without preserving which request produced it.

This validation adds maintenance and privacy cost. Resource identifiers can be sensitive, while hashing them makes cross-system comparison harder when canonical forms differ. Keeping a redacted stable token beside the protected raw value requires an agreed canonicalization rule and retention policy. Serializing all calls would reduce correlation races, but it also gives up the latency benefit of parallel tool use, so it is a poor substitute for correct joins.

The agent-platform owner is responsible for call correlation. Each tool or source owner is responsible for stating which response identity is authoritative and how aliases map to it. A handoff from QA needs the run ID, both call IDs, redacted requested identities, returned identities, raw envelope locations, revisions, completion order, and the action that followed. That package lets the adapter team reproduce the join without receiving unrestricted customer data.

Identity matching does not catch a source that returns the wrong business status under the correct entity identity and revision. That is a source-integrity or domain-rule defect. The observation contract can preserve it and help contain the decision, but a source-level test must prove the status itself.

Read the evidence before blaming the model

A good failure record lets you answer four questions without rerunning the workflow:

1. Which invocation produced each observation?
2. What did the tool boundary receive before summarization?
3. Which exact field disagreed under which normalization rule?
4. What action did the agent propose or execute after classification?

Joinability is the first check. Every result needs a toolCallId that matches one invocation. If two results share a call ID, the instrumentation is corrupt. If a result has no matching call, the trace is incomplete. Either defect invalidates conclusions about agent reasoning.

Next compare raw envelopes, not screenshots of chat text. Confirm that errors, fallback flags, revisions, and source names survived serialization. If the wrapper has already thrown away an error array, a model trace cannot recover it.

Then inspect timing without overreading it. startedAt and completedAt can prove call ordering from the recorder's point of view. They cannot prove business freshness. A later completion carrying an older source revision is expected under caching. A server timestamp in the future may indicate clock skew rather than prophetic data.

Finally inspect the resolution event and the side-effect event. There should be a direct link from the chosen evidence IDs to the decision. "Agent considered both tools" is too vague. A machine-readable event such as resolutionKind, ruleId, disputedFields, and evidenceIds is reviewable.

The following diagnostic reads a JSON Lines trace. It prints only records for one run and exposes a conflict whose final action was not blocked. Save it as inspect-conflict.sh. The sample identifiers and values are synthetic fixtures, not production measurements.

#!/usr/bin/env bash
set -euo pipefail

trace_file="$1"
run_id="$2"

if [[ -z "$trace_file" || -z "$run_id" ]]; then
  echo "usage: inspect-conflict.sh TRACE.jsonl RUN_ID" >&2
  exit 2
fi

jq -s --arg run "$run_id" '
  map(select(.runId == $run))
  | {
      observations: [
          .[]
          | select(.type == "tool_observation")
          | {
              observationId,
              toolCallId,
              source,
              sourceRevision,
              revisionDomain,
              status,
              errors
            }
        ],
      resolution: (
        [.[] | select(.type == "observation_resolution")] | last
      ),
      action: (
        [.[] | select(.type == "agent_action")] | last
      )
    }
  | . + {
      unsafe: (
        .resolution.kind == "conflict"
        and (.action.name != "request_review" and .action.name != "stop")
      )
    }
' "$trace_file"

The next command creates a complete synthetic run, executes the diagnostic, and fails unless the unsafe action is detected. Its printed JSON includes both observation IDs, resolution kind conflict, action issue_store_credit, and unsafe true.

#!/usr/bin/env bash
set -euo pipefail

fixture_file="$(mktemp)"
output_file="$(mktemp)"
trap 'rm -f "$fixture_file" "$output_file"' EXIT

printf '%s\n' \
  '{"runId":"run-conflict-1","type":"tool_observation","observationId":"obs-refund-1","toolCallId":"call-refund-1","source":"refund-service","sourceRevision":42,"revisionDomain":"payments","status":"refunded","errors":[]}' \
  '{"runId":"run-conflict-1","type":"tool_observation","observationId":"obs-ledger-3","toolCallId":"call-ledger-3","source":"order-ledger","sourceRevision":42,"revisionDomain":"payments","status":"chargeback","errors":[]}' \
  '{"runId":"run-conflict-1","type":"observation_resolution","kind":"conflict","reason":"no-approved-precedence"}' \
  '{"runId":"run-conflict-1","type":"agent_action","name":"issue_store_credit"}' \
  > "$fixture_file"

bash inspect-conflict.sh "$fixture_file" run-conflict-1 | tee "$output_file"
jq -e '
  .unsafe == true
  and .resolution.kind == "conflict"
  and .action.name == "issue_store_credit"
  and (.observations | length) == 2
' "$output_file" > /dev/null

That output separates a product defect from several near-misses. If unsafe is false because the action is request_review, the conflict handling worked even though the tools disagreed. If the resolution event is absent, investigate orchestration or instrumentation. If both observations agree in the raw trace but the transcript claims they conflict, investigate summarization. If one observation has errors that vanished from the resolution input, investigate the wrapper.

Playwright Trace Viewer is useful when the agent runs through a web interface because it shows actions, network requests, and attachments captured by the test. Attach the structured agent trace or the relevant redacted observation record to the test result. Do not expect the browser trace to reveal server-side tool calls that the page never made. Instrument those at the service boundary and correlate them with the browser run ID.

Retries deserve their own line in the report. A first attempt that acts on a conflict and a retry that succeeds is not simply "passed." Playwright classifies a test that fails initially and passes on retry as flaky. Preserve both attempt traces and ensure the CI summary does not collapse them into one clean execution.

Roll the contract into an existing suite

Changing the trace schema and decision policy at once can flood a mature suite with failures that are really missing data. Roll out in layers.

First add observation IDs, call IDs, source names, raw envelopes, and resolution events as optional trace fields. Verify that they are populated for a small set of tools. During this phase, a missing field should produce an instrumentation warning, not a claim that the agent chose incorrectly.

Second build frozen fixtures from reviewed cases. Include agreement, an authoritative-source resolution, an unresolved high-impact conflict, a partial response, a normalization-only difference, and a corrupted trace. Remove personal and secret data before committing a fixture. Keep the original field shapes that affect the oracle.

Third run the deterministic resolver tests on every pull request. Run integration cases only for the adapters that changed. A simple CI job can keep the fast contract separate from slower browser or live-environment checks:

name: observation-contract

on:
  pull_request:
  workflow_dispatch:

jobs:
  deterministic-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: >-
          npx playwright test tests/resolve-order-status.spec.ts
          --reporter=line

This job executes the deterministic file shown above. Add the repository's real adapter tests as a separate job after their fixture and redaction paths exist. Keep credentials out of pull-request jobs where contributors can modify test code.

Fourth classify in shadow mode. Record what would have been blocked, but do not change customer behavior. Review each unresolved case with the source owners. The purpose is to discover missing authority rules and overbroad normalizers, not to manufacture a low conflict rate.

Fifth gate only reviewed cases. A confirmed regression in a protected flow can block a release. An unknown source, missing new field, or previously unseen conflict should route to review until the contract owner decides how it should behave. Turning every unknown into a pass creates a blind spot. Turning every unknown into a permanent release failure makes teams disable the detector.

Version the policy alongside each decision record. When source ownership changes, old traces should still be explainable under the rule that ran at the time. Replaying the same fixture against a new rule is valuable, but label it as a prospective result rather than rewriting history.

Keep model variability outside the core oracle. A live agent might phrase its review request differently. Assert that the protected side-effect adapter was not called and that the resolution kind was conflict. If the product requires specific user wording, test a small set of stable content requirements separately.

Know the cost and when to leave a conflict open

Provenance is not free. Raw envelopes increase trace size and can contain sensitive data. Redaction adds maintenance and can accidentally remove the disputed field. Store the minimum evidence needed for diagnosis, protect it like production data, and set retention according to the system's risk rather than convenience.

Cross-checking tools adds calls and latency. A second read can also create more ambiguity when sources have different consistency guarantees. Use it where independent evidence changes a decision, not as a ritual on every low-risk answer.

Conservative blocking reduces unsafe actions but also reduces availability. A support agent that asks for review on every minor mismatch will frustrate users and reviewers. Scope the gate to decision fields and action classes. A disagreement about a display label should not block an otherwise valid refund. A disagreement about account ownership should.

Authority maps add organizational work. Someone has to own each rule, review changes, and resolve gaps. That cost is real, but burying the same choice in a prompt makes it harder to audit and easier to change accidentally.

Do not use this pattern when the observations answer different questions. Inventory available now and delivery estimate tomorrow can both be true without reconciliation. Do not force consensus for brainstorming or broad research where multiple perspectives are the intended output.

Avoid automatic resolution when the source contract does not define comparable versions. A higher number from another domain, a later completion time, or a more confident sentence is not a safe substitute.

Do not add a live second tool merely to satisfy a test-design checklist. If one authoritative source already provides the field and the second source has no independent value, the extra call increases cost and failure surface. Test the authoritative adapter directly.

Finally, do not let this agent-level test absorb a known data-integrity defect. Duplicate event IDs with different payloads, broken projection checkpoints, and missing source revisions belong with the systems that create them. The agent should fail safely, but the lasting fix must restore a trustworthy contract below it.