Why your offline trace score disagrees with production

Friday's release clears every stored-trace check. On Monday, support finds the agent promising refunds after the payment tool declined them. The offline score and the production score are both internally consistent, but they did not grade the same evidence.

That distinction matters more than the choice of judge model. A trace grade is a decision over a recorded execution. Change the traffic, tool response, trace schema, redaction policy, or rubric, and the meaning of the number changes with it. A useful evaluation system makes those changes visible before anyone calls the gap a regression.

Why the same agent earns two different grades

An offline run starts with selected cases. The team controls the input, expected outcome, tool fixtures, agent version, and grader version. Replaying those cases is valuable because a failure can be reproduced. It is also a partial view. The dataset contains the situations somebody anticipated and chose to retain.

An online run starts with live traffic. Users omit details, paste malformed identifiers, switch languages, retry halfway through a workflow, and ask for exceptions the test set never covered. Dependencies return current errors. Feature flags split behavior. A production sample therefore answers a different question: how the deployed system behaved for the traffic and services it actually encountered.

Trace grading adds another layer. OpenAI describes a trace as an end-to-end log of an agent's decisions, tool calls, and reasoning steps. The practical point for QA is narrower: a grader can use only the evidence that reached the trace. If the refund tool returned a denial but the exporter dropped tool outputs, a trace-level judge cannot distinguish "the agent ignored a denial" from "the tool result was never recorded." Those are different defects with different owners.

Consider a support agent that may issue a refund only after a successful eligibility check. The offline fixture records four events:

1. The user asks for a refund.
2. The agent calls check_eligibility with the order ID.
3. The tool returns eligible: false.
4. The assistant declines the refund and offers escalation.

The production exporter records the request, the tool-call name, and the final answer, but redacts the arguments and tool result. The final answer happens to approve the refund. A final-answer grader can flag the unsafe promise. It cannot prove whether the agent contradicted the tool, called the tool with the wrong order, or received no result. A trace-completeness check must fail before the behavior grade is interpreted.

Start by writing a small, vendor-neutral contract for the evidence every grade requires. The following Python program is runnable as a file. It verifies ordering, required fields, and the relationship between a denial and the final answer. It does not ask a model to infer facts that should have been recorded.

from dataclasses import dataclass
from typing import Any

@dataclass(frozen=True)
class Event:
    kind: str
    name: str | None = None
    payload: dict[str, Any] | None = None
    text: str | None = None

def grade_refund_trace(events: list[Event]) -> tuple[bool, list[str]]:
    reasons: list[str] = []
    checks = [e for e in events if e.kind == "tool_result" and e.name == "check_eligibility"]
    answers = [e for e in events if e.kind == "assistant_message"]

    if len(checks) != 1:
        reasons.append(f"expected one eligibility result, found {len(checks)}")
    if len(answers) != 1:
        reasons.append(f"expected one final answer, found {len(answers)}")
    if reasons:
        return False, reasons

    result = checks[0].payload
    answer = answers[0].text
    if result is None or "eligible" not in result:
        reasons.append("eligibility result is missing the eligible field")
    if answer is None:
        reasons.append("assistant message is missing text")
    if reasons:
        return False, reasons

    denied = result["eligible"] is False
    promised_refund = "refund has been approved" in answer.lower()
    if denied and promised_refund:
        reasons.append("assistant approved a refund after an ineligible result")

    tool_index = events.index(checks[0])
    answer_index = events.index(answers[0])
    if tool_index > answer_index:
        reasons.append("final answer appeared before the eligibility result")

    return not reasons, reasons

fixture = [
    Event("user_message", text="Refund order A-17"),
    Event("tool_call", name="check_eligibility", payload={"order_id": "A-17"}),
    Event("tool_result", name="check_eligibility", payload={"eligible": False}),
    Event("assistant_message", text="This order is not eligible. I can escalate it."),
]

passed, failures = grade_refund_trace(fixture)
assert passed, failures
print("PASS", failures)

This check has a deliberate limitation. It recognizes one explicit policy phrase in English. It will miss softer promises such as "the credit should arrive tomorrow," translations, and references spread across several turns. That is where a calibrated model judge can add coverage. The deterministic layer still earns its place because it protects the evidence contract and catches an unambiguous violation without sampling variance.

The offline and online paths should share this event contract even when they do not share storage. Store normalized fixtures in the repository or test artifact system. Normalize sampled production traces into the same shape after redaction. Never make the grader understand five exporter formats. That couples policy judgment to telemetry plumbing and turns a harmless logging change into an apparent quality shift.

Three dimensions commonly create a gap:

• Population drift changes what users ask. Compare intent, locale, workflow length, tool choice, and error category, not only aggregate scores.
• Evidence drift changes what the grader can see. Compare schema version, missing fields, truncation, redaction, and event ordering.
• Decision drift changes how identical evidence is judged. Compare rubric text, grader implementation, judge model, prompt, and threshold.

A fourth dimension, dependency drift, is often mislabeled as model drift. An agent may make the same tool call in both environments while a live service returns a new error shape or times out. Record the tool outcome category separately from its payload. That lets a QA engineer see a dependency change without retaining confidential response data.

Build one grading contract before comparing scores

A grading contract is more than a prompt. It states the unit being graded, required evidence, criteria, result schema, and action taken for each result. Write it as if another team will have to reproduce the decision during an incident, because eventually they will.

For a purchase agent, one contract might grade an entire trace rather than each turn. It requires a user request, every tool call with a stable call ID, a matching result for each call, the final customer-facing answer, and the deployed agent version. Its hard criteria are straightforward:

• Every side-effecting call has a matching result.
• No confirmation is claimed after a failed side effect.
• The account identifier passed to the tool matches the authenticated account.
• A tool result occurs after its call and before any answer that relies on it.

The same contract can add a semantic criterion: the final answer accurately explains the outcome without disclosing internal fields. A model judge is reasonable for that criterion because wording varies. Keep its output separate from the hard criteria. If both are collapsed into one average, a polished explanation can compensate for a missing authorization check.

Result schemas should preserve per-criterion decisions. A useful record includes pass, fail, or review; criterion ID; concise reason; evidence event IDs; grader version; and trace contract version. The aggregate release decision is derived later. This is less convenient than a single score, but it gives incident responders something they can use.

Do not ask an online grader to recover missing reference labels from the trace. Suppose a travel agent should choose the cheapest refundable flight within a user's time window. The trace may include the selected offer but not the complete search result set. A judge cannot establish optimality from one offer. Either retain a safe representation of the candidate set, record the selected offer's rank at execution time, or grade only the claims supported by available evidence.

The same rule applies offline. A hand-written expected answer is not enough when tool results are nondeterministic. Store the fixture responses that made the expected behavior correct. Otherwise, a replay may compare today's agent against yesterday's answer under a new tool outcome.

Here is a TypeScript diagnostic that compares normalized offline and online NDJSON files. It reports coverage by workflow, schema version, and missing evidence. It does not calculate a quality score, because comparing populations comes first.

import { readFileSync } from "node:fs";

type Trace = {
  id: string;
  workflow: string;
  schemaVersion: string;
  events: Array<{ kind: string; name?: string; payload?: unknown; text?: string }>;
};

function load(path: string): Trace[] {
  return readFileSync(path, "utf8")
    .split("\n")
    .filter((line) => line.trim().length > 0)
    .map((line) => JSON.parse(line) as Trace);
}

function summarize(traces: Trace[]) {
  const counts = new Map<string, number>();
  for (const trace of traces) {
    const hasToolResult = trace.events.some((event) => event.kind === "tool_result");
    const hasAnswer = trace.events.some((event) => event.kind === "assistant_message");
    const evidence = hasToolResult && hasAnswer ? "complete" : "incomplete";
    const key = [trace.workflow, trace.schemaVersion, evidence].join(" | ");
    counts.set(key, (counts.get(key) ?? 0) + 1);
  }
  return counts;
}

function print(label: string, counts: Map<string, number>) {
  console.log(label);
  for (const [key, count] of [...counts].sort()) {
    console.log(String(count).padStart(5), key);
  }
}

const [offlinePath, onlinePath] = process.argv.slice(2);
if (!offlinePath || !onlinePath) {
  throw new Error("usage: npx tsx compare-traces.ts offline.ndjson online.ndjson");
}

print("OFFLINE", summarize(load(offlinePath)));
print("ONLINE", summarize(load(onlinePath)));

Run this before comparing grade rates. If offline contains only refund workflows with complete schema v3 traces while online contains returns, cancellations, and account recovery under schema v2 and v3, there is no meaningful aggregate comparison. Slice to a matched population or repair collection first.

A matched slice does not need to imitate production perfectly. It needs an explicit selection rule. For example, select English refund traces from the same agent version, with one eligibility call and a complete result, then compare them with offline refund fixtures. Record the query or sampling rule alongside the output. Quietly hand-picking production failures after seeing their grades creates a useful bug list but a biased evaluation.

Criteria need stable identifiers. Human-readable names change during editing. Use IDs such as tool.result.present and refund.no_promise_after_denial, then version their definitions. A wording improvement that does not change the decision can remain within a patch version. A changed boundary, such as allowing store credit after refund denial, needs a new contract version and a dual-run period.

Do not overwrite old grades after a rubric update. Regrade a copy and retain both decisions. Historical reports then answer two separate questions: what the release gate decided at the time, and what the current policy would decide on the old trace. Those are both valuable and should not be confused.

Follow the evidence when dashboards disagree

Start with one trace that received different decisions. Aggregate charts hide the exact boundary. Pull the normalized evidence, both grader results, and the metadata needed to reproduce them. Then locate the first stage where the two paths differ.

A compact comparison record looks like this:

trace_id: tr_refund_017
agent_version:
  offline: support-2026-08-01
  online: support-2026-08-01
trace_contract:
  offline: refund-trace/v3
  online: refund-trace/v2
selection:
  offline: fixture/refund/ineligible
  online: production-shadow/refund
evidence:
  offline_tool_result: present
  online_tool_result: missing
grader:
  offline: refund-policy/4.2
  online: refund-policy/4.2
decision:
  offline: pass
  online: review

These values are illustrative labels, not measurements from a production experiment. Their purpose is to show the shape of a useful incident record. The earliest meaningful divergence is the trace contract, followed by missing evidence. Retuning the judge would treat the symptom.

Real grader output should make this visible. A hard check might print:

python grade_refund_trace.py fixtures/refund-denied.json
# FAIL trace.complete: missing tool_result for call_id call_81
# SKIP refund.no_promise_after_denial: required evidence is unavailable

The distinction between FAIL and SKIP matters. If missing telemetry automatically becomes a behavior failure, the agent team inherits exporter defects. If it automatically becomes a pass, unsafe behavior disappears. Mark it review or incomplete, route it to the telemetry owner, and keep it out of the behavioral denominator.

The trace viewer is useful for ordering and linkage. Check whether the expected span exists, whether a tool call and tool result share the correlation identifier your system emits, whether the final answer follows the result, and whether attributes were truncated or redacted. OpenAI's trace-grading documentation presents traces as the basis for inspecting workflows and adding criteria. It does not remove the need to validate your application's payload completeness.

A near-miss can look identical in a top-line report. Imagine online grades fall immediately after a deployment. The first suspicion is a worse model prompt. In one incident, however, the exporter began serializing boolean false as the string "false." A strict check comparing the value to a boolean could treat the field as malformed. In another, the trace is complete but production traffic shifted toward edge cases because a marketing campaign attracted new users. Both lower the pass rate. Field type counts expose the first. Intent and cohort counts expose the second.

Another near-miss comes from clocks. Distributed spans can arrive out of timestamp order even when the execution order was correct. If the contract grades sequence by wall-clock timestamps, network buffering looks like an agent that answered too early. Prefer explicit parent-child relationships, call IDs, and monotonic sequence numbers emitted by the workflow. Use wall-clock time for latency, not as the sole proof of causality.

Truncation creates a quieter failure. A long tool result may be clipped online while fixtures retain it in full. If the grader needs one field near the end of the payload, online traces become ungradable. Do not simply increase retention without a data review. Emit a small, purpose-built outcome object for evaluation, such as status, policy code, and selected item ID. The cost is instrumentation work and another contract to maintain. The benefit is less sensitive data and fewer decisions based on partial text.

Redaction can also change semantics. Replacing every number with a token may hide whether the assistant quoted the correct price. Keep a stable one-way comparison value or a boolean calculated at execution time when the raw value cannot leave production. The evaluator can assert price_matches_selected_offer without storing the amount. This shifts some oracle logic into the application, so test that calculation independently.

When a model judge disagrees across environments on identical normalized evidence, freeze the input and rerun the exact grader version. A repeated mismatch under identical configuration suggests nondeterminism or a hidden dependency. A stable difference points to configuration drift. Log the judge model identifier, rubric version, and sampling settings that your chosen grading surface exposes. Do not claim reproducibility from a seed alone unless the provider documents that guarantee.

Human review is the final diagnostic, not a ceremonial tie-breaker. Give reviewers the same evidence the grader received, the criterion, and a small set of labels. Hide the automated decision during an audit sample when possible. Otherwise, reviewers tend to anchor on it. Record disagreement by criterion so the team can tell whether the rubric is unclear or the judge is weak.

Roll the fix into an existing suite

Begin with observation. Add trace contract validation to offline fixtures and a non-blocking production shadow sample. Report incomplete evidence separately. This reveals whether current traces can support the desired criteria without changing release behavior.

Next, backfill a small set of known incidents and ordinary successes. Include tool denials, malformed user input, retries, timeouts, and handoffs. Each case needs a human-reviewed expected decision per criterion. Do not label only the final outcome. A successful refund can still contain an unauthorized first attempt that a retry hid.

Run the new and old graders side by side. Store both outputs. Review disagreements before setting a threshold. If the new grader fails cases because a field is absent, fix instrumentation or narrow the criterion. If it catches a real violation that the old grader missed, add that case to the permanent calibration set.

Promote deterministic invariants first. They are easier to explain and usually cheaper to run. Keep semantic criteria in shadow mode until reviewer agreement is acceptable for the use case. "Acceptable" is a product risk decision, not a universal percentage. A copy-style criterion can tolerate more uncertainty than approval for a financial action.

CI wiring should fail on contract violations and confirmed policy violations, while preserving review as a separate result. This example assumes the repository already contains the two Python scripts it invokes. It uploads reports even when the gate fails.

name: agent-trace-gate

on:
  pull_request:
    paths:
      - "agent/**"
      - "evals/**"

jobs:
  trace-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Validate fixture contracts
        run: python evals/validate_traces.py evals/fixtures
      - name: Grade deterministic policies
        run: python evals/grade_traces.py evals/fixtures --report artifacts/grades.json
      - name: Upload criterion results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: trace-grades
          path: artifacts/grades.json

Online deployment needs a different control. Do not place a slow semantic judge in the customer request path merely to match CI. Grade asynchronously from a sampled, redacted stream. The immediate application should enforce hard safety rules itself. Online grading detects regressions and routes cases; it should not become the first line of defense for a rule the product already knows.

Sampling policy needs ownership. Random sampling estimates common behavior but rarely captures rare, severe failures. Add targeted strata for side-effecting workflows, tool errors, long conversations, and new versions. Keep the random sample too, or the dashboard will become a risk queue rather than a view of ordinary quality. Label each stream so nobody averages them together.

During rollout, compare counts at every stage: eligible traces, sampled traces, successfully normalized traces, complete traces, graded traces, and final decisions. This funnel reveals silent loss. If ten thousand traces are eligible but only complete ones reach the dashboard, an apparently improved pass rate may mean bad traces were filtered out. Use actual observed counts in your reports. Do not fill missing telemetry with estimates presented as facts.

Set an expiration date for dual-run code. Compatibility layers have a habit of becoming permanent. After the new contract covers the agreed workflows and historical disagreements are resolved, remove the old decision from the release gate. Retain the old result schema only as long as audit requirements demand it.

Pay the costs deliberately

Comparable grading costs engineering time. A shared trace contract forces producers to expose stable event IDs, outcome categories, and versions. Teams that previously logged free-form dictionaries must migrate exporters and fixtures. That work competes with product delivery.

Better evidence also costs storage and raises privacy risk. Full prompts and tool results are convenient for debugging, but they can contain personal or confidential data. Minimize fields before collection, redact before persistence, restrict access, and set retention according to your organization's policy. Trace grading is not permission to copy every production payload into an eval system.

Semantic judges add latency and provider cost. Asynchronous online grading avoids customer-facing delay, but results arrive after the interaction. That makes it useful for detection and investigation, not synchronous prevention. Smaller samples reduce cost and weaken coverage of rare failures. Targeted sampling improves rare-event coverage and distorts aggregate prevalence unless it is reported separately.

Deterministic checks cost flexibility. The refund phrase matcher in the first example is fast and explainable, but it does not understand paraphrases. Expanding it into a large phrase list becomes a brittle language parser. Use hard checks for structural facts and explicit invariants. Use a judge or human review for meaning that genuinely requires interpretation.

Versioning costs dashboard simplicity. Once results preserve trace, agent, rubric, and grader versions, a single trend line becomes several cohorts. That complexity is honest. Collapsing incompatible cohorts produces a cleaner chart with a false story.

Human calibration costs reviewer time. The work cannot be eliminated by asking a stronger model to label its own test set. Reviewers find ambiguous criteria, missing evidence, and cases where the expected answer is wrong. Spend that time on boundary cases and disagreement samples rather than reviewing every easy exact match.

There is also an organizational cost. Offline evaluation is often owned by the model or QA team, while online telemetry belongs to platform engineering and production quality belongs to operations. A shared contract needs an owner who can coordinate all three. Without one, each dashboard remains correct according to its own definitions and useless during a release dispute.

When offline and online should stay separate

Do not force one score when the two systems support different decisions. Offline evaluation is the right release gate for reproducible regression cases. Online grading is the right detector for current traffic, dependency behavior, and unknown failure modes. Their criteria can share IDs while their actions remain different.

Keep them separate when production redaction removes evidence available in fixtures. Report the online criterion as unsupported or use a privacy-safe derived signal. Lowering the offline standard to match sparse production traces throws away useful coverage.

Avoid direct comparison when the population is intentionally different. A production risk sampler overweights payment failures and policy exceptions. An offline suite may balance intents to exercise coverage. Compare within matched strata, not across the aggregate.

Do not use online trace grading as proof that a dangerous action is safe. Sampling observes only selected interactions after execution. Authorization, spending limits, and irreversible side effects need runtime enforcement and ordinary automated tests. A favorable sampled grade cannot replace those controls.

Skip a model judge when an exact assertion answers the question. Tool-call linkage, schema conformance, identifier equality, and prohibited action codes are deterministic. Adding a judge makes the result slower and harder to debug without gaining coverage.

Finally, stop trying to reconcile the scores when the rubric itself has changed. A new policy boundary creates a new evaluation question. Regrade historical traces under the new version if that comparison is useful, but do not splice the new decisions onto the old trend as if nothing changed. The most trustworthy dashboard is the one that admits where comparability ends.