Your trace sample is probably hiding the incident you need

The checkout agent fails only when it hands an order from sales to billing after an inventory timeout. Your trace dashboard contains plenty of clean sales conversations and no copy of the failing path. The sampler did exactly what it was configured to do: represent volume, not risk.

That sample cannot answer the incident question. Agent traffic is usually lopsided, while release risk sits in rare tools, long conversations, handoffs, retries, denied actions, and new workflow versions. A small uniform sample can look statistically tidy and still erase every case the QA team cares about.

Trace grading makes the consequence visible. OpenAI describes a trace as an end-to-end log of an agent's decisions, tool calls, and reasoning steps, and trace grading assigns structured scores or labels to that record. Unlike an output-only check, it can show where the workflow went wrong. That advantage exists only when the selected trace contains the relevant path and enough safe evidence to evaluate it.

Separate three decisions before writing a sampling rule. Capture decides whether the trace exists. Payload policy decides whether model and tool inputs or outputs are included. Evaluation selection decides which available traces are graded. Teams often call all three "sampling," then discover too late that a grading filter cannot recover a run whose tracing was disabled.

Decide what to capture before the run starts

Use an independent request log as the population ledger. It should record a safe request ID, timestamp, workflow version, route, experiment cohort, and coarse outcome. This ledger is not a substitute for a trace. It is the denominator that tells you what the trace sample omitted. If the only count of production requests comes from the tracing system, disabled or dropped traces are invisible by definition.

For head sampling, make the decision from facts available before execution. Good inputs include workflow name, deployment version, request channel, known risk tier, experiment cohort, and a stable request identifier. Do not claim to sample "all failures" at the start of a run because the outcome does not exist yet. You can oversample known precursors, trace broadly with payloads excluded, or replay a discovered failure later, but none of those recreates the original disabled trace.

Deterministic bucketing makes incidents easier to reason about. Hash a stable ID, map it into a fixed range, and include a configured portion of that range. The same request stays selected when retried by the collector, and two services implementing the same policy can agree. Use an identifier already intended for operational correlation, not an email address, prompt text, or secret token.

The Python example below applies application-level policy through the documented RunConfig.tracing_disabled field. It also excludes potentially sensitive model and tool payloads. The sample-rate values are illustrative configuration. Set them from traffic volume, evaluation cost, privacy review, and the minimum slice coverage your team needs.

import asyncio
import hashlib
from dataclasses import dataclass

from agents import Agent, RunConfig, Runner

BUCKETS = 10_000
RATE_BPS = {
    "checkout": 1_000,
    "account_recovery": 2_500,
    "faq": 100,
}

@dataclass(frozen=True)
class Request:
    request_id: str
    route: str
    prompt: str
    workflow_version: str

def bucket_for(request_id: str, policy_version: str) -> int:
    material = f"{policy_version}:{request_id}".encode("utf-8")
    digest = hashlib.blake2b(material, digest_size=8).digest()
    return int.from_bytes(digest, "big") % BUCKETS

def should_trace(request: Request, policy_version: str) -> bool:
    rate = RATE_BPS.get(request.route, 0)
    return bucket_for(request.request_id, policy_version) < rate

agent = Agent(
    name="Order support",
    instructions="Help with order questions and use only the tools configured by the app.",
)

async def handle(request: Request) -> str:
    policy_version = "trace-policy-7"
    selected = should_trace(request, policy_version)
    result = await Runner.run(
        agent,
        input=request.prompt,
        run_config=RunConfig(
            workflow_name="order_support",
            tracing_disabled=not selected,
            trace_include_sensitive_data=False,
            trace_metadata={
                "route": request.route,
                "workflow_version": request.workflow_version,
                "sampling_policy": policy_version,
                "sampling_decision": "included" if selected else "excluded",
            },
        ),
    )
    return result.final_output

if __name__ == "__main__":
    example = Request("req_01JTEST", "checkout", "Where is my order?", "orders-42")
    print(asyncio.run(handle(example)))

The metadata deliberately excludes prompt text and customer identity. It contains only the fields needed to group and audit traces. Even harmless-looking metadata can become identifying when combined, so treat its schema as production telemetry and review it accordingly.

Hash sampling is reproducible, not automatically representative. If request IDs have poor uniqueness, are reused across workflows, or are missing for a channel, the buckets can be biased or the decision can collapse to a default. Validate ID quality in the population ledger. Record excluded decisions there so missing traces can be separated from collector failure.

Preserve rare paths with explicit strata

Uniform selection answers, "What does ordinary traffic look like?" A QA sampling plan also needs to answer, "What breaks at important boundaries?" Build strata around behaviors, not just customer demographics or endpoints. For an agent, behavior includes tool choice, handoff path, conversation length band, retry state, guardrail result, and deployment cohort.

Some strata are known before the run. A request enters the billing route, uses a canary workflow version, or comes from a high-risk account-recovery channel. Oversample those immediately. Other facts appear only during execution, such as a particular tool error or circular handoff. To retain those original runs, you need tracing active while they happen or a tracing processor and storage design reviewed for that use. A head-sampled run that started with tracing disabled has no hidden trace waiting to be promoted.

This leads to two legitimate architectures. At lower volume, capture safe trace structure broadly with trace_include_sensitive_data=False, then select a smaller stratified set for grading. At higher volume or under stricter data controls, head-sample capture and accept that some incidents will have only request logs. The first buys observability with storage and governance cost. The second buys lower exposure and cost by giving up retrospective detail.

Always keep controls. If every captured trace contains an error, you cannot tell whether the path itself is unusual or whether the agent now fails on a common path. For each critical failure stratum, retain successful examples from the same route, workflow version, region or deployment class, and similar input shape. Matching does not need to be perfect, but the comparison factors should be explicit.

A policy file makes those choices reviewable. The example rates and minimums below are illustrative, not reported measurements. capture_mode is an application setting for this sample policy, not an OpenAI SDK key.

policy_version: trace-policy-7
population_ledger: request-events-v3
capture_mode: deterministic_head_sampling
default_rate_basis_points: 100
payloads:
  include_model_input_output: false
  include_tool_input_output: false
strata:
  - name: account_recovery
    match:
      route: account_recovery
    rate_basis_points: 2500
    minimum_daily_review_cases: 20
  - name: checkout_canary
    match:
      route: checkout
      deployment_cohort: canary
    rate_basis_points: 5000
    minimum_daily_review_cases: 30
  - name: billing_handoff
    match:
      expected_handoff: billing
    rate_basis_points: 3000
    minimum_daily_review_cases: 25
grading:
  include_success_controls_per_failure: 2
  quarantine_unknown_schema_versions: true

Minimum review counts are operational targets, not promises that traffic will supply enough cases. If the population contains fewer eligible events, the report should say coverage is unavailable rather than duplicating traces or filling the gap with unrelated cases. Synthetic and replay cases can test the behavior, but label them separately from production observations.

Avoid oversampling only the current incident. A policy that changes after every outage becomes a rear-view mirror. Maintain a core risk taxonomy, add temporary incident strata with an expiry date, and review which ones graduate into the permanent suite. Expiration prevents one old failure from consuming the evaluation budget forever while keeping the incident as a deterministic replay case.

Prove the sample against an independent population

A sample report needs two tables: the population distribution and the selected distribution. Compare them by route, workflow version, tool path, locale, input-length band, and outcome class that the independent ledger can safely provide. Overall sampling rate alone can hide a slice with zero traces.

Track inclusion probability or the applicable policy rate with each selected record. When you estimate population-level quality from unequal strata, raw sample averages overweight the oversampled groups. Weighted estimation may be appropriate, but release gates for high-severity slices often should remain slice-specific. Do not weight away a critical failure because its production frequency is low.

The audit below compares two CSV files. population.csv contains one row per request event, and sample.csv contains one row per captured trace manifest. Both must have request_id, route, and workflow_version. The script reports missing manifests, unexpected manifests, and per-slice counts. It does not inspect trace payloads.

import csv
import sys
from collections import Counter
from pathlib import Path

def read_rows(path: Path) -> list[dict[str, str]]:
    with path.open(newline="", encoding="utf-8") as handle:
        return list(csv.DictReader(handle))

population = read_rows(Path(sys.argv[1]))
sample = read_rows(Path(sys.argv[2]))

population_ids = {row["request_id"] for row in population}
sample_ids = {row["request_id"] for row in sample}

unexpected = sample_ids - population_ids
if unexpected:
    print(f"manifest_error unexpected_sample_ids={len(unexpected)}", file=sys.stderr)

def slice_key(row: dict[str, str]) -> tuple[str, str]:
    return row["route"], row["workflow_version"]

population_counts = Counter(slice_key(row) for row in population)
sample_counts = Counter(slice_key(row) for row in sample)

print("route,workflow_version,population,captured,observed_rate")
coverage_error = False
for key in sorted(population_counts):
    total = population_counts[key]
    captured = sample_counts[key]
    rate = captured / total
    print(f"{key[0]},{key[1]},{total},{captured},{rate:.6f}")
    if captured == 0:
        coverage_error = True
        print(
            f"coverage_error route={key[0]} workflow_version={key[1]} "
            f"population={total} captured=0",
            file=sys.stderr,
        )

if unexpected or coverage_error:
    raise SystemExit(1)

The diagnostic string is intentionally direct. coverage_error ... captured=0 means the trace evaluation has no evidence for a population slice that actually received traffic. That may come from a zero configured rate, a rule mismatch, export loss, bad joins, or a very small probabilistic sample. Compare expected inclusion decisions in the ledger with captured manifests to separate those causes.

Watch temporal coverage too. A sample can match the day's route totals while missing an outage window. Plot request and captured counts in the same time buckets, then compare deployment boundaries. For long conversations, decide whether the population unit is a request, conversation, or full workflow. Mixing units lets one large conversation contribute many traces and distort the sample.

Partial policy rollout is a second failure mode that resembles random undercoverage. One service instance may use trace-policy-7 while another still uses the previous route table. Aggregate counts show a reduced rate, but the real defect is configuration skew. Put sampling policy version and deployment instance class in the population ledger, then split the audit by both. If the observed selection matches each version separately but not the intended fleet policy, fix rollout. Raising the global rate would only make the skew harder to see.

Sampling decisions can also diverge across services if they hash different material. A gateway may bucket the logical request ID while a worker buckets an attempt ID, producing two unrelated decisions for one workflow. Maintain fixed conformance vectors containing policy version, input ID, expected bucket, and expected inclusion. Run those vectors in every implementation. They reveal encoding, delimiter, and hash changes before production coverage drifts.

Deduplicate retries carefully. An infrastructure retry may represent one user experience but multiple executions, and both facts can matter. Store a logical operation ID plus an attempt ID. Report user-level outcomes separately from execution-level tool behavior. Dropping all retries hides reliability defects; counting each as an independent user exaggerates their reach.

Tell selection loss from export delay or missing spans

"No trace" has several causes. Tracing may have been disabled by policy. The run may not have reached the instrumented code. A batch exporter may not have delivered yet. The trace may exist but lack the tool span needed for grading. A dashboard filter may exclude the workflow or date range. Test these hypotheses in that order with IDs and timestamps, not by repeatedly refreshing the page.

The OpenAI Agents SDK documents tracing as enabled by default and wraps runner invocations, model generations, function tools, guardrails, and handoffs in spans. A present trace with an agent span but no expected function span is different evidence from an absent trace ID. First verify that the agent actually selected the tool. Then check whether the tool is an SDK-instrumented function path or custom work that needs explicit instrumentation. Do not label an orchestration choice as collector loss before reading the span tree.

The default batch processor exports in the background every few seconds or when its queue reaches a size trigger, and performs a final flush at process exit. Long-running workers can finish a job before its trace appears in the dashboard. The documented flush_traces() call provides an immediate delivery point after the trace context closes. It blocks while buffered traces and spans are exported, so it adds latency and should be used only where that guarantee is needed.

This worker example follows the documented ordering. The finally block flushes after the trace context has closed, including when the agent run raises an exception.

from agents import Agent, Runner, flush_traces, trace

agent = Agent(
    name="Billing triage",
    instructions="Classify the billing request and use approved billing tools only.",
)

def process_job(prompt: str) -> str:
    try:
        with trace("billing_worker"):
            result = Runner.run_sync(agent, prompt)
        return result.final_output
    finally:
        flush_traces()

if __name__ == "__main__":
    print(process_job("Explain the pending invoice status."))

If flushing makes the trace appear, the problem was delivery timing, not sampling. If the population ledger says included but no manifest arrives after a successful flush, investigate exporter credentials, process logs, network errors, and trace IDs. If the ledger says excluded, the missing trace is policy behavior. That simple decision field saves hours of ambiguous incident debate.

A broken join can present almost exactly like export loss. The trace may have reached the backend, yet the capture audit reports it missing because the population ledger and trace manifest no longer use the same correlation value. This happens when one component records a logical request ID while another records an attempt ID, when a serializer changes letter case, or when a retry reuses one value for several executions. Raising the sampling rate does nothing because capture succeeded. The audit is losing the relationship between two records.

Separate those causes with three checkpoints. The population record should show the decision and correlation value that existed before the run. The exporter-side evidence should show whether a trace identifier was assigned or delivery was acknowledged. The evaluation manifest should show which trace identifier and correlation value it joined. When the population says included, exporter evidence is absent, and no trace identifier can be found after the delivery window, export is the leading fault. When exporter evidence contains a trace identifier but the manifest says unmatched, the join is faulty. When the manifest joins a trace but the required tool span is absent, neither sampling nor joining is the primary defect. Instrumentation coverage is.

Read the diagnostic row field by field rather than trusting its final status. In a healthy included row, the policy version matches the deployed version, the decision is included, a trace identifier is present, the join state is matched, and the export delay falls inside the team's stated delivery allowance. In a broken export row, the first two fields are healthy but the trace identifier remains absent after that allowance. In a broken join row, the trace identifier is present while the join state is unmatched. A misleading row can show a normal overall captured rate even though every unmatched record belongs to the new checkout version. That is why the row also needs workflow version and route, not just a fleet total.

Counts alone can conceal the same join defect. An illustrative audit might report 100 expected inclusions and 100 captured manifests, which looks perfect, while only 92 request IDs match and eight unrelated retry manifests fill the numerator. Those figures are an example, not a production measurement. Report expected inclusions, delivered manifests, uniquely joined requests, duplicate joins, and unmatched records as separate counts. Equality between the first two is not proof that they describe the same executions.

Do not add flush_traces() to every web request reflexively. Blocking export can increase tail latency and reduce throughput. Background export is appropriate when a short delay is acceptable. Immediate flush is more defensible in short-lived jobs, incident replay tools, or a diagnostic canary where trace availability is part of the test.

Protect payloads before expanding coverage

Trace volume is not only a cost question. Model inputs, model outputs, tool arguments, and tool results can contain customer data, credentials, internal URLs, medical details, or payment information. A broad sample with weak data controls creates a second production data store under an observability label.

The Agents SDK documentation states that generation spans and function spans may capture sensitive inputs and outputs, and that RunConfig.trace_include_sensitive_data can disable that payload capture. The default is documented as true. Set the field explicitly for production runs whose policy requires exclusion, and test the resulting trace. Do not rely on a developer's memory of a default.

Payload exclusion preserves useful structural evidence such as span relationships and timing while removing the model and tool content described by that setting. It also reduces what a grader can evaluate. A tool-argument correctness grader cannot operate on arguments you deliberately did not capture. That is a valid trade-off, but it must be visible in the evaluation coverage report rather than treated as a failed model case.

Metadata needs a schema and review. Do not place prompts, email addresses, account IDs, tokens, or raw error bodies in free-form metadata. Prefer coarse enums and version identifiers. Hashing a low-entropy identifier does not necessarily anonymize it, so involve privacy and security owners in the design instead of inventing a home-grown de-identification claim.

Some environments cannot use this trace backend at all. OpenAI's Agents SDK documentation states that tracing is unavailable for organizations operating under a Zero Data Retention policy with OpenAI APIs. In that situation, do not design a sampling scheme around the OpenAI Traces dashboard. Use an approved observability route or local replay artifacts that satisfy the organization's contract and controls.

Access and retention matter after collection. Limit who can view production traces, set retention based on the data class, and audit exports. A redacted trace can still reveal internal workflow names and business logic. Sampling fewer traces does not replace access control, and access control does not justify collecting unnecessary payloads.

Roll out the sampler without turning gaps into green scores

Start by logging decisions without changing capture. Run the proposed policy against the population ledger and calculate expected coverage by slice. This dry run finds unmatched routes, missing request IDs, and strata whose traffic is too low to meet review goals. It costs little and avoids deploying a sampler that silently selects nothing.

Next, shadow capture in a nonblocking environment. Compare included decisions with exported trace manifests. Exercise a successful tool call, a tool failure, a handoff, a guardrail path, and a short-lived worker with flush enabled. These are collector contract tests, not quality scores. A missing required span should fail observability verification before trace grading begins.

Then run trace grades as informational signals. Review failures with matched successes from the same slice. Promote only criteria with sufficient evidence and a named owner. When a slice has no traces, report "no coverage" and decide whether the release can proceed. Never convert missing evidence into a pass because the denominator would look cleaner.

For an existing suite, land the population and manifest schema changes before changing capture rates. Readers must tolerate both the old and new policy versions while the fleet is mixed. Next, deploy the correlation fields and conformance vectors everywhere that makes a sampling decision, but keep the old decision in force. Only after the shadow audit shows stable joins should a small deployment cohort use the new rates. Expand by route, not by fleet percentage alone, because a canary containing only FAQ traffic proves nothing about a billing handoff stratum.

The first break is usually reporting, not execution. A dashboard groups the new policy under an unknown value, a fixture rejects the added manifest field, or a low-volume slice shows no cases during a short canary. Treat schema compatibility and unavailable traffic separately from sampler quality. The change is working when expected decisions agree with captured manifests for each policy version, unmatched and duplicate joins stay visible, and the critical slices receive genuine eligible traffic. A stable fleet-wide average is insufficient evidence.

CI can validate policy structure and deterministic decisions without calling a production trace service. The job below checks the schema, locks fixture IDs to expected buckets, and audits a sanitized capture manifest. It uses only local fixtures, so it cannot prove that production exporting works; a separate canary owns that check.

name: trace-sampling-contract

on:
  pull_request:
    paths:
      - "observability/trace-policy.yaml"
      - "scripts/tracing/**"
      - "tests/tracing/**"

jobs:
  verify-policy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Validate policy schema
        run: python scripts/tracing/validate_policy.py observability/trace-policy.yaml
      - name: Verify stable bucket fixtures
        run: python -m unittest tests.tracing.test_sampling_policy
      - name: Audit sanitized manifest coverage
        run: >-
          python scripts/tracing/audit_capture.py
          tests/fixtures/population.csv
          tests/fixtures/captured.csv

Every sampling improvement costs something. Higher capture rates increase storage, export traffic, and governance scope. More strata make policy harder to reason about and can create tiny samples. Immediate flushing adds latency. Payload exclusion reduces grader capability. Matched controls consume review budget. State those costs next to the risk they buy down so later optimization remains an engineering decision.

Ownership should follow the first divergent checkpoint. The service team owns the pre-run decision and stable correlation value. The observability team owns processor delivery and backend visibility. The evaluation team owns manifest joins, slice denominators, and grading selection. Privacy or security owners approve the payload and metadata schema. A handoff should include a bounded UTC window, route and workflow version, policy version, redacted example request IDs, any trace identifiers, expected and observed counts, duplicate and unmatched counts, and the first checkpoint where evidence disappears. Sending only a dashboard screenshot forces the receiving team to reconstruct the incident.

That division adds maintenance. Dual-version readers and join diagnostics require more fields, fixtures, and retention than a single captured total. High-cardinality correlation data can also make audits slower and more expensive. Keep it for the period needed to investigate sampling and rollout, under the same access controls as the trace metadata, instead of retaining it indefinitely by default.

This technique does not prove that a captured trace faithfully records every action the agent performed. A sampler can include the correct execution, the exporter can deliver it, and the join can succeed while custom tool work remains uninstrumented or a span records an incomplete payload. Sampling audits establish population coverage. Separate instrumentation contract tests must establish trace completeness.

Do not sample when volume is already low enough to capture safely and affordably. Full coverage is easier to audit and avoids selection variance. Do not collect production traces when policy prohibits the data or the team cannot secure access and retention. Do not use production sampling as a substitute for deterministic incident replay; rare safety boundaries deserve authored tests even if traffic has never hit them.

Likewise, avoid a population claim when selection probability is unknown. Manually bookmarked traces, support escalations, and investigator-curated examples are valuable case collections, but they are not probability samples. Label them as incident or audit cohorts and report their results separately. Combining them with sampled traffic without provenance can improve apparent failure coverage while making the overall rate uninterpretable.

Finally, do not interpret a production sample as a benchmark without documenting how it was selected. It reflects traffic, policy, collector health, and evaluation filters at a point in time. Keep the population ledger, policy version, inclusion decision, payload mode, and grading selection together. That record is what lets the next engineer decide whether a green trace score represents the system or merely the part of it the sampler allowed them to see.