Your agent's token totals disagree. Here is where to look

The invoice jumps after an agent release, yet the product dashboard says each run uses fewer tokens. One view counted only the final model call. The other included planning calls and retries. Until those records share an identity and a pricing contract, neither total is trustworthy.

This is not a subtraction problem. Cached tokens are not free tokens that vanish from input usage, and input plus output is not a cost formula. A sound reconciliation test keeps four questions separate: what the provider reported, which responses belong to the run, which records are duplicates, and which rate policy applied.

Decide which total you are reconciling

Teams often put three numbers under the label “tokens.” The first is an estimate made before a request. The second is usage returned with a completed model response. The third is an amount derived for billing or internal chargeback. They answer different questions and should have different field names.

A pre-request count helps with context limits, routing, and budgets. OpenAI's token-counting guide documents an endpoint that accepts the Responses request shape and includes framing that may not appear in the visible text. A local tokenizer may be an estimate, especially when the payload includes message framing, tool definitions, images, or files. Never use that estimate as the authoritative actual simply because it was available earlier in the trace.

Response usage is evidence about one provider response. Keep the raw usage object next to the provider name, response ID, API family, model identifier, service tier when supplied, request attempt ID, and receipt time. Store the raw object before flattening it. Field names and detail categories can differ across APIs and can evolve. A normalized row without its source shape is hard to repair after an adapter bug.

Run usage is an aggregation across response identities. An agent can call a model to plan, call a tool, ask the model to interpret the result, retry after a transient error, and call a final model to compose the user response. The browser may show one answer while the run contains several billable responses. A dashboard that attaches usage only to the terminal span systematically undercounts that workflow.

Billing is a priced aggregation. It needs mutually exclusive billable buckets and the rate schedule that was effective for that request. Input and output can have different rates. Cached input can have its own rate. Some current model families also distinguish cache writes. A token total has no currency meaning until those categories, rates, units, and effective dates are attached.

Write a short contract for each field in your telemetry schema. For example, input_tokens means the full provider-reported input count for one response. cached_read_tokens means the reported cached portion of that input, not an amount to deduct from the full count. output_tokens means the provider-reported output count under that API’s definition. total_tokens is preserved when the provider sends it, rather than silently recalculated and substituted.

The distinction prevents the exact error that appears in many cost dashboards: input + output - cached. Consider the usage example in OpenAI's prompt-caching documentation. It shows 2,006 prompt tokens, 300 completion tokens, 2,306 total tokens, and 1,920 cached tokens inside prompt-token details. Subtracting the cached detail would produce 386, which contradicts the documented total. The cached count describes part of the input and its pricing treatment. It does not erase those tokens from usage.

Define completeness before arithmetic. A run with five expected response IDs and four usage records is incomplete, not a zero-token fifth call. A run with six telemetry events but five distinct response IDs may contain a duplicate export. A run with two distinct response IDs for one logical step may contain a retry, and both responses can be legitimate usage. Those cases require identity checks before addition.

Your source of truth also depends on the question. Use response records to debug a single run. Use an authoritative provider usage or billing export to audit account-level charges when available. Use the application ledger to allocate usage to users, features, or experiments. Reconciliation compares sources and explains differences. It does not declare the most convenient dashboard authoritative in advance.

Normalize response usage without corrupting subcounts

Build one adapter per documented API shape. Do not scatter expressions such as usage.prompt_tokens or usage.input_tokens across dashboards. An explicit adapter can validate required fields, retain details, and fail when an unknown schema appears.

The Python example below accepts the usage names in the Responses API reference and the corresponding names in the Chat API reference. It validates non-negative integers, the documented top-level sum, and the fact that cached reads cannot exceed input. It does not subtract cached reads from total tokens. The Chat Completions fixture uses the numbers published in the official prompt-caching documentation.

from dataclasses import dataclass
from typing import Any, Literal


@dataclass(frozen=True)
class Usage:
    input_tokens: int
    output_tokens: int
    total_tokens: int
    cached_read_tokens: int | None
    cache_write_tokens: int | None


def _integer(value: Any, field: str) -> int:
    if type(value) is not int or value < 0:
        raise ValueError(f"{field} must be a non-negative integer")
    return value


def _optional_integer(values: dict[str, Any], field: str) -> int | None:
    if field not in values:
        return None
    return _integer(values[field], field)


def normalize_openai_usage(
    raw: dict[str, Any], api: Literal["responses", "chat_completions"]
) -> Usage:
    if api == "responses":
        input_name, output_name = "input_tokens", "output_tokens"
        details_name = "input_tokens_details"
    elif api == "chat_completions":
        input_name, output_name = "prompt_tokens", "completion_tokens"
        details_name = "prompt_tokens_details"
    else:
        raise ValueError(f"unsupported API family: {api!r}")

    input_details = raw[details_name] if details_name in raw else {}
    if not isinstance(input_details, dict):
        raise ValueError("input token details must be an object")

    usage = Usage(
        input_tokens=_integer(raw[input_name], input_name),
        output_tokens=_integer(raw[output_name], output_name),
        total_tokens=_integer(raw["total_tokens"], "total_tokens"),
        cached_read_tokens=_optional_integer(input_details, "cached_tokens"),
        cache_write_tokens=_optional_integer(input_details, "cache_write_tokens"),
    )
    if usage.total_tokens != usage.input_tokens + usage.output_tokens:
        raise ValueError("total_tokens does not equal input_tokens + output_tokens")
    if (
        usage.cached_read_tokens is not None
        and usage.cached_read_tokens > usage.input_tokens
    ):
        raise ValueError("cached tokens cannot exceed input tokens")
    return usage


def test_documented_chat_completions_example() -> None:
    raw = {
        "prompt_tokens": 2006,
        "completion_tokens": 300,
        "total_tokens": 2306,
        "prompt_tokens_details": {
            "cached_tokens": 1920,
            "cache_write_tokens": 0,
        },
    }
    usage = normalize_openai_usage(raw, "chat_completions")
    assert usage.total_tokens == 2306
    assert usage.cached_read_tokens == 1920


def test_impossible_cached_detail_is_rejected() -> None:
    raw = {
        "input_tokens": 80,
        "output_tokens": 20,
        "total_tokens": 100,
        "input_tokens_details": {"cached_tokens": 81},
    }
    try:
        normalize_openai_usage(raw, "responses")
    except ValueError as error:
        assert str(error) == "cached tokens cannot exceed input tokens"
    else:
        raise AssertionError("invalid usage was accepted")


def test_non_object_details_are_rejected() -> None:
    raw = {
        "input_tokens": 80,
        "output_tokens": 20,
        "total_tokens": 100,
        "input_tokens_details": [],
    }
    try:
        normalize_openai_usage(raw, "responses")
    except ValueError as error:
        assert str(error) == "input token details must be an object"
    else:
        raise AssertionError("an invalid details value was accepted")

The cached-subset test has a real falsifying condition. Changing the adapter to skip that check makes the invalid record pass and fails the test. The list-valued details case also proves that a falsy but malformed object cannot slip through the optional-field path. Add negative rows for missing required fields, strings where integers are expected, negative counts, and inconsistent top-level totals.

Keep optional detail fields optional. An absent cache detail does not prove that the provider performed no caching for every API and model. It means the specific response did not report that detail under the contract you parsed. Your adapter can normalize a documented absent field to zero only when the API contract supports that interpretation. Otherwise preserve None and mark the record unsuitable for that detailed comparison.

Do not force different providers into a false universal schema. input_tokens and output_tokens are useful top-level concepts, but nested categories can carry provider-specific semantics. Preserve provider, api_family, and schema_version. If two APIs define reasoning or media details differently, keep separate extensions rather than inventing an equivalence.

Stream handling deserves a test of its own. Many client libraries expose partial events before a final response object. Attach usage only from the documented event or final object that carries it. Do not add a cumulative usage snapshot on every stream event. If snapshots report 10, then 20, then 30 tokens, summing them produces 60 when the final cumulative count is 30.

Capture raw JSON before any retry wrapper discards it. A wrapper may return only the successful response and hide an earlier response that was received but rejected by application validation. Conversely, an attempt that failed before the provider produced a usage-bearing response should not be assigned a made-up zero. Represent its usage status as unknown or unavailable and let completeness policy decide what happens.

Schema contract tests should run without contacting a model. Save reviewed, redacted response fixtures from every supported API family. When an SDK upgrade changes serialization, the adapter test fails immediately. A live smoke test can then confirm the new shape, but the accounting logic remains deterministic.

Reconcile a run across retries and duplicate events

Response identity is the central key. A logical agent step is not specific enough because one step can retry. A trace span ID is not specific enough because instrumentation can emit the same response on more than one span. An event ID is not specific enough because exporters often generate a new event ID for a replay. Use provider, API family, and provider response ID as a composite identity so unrelated namespaces cannot collide. Keep the other identifiers as correlation attributes.

Deduplication must compare content as well as keys. Two events with the same response ID and identical usage are duplicate observations of one response. Two events with the same response ID but different usage indicate corruption or a mutable record. Silently keeping the first or last hides the problem. Fail reconciliation and retain both raw records.

Retries use different response IDs when the provider created distinct responses. Count each reported response even if the application discards its text. That rule explains why agent usage can exceed the token count visible on the final answer span. It also prevents an optimization project from “saving” tokens on paper by hiding retry attempts from telemetry.

The following runnable reconciler takes the composite response identities expected from completed model-call spans and usage events exported by telemetry. It deduplicates identical observations, rejects conflicting duplicates, and refuses to treat missing usage as zero.

from dataclasses import dataclass


@dataclass(frozen=True)
class Usage:
    input_tokens: int
    output_tokens: int
    total_tokens: int
    cached_read_tokens: int | None = None
    cache_write_tokens: int | None = None


@dataclass(frozen=True)
class UsageEvent:
    run_id: str
    provider: str
    api_family: str
    response_id: str
    attempt_id: str
    export_event_id: str
    usage: Usage


ResponseKey = tuple[str, str, str]


def response_key(event: UsageEvent) -> ResponseKey:
    return (event.provider, event.api_family, event.response_id)


def reconcile_run(
    run_id: str,
    expected_responses: set[ResponseKey],
    events: list[UsageEvent],
) -> Usage:
    observed: dict[ResponseKey, Usage] = {}
    for event in events:
        if event.run_id != run_id:
            continue
        identity = response_key(event)
        for field in ("input_tokens", "output_tokens", "total_tokens"):
            value = getattr(event.usage, field)
            if type(value) is not int or value < 0:
                raise ValueError(f"{identity}: invalid {field}")
        if event.usage.total_tokens != (
            event.usage.input_tokens + event.usage.output_tokens
        ):
            raise ValueError(f"{identity}: inconsistent total_tokens")
        for field in ("cached_read_tokens", "cache_write_tokens"):
            value = getattr(event.usage, field)
            if value is not None and (type(value) is not int or value < 0):
                raise ValueError(f"{identity}: invalid {field}")
        if (
            event.usage.cached_read_tokens is not None
            and event.usage.cached_read_tokens > event.usage.input_tokens
        ):
            raise ValueError(f"{identity}: cached_read_tokens exceeds input_tokens")
        previous = observed.get(identity)
        if previous is not None and previous != event.usage:
            raise ValueError(f"conflicting usage for {identity}")
        observed[identity] = event.usage

    missing = expected_responses - observed.keys()
    unexpected = observed.keys() - expected_responses
    if missing or unexpected:
        raise ValueError(
            f"response identity mismatch: missing={sorted(missing)}, "
            f"unexpected={sorted(unexpected)}"
        )

    values = list(observed.values())

    def sum_if_complete(field: str) -> int | None:
        details = [getattr(item, field) for item in values]
        if any(value is None for value in details):
            return None
        return sum(value for value in details if value is not None)

    return Usage(
        input_tokens=sum(item.input_tokens for item in values),
        output_tokens=sum(item.output_tokens for item in values),
        total_tokens=sum(item.total_tokens for item in values),
        cached_read_tokens=sum_if_complete("cached_read_tokens"),
        cache_write_tokens=sum_if_complete("cache_write_tokens"),
    )


def test_retry_counts_once_per_response_and_export_duplicate_is_ignored() -> None:
    first = UsageEvent(
        "run-7", "openai", "responses", "resp-a", "attempt-1", "event-1",
        Usage(100, 20, 120),
    )
    retry = UsageEvent(
        "run-7", "openai", "responses", "resp-b", "attempt-2", "event-2",
        Usage(90, 30, 120),
    )
    duplicate_export = UsageEvent(
        "run-7", "openai", "responses", "resp-b", "attempt-2", "event-3",
        Usage(90, 30, 120),
    )

    total = reconcile_run(
        "run-7",
        {("openai", "responses", "resp-a"), ("openai", "responses", "resp-b")},
        [first, retry, duplicate_export],
    )
    assert total == Usage(190, 50, 240)


def test_conflicting_duplicate_is_rejected() -> None:
    first = UsageEvent(
        "run-8", "openai", "responses", "resp-a", "attempt-1", "event-1",
        Usage(100, 20, 120),
    )
    conflicting = UsageEvent(
        "run-8", "openai", "responses", "resp-a", "attempt-1", "event-2",
        Usage(101, 20, 121),
    )
    try:
        reconcile_run(
            "run-8", {("openai", "responses", "resp-a")}, [first, conflicting]
        )
    except ValueError as error:
        assert "conflicting usage" in str(error)
    else:
        raise AssertionError("conflicting observations were deduplicated")


def test_response_ids_are_scoped_by_provider_and_api_family() -> None:
    events = [
        UsageEvent(
            "run-namespace", "provider-a", "api-a", "resp-1", "attempt-1", "event-1",
            Usage(40, 10, 50),
        ),
        UsageEvent(
            "run-namespace", "provider-b", "api-b", "resp-1", "attempt-2", "event-2",
            Usage(60, 20, 80),
        ),
    ]
    expected = {
        ("provider-a", "api-a", "resp-1"),
        ("provider-b", "api-b", "resp-1"),
    }
    assert reconcile_run("run-namespace", expected, events) == Usage(100, 30, 130)


def test_event_from_another_run_is_not_aggregated() -> None:
    current = UsageEvent(
        "run-current", "openai", "responses", "resp-a", "attempt-1", "event-1",
        Usage(40, 10, 50),
    )
    foreign = UsageEvent(
        "run-foreign", "openai", "responses", "resp-foreign", "attempt-1", "event-2",
        Usage(900, 100, 1000),
    )
    total = reconcile_run(
        "run-current", {("openai", "responses", "resp-a")}, [current, foreign]
    )
    assert total == Usage(40, 10, 50)


def test_missing_usage_is_not_coerced_to_zero() -> None:
    only_event = UsageEvent(
        "run-9", "openai", "responses", "resp-a", "attempt-1", "event-1",
        Usage(40, 10, 50),
    )
    try:
        reconcile_run(
            "run-9",
            {("openai", "responses", "resp-a"), ("openai", "responses", "resp-b")},
            [only_event],
        )
    except ValueError as error:
        assert "('openai', 'responses', 'resp-b')" in str(error)
    else:
        raise AssertionError("an incomplete run was reported as reconciled")


def test_missing_cache_detail_makes_the_aggregate_detail_unknown() -> None:
    events = [
        UsageEvent(
            "run-10", "openai", "responses", "resp-a", "attempt-1", "event-1",
            Usage(100, 20, 120, 80, 0),
        ),
        UsageEvent(
            "run-10", "openai", "responses", "resp-b", "attempt-2", "event-2",
            Usage(90, 30, 120),
        ),
    ]
    expected = {
        ("openai", "responses", "resp-a"),
        ("openai", "responses", "resp-b"),
    }
    total = reconcile_run("run-10", expected, events)
    assert total.input_tokens == 190
    assert total.total_tokens == 240
    assert total.cached_read_tokens is None


def test_invalid_event_is_rejected_before_aggregation() -> None:
    invalid = UsageEvent(
        "run-11", "openai", "responses", "resp-a", "attempt-1", "event-1",
        Usage(100, 20, 99),
    )
    try:
        reconcile_run(
            "run-11", {("openai", "responses", "resp-a")}, [invalid]
        )
    except ValueError as error:
        assert "inconsistent total_tokens" in str(error)
    else:
        raise AssertionError("an inconsistent response total was aggregated")

The numeric fixtures in this block are synthetic and illustrate arithmetic only. They are not measurements from a production run. Their value comes from the relationships: a duplicate response must not increase the total, a distinct retry must increase it, and an absent expected response must prevent a complete result.

Build the expected-response set from a trace contract, not from the usage events themselves. Store the same provider and API-family namespace on that side. If the usage list supplies both sides, a dropped event disappears from the expected set and the oracle cannot fail. Completed model-call spans, a durable request ledger, or provider response records can supply the independent inventory. Verify that this inventory has its own failure detection.

An agent that launches calls in parallel needs deterministic membership, not timestamp ordering. Correlate child calls to the run using an immutable run ID propagated at request time. A response arriving after the parent span closes still belongs to that run if the contract says the call completed. Wall-clock windows alone can assign late responses to the next run or lose them entirely.

Tool calls are not automatically token events. The model request before a tool call and the model request after its result may each have usage. The tool’s HTTP request may be billed by another unit or not billed at all. Do not create a token row for the tool span unless an actual model response reported usage there. Keep non-token tool costs in a separate ledger.

Fan-out also changes the useful denominator. “Tokens per user answer” includes every branch that contributes to the answer. “Tokens per model response” divides by distinct response IDs. “Tokens per successful task” needs a task outcome oracle. Label the metric. A falling per-response average can coexist with a rising per-answer total when the agent makes more calls.

Convert usage to money only with versioned rates

Token reconciliation can pass while cost reconciliation fails. The usual cause is not arithmetic. It is a missing pricing dimension. Model, date, service tier, region, batch mode, context band, cached-read category, and cache-write category can affect the applicable rate. Use the provider's current pricing documentation for current estimates and retain the exact rate snapshot used for historical chargeback.

Never add prompt and completion tokens and multiply by one blended rate unless your billing contract truly has one rate. The sum is useful for volume. It is not inherently a billing quantity. Keep mutually exclusive categories, then multiply each category by its own rate.

Raw usage details can overlap. Cached reads are contained within input usage in the documented OpenAI shape. A cost adapter must turn overlapping details into exclusive billable buckets according to the provider, model family, and effective policy. Do that transformation in a versioned policy module. Do not make the generic cost function guess what a nested detail means.

The code below starts after that provider-specific split. Counts are exclusive, and rates use Decimal to avoid binary floating-point surprises. The figures are explicitly illustrative. They do not represent current or historical prices for any vendor.

from dataclasses import dataclass
from decimal import Decimal


MILLION = Decimal(1_000_000)


@dataclass(frozen=True)
class BillableTokens:
    standard_input: int
    cached_input: int
    cache_write: int
    output: int


@dataclass(frozen=True)
class RatesPerMillion:
    standard_input: Decimal
    cached_input: Decimal
    cache_write: Decimal
    output: Decimal


def estimated_cost(tokens: BillableTokens, rates: RatesPerMillion) -> Decimal:
    for count in (
        tokens.standard_input,
        tokens.cached_input,
        tokens.cache_write,
        tokens.output,
    ):
        if type(count) is not int or count < 0:
            raise ValueError("billable token counts must be non-negative integers")
    for rate in (
        rates.standard_input,
        rates.cached_input,
        rates.cache_write,
        rates.output,
    ):
        if not isinstance(rate, Decimal) or rate < 0:
            raise ValueError("rates must be non-negative Decimal values")

    amount = (
        Decimal(tokens.standard_input) * rates.standard_input
        + Decimal(tokens.cached_input) * rates.cached_input
        + Decimal(tokens.cache_write) * rates.cache_write
        + Decimal(tokens.output) * rates.output
    ) / MILLION
    return amount.quantize(Decimal("0.00000001"))


def test_each_billable_bucket_uses_its_own_rate() -> None:
    # Illustrative fixture and rates. These are not vendor prices or measurements.
    tokens = BillableTokens(
        standard_input=600_000,
        cached_input=300_000,
        cache_write=100_000,
        output=200_000,
    )
    rates = RatesPerMillion(
        standard_input=Decimal("2.00"),
        cached_input=Decimal("0.20"),
        cache_write=Decimal("2.50"),
        output=Decimal("8.00"),
    )
    assert estimated_cost(tokens, rates) == Decimal("3.11000000")

The test fails if any bucket is omitted, double counted, or multiplied by the wrong rate. It also keeps the unit visible. A common spreadsheet defect treats a per-million rate as a per-token rate, or divides twice because a rate column was already normalized.

Price policies need effective intervals. Store valid_from, an optional valid_to, currency, unit, source URL, and a policy version. Resolve the policy using the request’s billable timestamp under your accounting rule, not the date an analyst runs the report. When a provider changes pricing, add a new row. Editing the old row makes last month’s dashboard change under reviewers’ feet.

Do not hard-code live prices in a test article or in scattered unit tests. Pricing is external data and can change. Test the policy resolver with deliberately labeled fixtures, then populate production rates through a reviewed configuration process. A scheduled check can flag that an official price page changed, but a human should confirm how the new terms map to billable buckets.

Invoice reconciliation may still differ after token costs match. Storage, hosted tools, image generation, web search, fine-tuning, taxes, credits, rounding, and contract discounts can be separate line items. Only claim invoice parity for the products and adjustments your ledger models. Report the remainder by known category or as unexplained, never bury it by altering token counts.

Internal chargeback introduces allocation choices that are not provider facts. A shared planning call might benefit several child agents. You can allocate it to the parent, divide it among children, or place it in platform overhead. Choose one policy and version it. Do not label that allocation “provider usage,” because another reasonable policy could produce a different team total from the same responses.

Diagnose mismatches and roll out a trustworthy gate

Start with counts of identities, not token sums. For one run, print expected model responses, distinct usage-bearing responses, duplicate observations, conflicting duplicates, and missing responses. If those counts differ, token arithmetic is downstream noise.

A useful diagnostic report names the first broken invariant. This illustrative command output shows the shape:

python scripts/reconcile_usage.py --run-id run-2026-08-04-17
# Illustrative output:
# status: incomplete
# expected_response_keys: 4
# distinct_usage_response_keys: 3
# duplicate_observations: 1
# missing_response_keys: [openai/responses/resp_03]
# token_comparison: skipped
# cost_comparison: skipped

The script should derive those values from independent trace and usage sources. It should not print a token mismatch after completeness already failed, because a missing record explains why the sums cannot reconcile. Preserve a machine-readable report beside the human summary so CI and dashboards use the same result.

When identities reconcile but totals do not, compare each response row. A single response mismatch points to normalization or mutation after capture. Equal response rows with a different run total point to aggregation filters, joins, or grouping. Equal run totals with a different cost points to price policy, exclusive bucket mapping, or non-token charges. This decision tree keeps accounting, observability, and agent-runtime owners from debugging the same layer at once.

Duplicate joins are especially common in traces with tool spans. Joining usage to every descendant span can multiply one response by the number of tools it triggered. Inspect the row count before and after each join. Assert uniqueness on provider response ID at the usage grain. If a dashboard needs one row per span, allocate or reference the response rather than copying its full usage onto every row.

Another near-miss is a model alias change. The visible model name in application configuration may resolve to a different billable model or version. Preserve the model identifier returned by the response when the API supplies it, plus the requested identifier. Do not invent a mapping from a friendly alias to a price row without an authoritative rule.

Roll out capture in shadow mode. Store raw usage and normalized usage together. Compare the new run total with the existing dashboard but do not block releases. Sample mismatches and classify them: missing response identity, duplicate export, retry omission, schema parse error, pricing-policy gap, or accepted timing lag. This taxonomy should emerge from real records, not a generic risk list.

Once capture is complete, gate deterministic invariants first. Negative counts, cached reads greater than input, conflicting duplicates, and top-level sums that violate a documented API contract deserve immediate failure. Missing optional details and delayed account-level exports may deserve a pending state. Separate failed, incomplete, and reconciled; treating incomplete as passed recreates silent undercounting.

CI should use redacted fixtures, not live invoice data. Run adapter and aggregation contracts on every pull request. Run provider smoke checks or account-level audits on a schedule with controlled credentials. The workflow below keeps those responsibilities separate and does not expose secrets to pull-request code.

name: usage-reconciliation

on:
  pull_request:
  workflow_dispatch:

jobs:
  contract-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-test.txt
      - run: python -m pytest tests/usage -q

  provider-audit:
    if: >-
      github.event_name == 'workflow_dispatch' &&
      github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
    runs-on: ubuntu-latest
    environment: provider-audit
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -r requirements-test.txt
      - run: python scripts/audit_provider_usage.py --window-hours 24
        env:
          PROVIDER_ADMIN_KEY: ${{ secrets.PROVIDER_ADMIN_KEY }}

Read that if: condition for what it is. It records the intent that the provider audit runs manually from the default branch, and it stops an accidental dispatch from the wrong ref. It is not the thing that keeps the audit secret away from untrusted code. A workflow_dispatch run evaluates the workflow file as it exists on the ref you select, so a branch that edits the condition in the same commit as the code runs with the condition already removed. Guarding a secret with an expression that the guarded branch is allowed to rewrite proves nothing.

The controls that actually hold sit outside the file. Dispatching a workflow requires write access to the repository, which already excludes drive-by contributors and fork pull requests. Above that, bind PROVIDER_ADMIN_KEY to a protected environment, as the environment: provider-audit line does, and let that environment's deployment branch policy decide which refs may reach the secret at all. Add required reviewers if a human should approve each audit run. Those rules are evaluated by GitHub against repository settings, not by an expression the selected branch controls, so a branch cannot edit its way past them. Keep the if: line for clarity of intent, and do not count it a second time as a security boundary.

The rollout still has concrete costs. Raw payload retention increases storage and privacy exposure. Response-level joins add telemetry volume. Strict completeness can delay a report while exporters catch up. Versioned pricing needs an owner. Provider audits may require privileged credentials and rate-limit planning. Decide retention limits, redaction, and access before enabling broad capture.

Set service objectives for the telemetry pipeline separately from usage invariants. For example, you may allow account exports to arrive later than application traces while requiring every completed response to be captured locally. Do not invent a universal delay threshold. Measure your actual pipeline, document the chosen window, and label any numbers as your policy rather than provider behavior.

During migration, dual-write old and new fields. Avoid renaming prompt_tokens to input_tokens in place if historical queries still expect the old semantics. Backfill only from preserved raw responses and record the adapter version. A backfill inferred from aggregate totals cannot reconstruct cache details or per-response identity.

Know when exact reconciliation is not available

Do not compare a local estimate with a completed response and demand equality. They are different observations. Estimates can still catch prompt growth, but their acceptance range must come from measured behavior for the exact request types you support. Label them estimated_input_tokens and never overwrite actual usage with the estimate.

Avoid a strict run-total gate when the trace inventory is incomplete by design. Fire-and-forget child agents, work that continues after the user response, and provider calls made outside the traced process need an ownership decision first. Either bring them into the run contract or reconcile them under another scope. A partial inventory cannot prove a complete total.

Cancelled and failed requests require careful language. If no usage object is returned, you do not know from that application record whether the provider billed any work. Do not assert zero and do not manufacture a token count from elapsed time. Mark it unknown, then use an authoritative provider export or invoice for account-level reconciliation if the provider makes one available.

Do not expect cache hits to be deterministic across isolated test runs. Caching behavior depends on the provider’s documented eligibility and matching rules, and operational cache state can vary. Test arithmetic and field handling with fixtures. Use live cache observations for monitoring or controlled experiments, not as a single-run release oracle unless the provider contract guarantees the condition you assert.

Cross-provider “total tokens” comparisons can also mislead. Tokenization and accounting categories are not universal units of work. Compare cost, latency, and task quality for a defined workload, while preserving each provider’s native usage. Normalization is useful for reporting, but it does not make one provider’s token identical to another’s.

Skip invoice parity when your model covers only token line items. State the narrower claim: token-bearing responses reconcile to the application ledger under pricing policy version X. Finance can then compare that subtotal with the corresponding invoice lines. This is more credible than forcing taxes, credits, tools, and storage into a token formula.

Finally, do not turn a lower token total into a quality pass. An agent can save tokens by dropping context, skipping a safety check, or abandoning a retry. Pair cost regressions with task and safety outcomes. The trustworthy release decision is not “tokens went down.” It is “every expected response is accounted for, the priced categories are correct, and the agent still completes the work it was built to do.”