The eval suite stayed the same, but the bill did not

The nightly evaluation finishes successfully, but its cost is several times higher than the previous run. Nobody added test cases. A retry policy changed, a second grader ran on every answer, and long production traces were copied into both prompts.

Cost regressions rarely live in one obvious request. They multiply across cases, candidates, repeats, tools, and graders. A useful budget therefore tracks work at the case and stage level, then refuses to call a partially executed suite green.

Find the multipliers before setting a currency limit

Start with the execution graph. One evaluation case may trigger a candidate response, a retrieval query, two tool calls, a correctness grader, a style grader, and a retry after a timeout. Comparing two candidates doubles some stages but not necessarily all. Repeating variable cases multiplies them again.

A simple planning identity helps expose the shape:

planned work = cases × candidates × attempts × stages

Real suites do not have the same number of stages for every case, so this is not a billing formula. It is a review prompt. Expand the graph by case type and list which nodes can make billable calls. If the plan says 200 candidate requests but run telemetry contains 400, the difference should have a named cause such as retries or shadow candidates.

Separate at least these cost sources:

• candidate generation;
• model-based grading;
• embeddings or retrieval operations;
• hosted tools or external services with usage charges;
• retry and fallback attempts;
• storage or trace processing when it is billed;
• human review, tracked as operational cost rather than token spend.

Do not assume input and output tokens have the same price. Do not assume cached input, reasoning, images, audio, search calls, or other units follow a text-token formula. Provider and model pricing changes over time. Keep a versioned internal rate card, record the units returned for each request, and reconcile with the provider’s billing data. The article examples use abstract micro-units so they cannot be mistaken for current vendor prices.

A budget needs more than one boundary. The organization may have an account-level provider limit. The eval platform may have a daily budget. Each CI run needs its own cap. A team or suite may need an allocation. The outer limits protect the account; the inner limits tell you which workload exceeded its plan.

Currency is not the only scarce resource. Track request count, input tokens, output tokens, grader calls, elapsed time, and concurrency when they matter. A run can stay under a monetary cap after moving to a cheaper model while doubling latency. Another can exceed a token budget yet cost less because the workload changed. Keep units separate until a rate card intentionally converts them.

The budget policy should say what happens at exhaustion. Silently dropping remaining cases and computing a score is the worst option. The executed set becomes biased toward whatever ran first, often the fastest and easiest cases. Mark the run incomplete_budget, retain partial observations, and list unexecuted cases by slice.

Record a ledger that can explain one expensive case

An aggregate “this run cost 12 units” cannot identify a duplicate grader or retry storm. Store one immutable entry for each billable stage attempt. A normalized ledger lets different providers and internal services share a report without pretending their raw usage fields are identical.

The TypeScript below calculates text-token micro-cost from an explicit rate card. A caller must map provider response data into UsageEntry; the function does not claim a universal SDK response shape. Integer micro-units avoid accumulating binary floating-point errors in the ledger.

type Stage = "candidate" | "grader" | "embedding" | "tool";

type UsageEntry = {
  runId: string;
  caseId: string;
  attempt: number;
  stage: Stage;
  modelKey: string;
  inputTokens: number;
  cachedInputTokens: number;
  outputTokens: number;
};

type TextRate = {
  modelKey: string;
  pricingVersion: string;
  inputMicrosPerMillion: number;
  cachedInputMicrosPerMillion: number;
  outputMicrosPerMillion: number;
};

type CostedEntry = UsageEntry & {
  pricingVersion: string;
  costMicros: number;
};

function charge(tokens: number, microsPerMillion: number): number {
  if (!Number.isInteger(tokens) || tokens < 0) {
    throw new Error(`Invalid token count: ${tokens}`);
  }
  return Math.ceil((tokens * microsPerMillion) / 1_000_000);
}

export function costTextUsage(
  usage: UsageEntry,
  rates: TextRate[],
): CostedEntry {
  const rate = rates.find((item) => item.modelKey === usage.modelKey);
  if (!rate) throw new Error(`No rate for modelKey=${usage.modelKey}`);
  if (usage.cachedInputTokens > usage.inputTokens) {
    throw new Error("Cached input cannot exceed total input");
  }

  const uncachedInput = usage.inputTokens - usage.cachedInputTokens;
  const costMicros =
    charge(uncachedInput, rate.inputMicrosPerMillion) +
    charge(usage.cachedInputTokens, rate.cachedInputMicrosPerMillion) +
    charge(usage.outputTokens, rate.outputMicrosPerMillion);

  return { ...usage, pricingVersion: rate.pricingVersion, costMicros };
}

Keep non-text units in their own calculators and ledger fields. Forcing every tool call into fake tokens makes reconciliation harder. A tool entry might store billableUnits plus a documented unit name. The run summary can convert it with the same versioned-rate principle.

Add identity fields that support grouping: suite, case version, candidate, grader, provider request ID when available, attempt reason, worker, and timestamps. Avoid storing prompts merely for cost analysis if that creates unnecessary sensitive-data retention. Link to a separately controlled trace when investigation requires content.

Idempotency matters when workers retry artifact writes. Give each stage attempt a stable ledger key such as run, case, candidate, stage, grader, and attempt. Reject duplicate keys or make insertion idempotent. Otherwise a reporting retry can double-count spend even though the provider billed once. Conversely, never reuse one key for a second provider request, because that hides real retry cost.

Worked example: a grader quietly runs twice

An illustrative run has one candidate stage and one correctness grader per case. A refactor leaves the old grading hook enabled while adding a new centralized hook. Candidate usage remains flat, but grader entries per case rise from one to two.

Group the ledger by stage, caseId, and grader identity. The duplication becomes obvious before anyone debates prompt length. The fix is execution wiring, not a cheaper model. Add an assertion that each configured single-run grader produces exactly one terminal entry per eligible case.

Two metering sources can impersonate a duplicated grader

There is a second failure with almost the same stage summary. The model client emits a usage entry when the grader finishes, while a gateway billing importer later emits another entry for the same upstream request. Both records are individually valid observations, and their local event identities differ. If the run total treats both as primary charges, it shows two grader rows for one candidate row even though the grader executed once.

Separate execution duplication from dual-source metering with identity and provenance. In a healthy single-grader case, the execution plan contains one grader node, the execution record contains one terminal attempt, and authoritative usage contains one billed request. The ledger may contain a provisional client observation plus a gateway reconciliation observation, but the summary designates only one charge or applies an explicit reconciliation relationship. In a real double execution, two distinct attempts crossed the model boundary and both contribute authoritative usage.

The most useful diagnostic view puts caseId, stage, grader identity, attempt, metering producer, observation role, upstream request identity when available, input units, output units, and terminal state next to each other. For a healthy reconciled row set, the observed record count may be two while unique billable attempts and effective charges remain one. For the broken execution path, unique request count and effective charge count are both two. For the broken accounting path, record count is two, request count is one, and both observations have incorrectly been included as charges.

A misleading value is a locally generated event identifier. Client and gateway observations need different event IDs because they came from different producers, so uniqueness at that level says nothing about inference cardinality. Matching token counts are also weak evidence. Two actual retries can consume identical units, while client estimates can differ from reconciled usage for one request. Read the producer, upstream attempt relationship, and accounting role together. If request-level billing evidence is unavailable, use the execution or gateway audit record to establish call cardinality and label the conclusion with that granularity limit.

This distinction changes the fix and the cost. Double execution belongs to the runner or integration path and consumes real latency and inference budget. Dual-source charging belongs to the accounting model and inflates internal reports without increasing the provider bill. Choosing one producer as the primary view is simple but delays accurate totals when that source arrives late. Reconciling provisional observations into authoritative ones gives earlier visibility, but it requires lineage, update rules, and reports that do not sum both states. That maintenance cost is specific: more retained records, higher-cardinality joins, and extra tests for late or missing reconciliation.

Worked example: retries consume the whole allowance

A transient timeout causes three attempts for several long cases. If retry telemetry records only the successful attempt, cost reports understate spend and reliability reports look healthy. Store every attempt with attemptReason and its observed usage, including a response that arrived after the client stopped waiting when such usage is available from authoritative billing records.

Do not blindly disable retries. A limited retry can make the suite resilient to transient failures. The trade-off is extra spend and a blurred product signal. Report first-attempt success separately, cap attempts, and reserve budget for the maximum allowed policy. A test that passes only after retries may still deserve operational attention.

Worked example: trace payload growth hits two stages

Production failures are added to the eval dataset with full traces. Candidate prompts include every span, and the trace grader receives the same payload again. Case count stays constant while input grows in two stages.

Inspect input tokens by case and stage. If trace length explains the outliers, reduce the payload deliberately: keep relevant tool arguments, outputs, errors, and neighboring decisions; replace unrelated spans with stable summaries or references. The cost is possible loss of causal context. Validate trimmed and full variants on reviewed traces before making truncation the default.

Reserve before calls and reconcile after responses

Actual usage is known too late to prevent the request that caused an overrun. Estimate an upper bound before scheduling, reserve it atomically, then replace the reservation with actual cost after completion. This pattern prevents many workers from each seeing the same remaining balance and spending it simultaneously.

The next class is runnable for a single Node process. It demonstrates reservation and reconciliation, not a distributed lock. A multi-worker runner needs an atomic shared store or queue that provides equivalent semantics.

type Reservation = {
  id: string;
  caseId: string;
  reservedMicros: number;
  settled: boolean;
};

export class RunBudget {
  readonly limitMicros: number;
  private committedMicros = 0;
  private reservations = new Map<string, Reservation>();

  constructor(limitMicros: number) {
    if (!Number.isSafeInteger(limitMicros) || limitMicros < 0) {
      throw new Error("limitMicros must be a non-negative safe integer");
    }
    this.limitMicros = limitMicros;
  }

  reserve(id: string, caseId: string, estimateMicros: number): Reservation {
    if (this.reservations.has(id)) throw new Error(`Duplicate reservation: ${id}`);
    if (!Number.isSafeInteger(estimateMicros) || estimateMicros < 0) {
      throw new Error("estimateMicros must be a non-negative safe integer");
    }
    const outstanding = [...this.reservations.values()]
      .filter((item) => !item.settled)
      .reduce((sum, item) => sum + item.reservedMicros, 0);
    if (this.committedMicros + outstanding + estimateMicros > this.limitMicros) {
      throw new Error(`BUDGET_EXHAUSTED caseId=${caseId}`);
    }

    const reservation = { id, caseId, reservedMicros: estimateMicros, settled: false };
    this.reservations.set(id, reservation);
    return reservation;
  }

  settle(id: string, actualMicros: number): void {
    const reservation = this.reservations.get(id);
    if (!reservation) throw new Error(`Unknown reservation: ${id}`);
    if (reservation.settled) throw new Error(`Already settled: ${id}`);
    if (!Number.isSafeInteger(actualMicros) || actualMicros < 0) {
      throw new Error("actualMicros must be a non-negative safe integer");
    }
    reservation.settled = true;
    this.committedMicros += actualMicros;
  }

  snapshot(): { limitMicros: number; committedMicros: number; openMicros: number } {
    const openMicros = [...this.reservations.values()]
      .filter((item) => !item.settled)
      .reduce((sum, item) => sum + item.reservedMicros, 0);
    return { limitMicros: this.limitMicros, committedMicros: this.committedMicros, openMicros };
  }
}

Estimate conservatively from the known input, configured maximum output, eligible graders, tool allowance, and retry policy. A reservation is not a prediction of the final bill. It is capacity held against a limit. Overly generous reservations reduce parallelism and may stop work that would have fit. Tight reservations risk actual usage exceeding the held amount.

Decide how reconciliation handles actual cost above the reservation. The ledger must record it. The scheduler should stop new work and mark the run over budget; it cannot undo a completed call. Do not clamp actual cost to the estimate merely to keep the accounting green.

Cancelled requests need a terminal reservation state. Release unused capacity only after the execution layer knows no billable work will complete, or reconcile later from authoritative usage. A worker crash should not leave capacity locked forever. Use reservation expiry carefully, since expiring while a remote request still runs can permit an overrun.

Diagnose an overrun without reading every prompt

First verify that the rate-card version and currency match the run. A pricing-table mistake can create a reporting spike without a provider-usage spike. Then compare unit counts by stage. If units rose, drill into cases and attempts. If units stayed flat but cost rose, inspect model mix, rate changes, cached versus uncached usage, and non-token units.

This jq command groups an NDJSON ledger by stage and prints total entries, tokens, and micro-cost. It operates on numeric fields from the normalized record rather than parsing provider-specific responses during diagnosis.

jq -s '
  sort_by(.stage)
  | group_by(.stage)
  | map({
      stage: .[0].stage,
      entries: length,
      inputTokens: (map(.inputTokens // 0) | add),
      outputTokens: (map(.outputTokens // 0) | add),
      costMicros: (map(.costMicros) | add)
    })
' artifacts/eval-cost-ledger.ndjson

Illustrative output from a normalized ledger might look like this:

[
  {"stage":"candidate","entries":120,"inputTokens":480000,"outputTokens":72000,"costMicros":310000},
  {"stage":"grader","entries":240,"inputTokens":690000,"outputTokens":36000,"costMicros":455000}
]

Those figures do not represent an experiment or vendor price. Their diagnostic shape shows twice as many grader entries as candidate entries. Whether that is expected depends on configuration. Compare counts with the run plan before declaring duplication.

Next, sort by cost per case and inspect the top outliers. A single enormous prompt suggests payload growth. Many cases with attempt three suggest retry pressure. A shift from cached to uncached input can change cost without changing total input. A new grader label across every case points to suite configuration.

Check execution completeness alongside cost. An inexpensive run may be cheap because half its workers failed before making requests. Report planned, reserved, started, completed, invalid, and skipped case-stage nodes. The run should not qualify for release merely because its cost stayed low.

Billing data may arrive later than evaluation artifacts. Label preliminary cost as estimated or usage-derived, then reconcile when authoritative data is available. Do not mutate the original ledger. Append reconciliation entries or produce a versioned report with differences explained.

Worked example: case order creates a cheap but biased run

A runner processes cases alphabetically and stops reserving work near its limit. Authentication cases happen to run first, while multilingual and long-context cases sit near the end. The budget is enforced exactly as configured, yet every partial run excludes the same slices. Trend charts then compare different amounts of work while repeatedly missing the expensive paths.

Build the selection plan before execution. Choose required cases and a recorded sample inside each lower-risk slice, estimate the whole plan, and reject or revise it before making calls if it cannot fit. Randomizing queue order alone is insufficient because it changes which coverage disappears rather than guaranteeing coverage. When runtime variance still prevents completion, the coverage artifact should identify the affected slices and the run should remain incomplete.

Worked example: routing changes the rate but not the token count

A fallback configuration begins sending grader requests to a different model key. Input and output totals stay nearly flat, but cost changes under the approved rate card. Grouping only by stage hides the shift. Group by stage, model key, and attempt reason, then compare the routing plan with actual entries.

If fallback was expected, reserve against the permitted worst-case route and report how often it occurred. If it was not expected, investigate availability and configuration. Repricing the entries to the cheaper model would make the report look normal while falsifying which service performed the work.

Reduce spend without hiding a coverage regression

Use tiers. A pull request can run deterministic checks, a fixed set of critical model cases, and a stratified sample of standard cases. A scheduled run can cover the full maintained dataset. A release candidate can add repeated or adversarial cases. State which risks each tier covers.

Always include known high-severity regressions. Sample within lower-risk groups, not across the entire dataset, so a large easy slice cannot crowd out a small important one. Record the sampling algorithm, seed, eligible case IDs, selected IDs, and excluded IDs. That makes two runs explainably different.

Sampling reduces cost and detection power. A failure outside the selection remains invisible until a broader run. Mitigate this with rotating samples, full scheduled coverage, and incident-driven critical cases. Do not advertise the pull-request sample as equivalent to the full suite.

Route outputs to graders selectively. A deterministic schema check does not need a model judge. A failed prerequisite may make later semantic grading meaningless. Conversely, do not replace a nuanced safety review with a cheap string check merely to save tokens. Cost optimization is acceptable only when the new oracle retains the behavior the criterion claims to measure.

Cache reusable deterministic artifacts such as tokenized fixtures, retrieved documents pinned to an index version, and parsed traces. Be careful with model responses. Reusing a response is valid for testing evaluator code, but it does not test a changed candidate. Label replay mode so cached observations are never presented as a fresh end-to-end run.

Shorten judge inputs to the evidence they need. A style grader may need only the final answer. A tool-selection grader needs the requested task and tool decision, not every rendered UI event. A conversation-completeness grader may need the full turn history. Trimming by grader purpose saves input and often improves focus, but an omitted fact can invalidate the grade.

Use early stopping only with an explicit policy. Stopping after the first critical regression saves money, but the report no longer describes full-suite quality. Mark remaining cases unexecuted and schedule a diagnostic or full run when broader impact matters. Never compare that partial aggregate with a complete baseline.

Wire the budget into CI and know when to relax it

Keep policy in a reviewed file or environment owned by the evaluation system. The workflow should run a plan step before any billable work, enforce reservations during execution, and upload both usage and coverage artifacts even on failure.

name: budgeted-llm-evals

on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/ai/**"
      - "evals/**"
  schedule:
    - cron: "31 1 * * *"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 40
    env:
      EVAL_BUDGET_POLICY: evals/budgets/ci.json
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Create case and cost plan
        run: npm run eval:plan -- --policy "$EVAL_BUDGET_POLICY" --out artifacts/plan.json
      - name: Run with reservations and usage capture
        run: >-
          npm run eval:budgeted --
          --plan artifacts/plan.json
          --ledger artifacts/eval-cost-ledger.ndjson
          --coverage artifacts/coverage.json
      - name: Verify spend and completed coverage
        run: >-
          npm run eval:budget:verify --
          --plan artifacts/plan.json
          --ledger artifacts/eval-cost-ledger.ndjson
          --coverage artifacts/coverage.json
      - name: Preserve budget evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: llm-eval-budget-evidence
          path: artifacts/

Roll out with observation before enforcement. Capture ledger entries for existing runs, reconcile them, and identify normal variation by suite and stage. Set the first limit from an approved operating envelope, not from a suspiciously cheap single run. Then alert near the boundary before failing builds.

An existing unmetered suite needs a staged migration. First add stable run, case, stage, and attempt identities while leaving scheduling unchanged. Confirm that every executed node produces one terminal ledger record and that rerunning the reporter does not duplicate entries. Next, reconcile several completed runs against authoritative billing at the level your provider makes available. Differences may reveal missing fallbacks, unrecorded tools, rate-card mistakes, or reporting delays.

After the ledger is trustworthy, add a plan artifact and compare planned nodes with completed nodes. Keep reservations in observation mode and log where they would have refused work. This exposes poor estimates without interrupting CI. Only then enforce a generous run limit, followed by tighter suite allocations once coverage owners agree which work is required. Each stage proves a different control. Turning on a hard cap before identity and reconciliation are correct converts accounting bugs into skipped tests.

During migration, retain the old aggregate cost report but derive it from the new ledger. Do not maintain two independent totals. Reviewers should be able to move from the familiar run number to its stage and case entries, while the team gains confidence that the detailed model adds up to the same controlled result.

Ownership has to follow the failing boundary. The evaluation-platform team owns ledger semantics, reservation atomicity, and incomplete-run behavior. The model or tool integration owner owns request attribution, retry reasons, and usage capture. The suite owner decides which cases cannot be sampled away. Finance or cloud-cost operations owns reconciliation rules and the approved rate-card source. A handoff should include the run and plan identities, the affected case-stage keys, expected and observed cardinality, attempt reasons, the pricing version and currency, the billing time window, reconciliation delay assumptions, skipped slices, and a redacted reproduction. It should also name the decision required. Sending a screenshot of the total forces the receiving team to rebuild the investigation and often sends the issue to the wrong owner.

Provide a reviewed override for exceptional full runs, incident reproduction, or deliberate model comparison. An override should name an owner, reason, expiry, and expanded limit. Hidden environment edits teach engineers to bypass the system; a visible exception preserves control and auditability.

Do not apply a hard per-run currency gate while the rate card is known to be stale. Gate on trusted usage units and repair reconciliation first. Do not stop a safety-critical incident investigation because its ordinary CI allocation is too small. Move that work to an approved incident budget rather than pretending it is a routine run.

Avoid optimizing a new suite before its minimum coverage is understood. Early evaluation work is partly discovery. Record cost, but let reviewers learn which cases and graders carry signal. Tighten the budget once the team can state what must never be sampled away.

This technique does not catch a quality regression whose execution shape and usage remain normal. A candidate can produce a newly unsafe answer with the same model, token count, grader count, and cost as yesterday. The ledger can prove that the expected work ran and was accounted for. It cannot prove that the prompts, graders, or product outputs were correct. Keep behavioral gates and reviewed cases independent of the cost gate.

The budget has done its job when an engineer can answer four questions from artifacts alone: what work was planned, what work executed, which stage spent the allowance, and what coverage was lost when scheduling stopped. A cheap green check that cannot answer those questions is not controlled evaluation. It is incomplete work with a small bill.