Stop nested agent tools from outliving the request

An agent answers with “done,” but its inventory tool is still running after the request has timed out. The next attempt starts another copy of the same call, and both reach the service. A timeout existed at the top of the workflow, but no child knew when that budget ended.

Why a top-level timeout does not protect child tools

A timeout is a duration. A deadline is a point in time. The difference matters as soon as work queues, retries, or crosses a process boundary.

Suppose a request receives a six-second budget. Planning and model work consume part of it before the first tool starts. If the tool is then given a fresh six-second timeout, the workflow has reset the clock. A retry can reset it again. Nothing in those local timeouts preserves the original promise to the caller.

An absolute cutoff does preserve it. The root records an expiry such as deadlineAtMs, and every descendant receives that same value. A child can select an earlier expiry when its operation deserves a smaller budget, but it cannot select a later one. At any boundary, remaining time is simply the cutoff minus the current time.

All timings below are illustrative test inputs, not claimed production measurements.

Cancellation is related to a deadline, but it is not the same thing. An absolute timestamp answers, “When is this work no longer useful?” An AbortSignal communicates that the cutoff or another cancellation event has occurred. An operation still has to observe the signal. JavaScript cannot use an AbortController to interrupt arbitrary synchronous code, undo a database commit, or force a remote server to forget a request it already accepted.

AbortSignal exposes cancellation state and its reason. Calling throwIfAborted() throws that stored reason, while AbortSignal.any() combines signals and adopts the first abort reason. This is enough to join parent cancellation with a shorter child limit.

A reliable parent-child contract has five invariants:

1. A child's expiry is equal to or earlier than its parent's expiry.
2. Aborting a parent aborts every descendant that is still running.
3. A child's local expiry does not abort its parent or a sibling.
4. Code checks for an expired signal before it starts another side effect.
5. Logs identify the request, tool call, parent call, absolute cutoff, remaining budget, attempt, and abort reason.

The third rule catches a subtle implementation bug. Teams sometimes hand the same AbortController to every tool and call abort() when one tool reaches its local limit. That cancels unrelated siblings and makes a parallel fan-out look like a system-wide timeout. Children should observe the parent signal, but each child needs its own controller for an earlier local cutoff. Cancellation flows down the call tree, not back up it.

Retries belong to the same tree. An attempt is not a new user request, so it does not earn a new root budget. Before retrying, calculate whether the remaining time can cover backoff, the minimum useful execution window, and cleanup. If it cannot, fail as deadline-exceeded before sending the next request. That decision prevents “helpful” retry code from creating late duplicate writes.

Queueing counts too. A deadline must be created when the request budget begins, not when a worker eventually receives the job. Put the absolute expiry in the queued payload. When a worker starts, it recalculates what remains. Giving the worker the original duration silently erases queue dwell time and is the easiest way to reproduce this bug under load while missing it in local tests.

Build one absolute deadline into every tool call

The following TypeScript module creates a root deadline or a child deadline. It uses only standard AbortController and AbortSignal behavior. The child cutoff is the earlier of its requested duration and the parent's absolute expiry. dispose() clears the local timer after the operation settles, so successful calls do not leave their timer running.

// src/deadline.ts
export class DeadlineExceeded extends Error {
  readonly code = "DEADLINE_EXCEEDED";

  constructor(
    readonly operation: string,
    readonly expiresAtMs: number,
  ) {
    super(
      `${operation} exceeded deadline at ${new Date(expiresAtMs).toISOString()}`,
    );
    this.name = "DeadlineExceeded";
  }
}

export interface DeadlineContext {
  readonly requestId: string;
  readonly operation: string;
  readonly expiresAtMs: number;
  readonly signal: AbortSignal;
  remainingMs(atMs?: number): number;
  child(operation: string, timeoutMs: number): ManagedDeadline;
}

export interface ManagedDeadline extends DeadlineContext {
  dispose(): void;
}

interface OpenDeadlineOptions {
  requestId: string;
  operation: string;
  timeoutMs: number;
  parent?: DeadlineContext;
  cancelSignal?: AbortSignal;
}

export function openDeadline(options: OpenDeadlineOptions): ManagedDeadline {
  if (!Number.isFinite(options.timeoutMs) || options.timeoutMs <= 0) {
    throw new RangeError("timeoutMs must be a positive finite number");
  }

  const nowMs = Date.now();
  const requestedExpiryMs = nowMs + options.timeoutMs;
  const expiresAtMs = options.parent
    ? Math.min(requestedExpiryMs, options.parent.expiresAtMs)
    : requestedExpiryMs;
  const delayMs = expiresAtMs - nowMs;

  if (delayMs <= 0) {
    throw new DeadlineExceeded(options.operation, expiresAtMs);
  }

  const localController = new AbortController();
  const timer = setTimeout(() => {
    localController.abort(
      new DeadlineExceeded(options.operation, expiresAtMs),
    );
  }, delayMs);

  const sources: AbortSignal[] = [localController.signal];
  if (options.parent) sources.push(options.parent.signal);
  if (options.cancelSignal) sources.push(options.cancelSignal);

  const signal =
    sources.length === 1 ? sources[0] : AbortSignal.any(sources);

  if (signal.aborted) {
    clearTimeout(timer);
    signal.throwIfAborted();
  }

  let disposed = false;
  const deadline: ManagedDeadline = {
    requestId: options.requestId,
    operation: options.operation,
    expiresAtMs,
    signal,
    remainingMs(atMs = Date.now()) {
      return Math.max(0, expiresAtMs - atMs);
    },
    child(operation, timeoutMs) {
      return openDeadline({
        requestId: options.requestId,
        operation,
        timeoutMs,
        parent: deadline,
      });
    },
    dispose() {
      if (!disposed) {
        clearTimeout(timer);
        disposed = true;
      }
    },
  };

  return deadline;
}

Callers dispose the deadline in finally. Disposal clears this module's timer; it neither declares success nor cancels the parent.

Here is an HTTP tool adapter that takes a parent context. The custom x-deadline-at-ms header is an application contract in this example, not a standard HTTP header. Both services must agree on its units and interpretation. The signal controls the local Fetch operation, while the header lets the downstream service enforce the same cutoff instead of starting a fresh duration.

// src/inventory-tool.ts
import { DeadlineContext } from "./deadline";

interface InventoryReply {
  available: boolean;
  location: string | null;
}

export async function readInventory(
  parent: DeadlineContext,
  sku: string,
): Promise<InventoryReply> {
  const call = parent.child("inventory.read", 2_000);

  try {
    call.signal.throwIfAborted();

    const response = await fetch(
      `https://inventory.example.test/items/${encodeURIComponent(sku)}`,
      {
        signal: call.signal,
        headers: {
          "x-request-id": call.requestId,
          "x-deadline-at-ms": String(call.expiresAtMs),
        },
      },
    );

    if (!response.ok) {
      throw new Error(`inventory returned HTTP ${response.status}`);
    }

    return (await response.json()) as InventoryReply;
  } catch (error) {
    if (call.signal.aborted) {
      throw call.signal.reason;
    }
    throw error;
  } finally {
    call.dispose();
  }
}

Fetch accepts an AbortSignal, and aborting its controller can abort the request, response-body consumption, and streams. That is a concrete capability of Fetch. Do not generalize it to every SDK. For a database client, message producer, or vendor SDK, check its documented cancellation option. If it accepts no signal, you can refuse to start after expiry and ignore a late result, but you cannot honestly claim that the underlying operation was stopped.

The downstream service should reject an expired timestamp before work begins and keep its transport timeout within the remaining budget. Across machines, record both the received cutoff and the service's current time so clock skew is visible. Keep the original cutoff authoritative even if a remaining duration is sent as supporting data.

Prove propagation with failures that isolate one boundary

One happy-path test proves very little here. The important behavior appears when the child asks for too much time, the parent is cancelled, or parallel children reach different local cutoffs. These Playwright tests exercise those relationships without launching a browser. Playwright Test is simply the runner and assertion library.

// e2e/agent-deadline.spec.ts
import { test, expect } from "@playwright/test";
import { DeadlineExceeded, openDeadline } from "../src/deadline";

test("a child cannot extend the parent cutoff", async ({}, testInfo) => {
  const root = openDeadline({
    requestId: "req-sequential",
    operation: "agent.run",
    timeoutMs: 5_000,
  });
  const child = root.child("catalog.search", 30_000);
  const grandchild = child.child("catalog.rank", 60_000);

  try {
    await testInfo.attach("deadline-contract", {
      body: Buffer.from(
        JSON.stringify({
          rootExpiresAtMs: root.expiresAtMs,
          childExpiresAtMs: child.expiresAtMs,
          grandchildExpiresAtMs: grandchild.expiresAtMs,
        }),
      ),
      contentType: "application/json",
    });

    expect(child.expiresAtMs).toBe(root.expiresAtMs);
    expect(grandchild.expiresAtMs).toBe(root.expiresAtMs);
  } finally {
    grandchild.dispose();
    child.dispose();
    root.dispose();
  }
});

test("parent cancellation reaches a running descendant", () => {
  const caller = new AbortController();
  const reason = new Error("caller disconnected");
  const root = openDeadline({
    requestId: "req-disconnect",
    operation: "agent.run",
    timeoutMs: 10_000,
    cancelSignal: caller.signal,
  });
  const child = root.child("orders.lookup", 10_000);

  try {
    caller.abort(reason);

    expect(root.signal.aborted).toBe(true);
    expect(child.signal.aborted).toBe(true);
    expect(child.signal.reason).toBe(reason);
  } finally {
    child.dispose();
    root.dispose();
  }
});

test("one child's local deadline does not cancel its sibling", async () => {
  const root = openDeadline({
    requestId: "req-parallel",
    operation: "agent.run",
    timeoutMs: 5_000,
  });
  const shortChild = root.child("profile.read", 25);
  const sibling = root.child("policy.read", 2_000);

  try {
    await new Promise<void>((resolve) => {
      shortChild.signal.addEventListener("abort", () => resolve(), {
        once: true,
      });
    });

    expect(shortChild.signal.reason).toBeInstanceOf(DeadlineExceeded);
    expect(sibling.signal.aborted).toBe(false);
    expect(root.signal.aborted).toBe(false);
  } finally {
    shortChild.dispose();
    sibling.dispose();
    root.dispose();
  }
});

The first test catches a sequential reset. The child requests a longer window, but Math.min() clamps it to the existing root cutoff. The grandchild then requests a longer window again, and the second assertion checks that the clamp still holds two levels down. That second level is the one worth keeping: an implementation that hands a child the parent's requested duration instead of its absolute expiry looks correct at depth one and starts drifting past the root at depth two. No timer has to fire for either faulty calculation to fail.

The second case proves that an explicit disconnect reason reaches the child. It distinguishes caller cancellation from DEADLINE_EXCEEDED, which keeps client and budget incidents in separate queues.

The third case covers parallel fan-out. One branch expires without cancelling its sibling or root. It fails if a refactor shares one controller across the tree.

A queue-delay case needs a different fixture. Create the root cutoff before enqueueing, serialize expiresAtMs, then start a test worker after controlled queue release. Assert that the worker receives the original value and refuses to call the tool when remainingMs() is zero. Do not merely assert that the worker has a timeout field. The bug is a plausible timeout field containing a newly reset duration.

A retry case should record every attempt under the same request ID and root expiry. Make the first dependency response retryable, then hold the retry gate until the remaining budget is smaller than your declared minimum attempt window. The correct result is no second network call. Counting calls is stronger evidence than checking only the final error, because an implementation can return deadline-exceeded after it has already launched a doomed retry.

For write tools, record an idempotency key in a server-side fixture. Cancel after the first request is accepted, then exercise the retry path. Assert one committed effect, even if two transport attempts are visible. The deadline limits waiting; idempotency protects state after a late cancellation.

Read the evidence before blaming a slow dependency

Start with the first boundary where the parent cutoff changes or disappears. A final “timed out” label cannot tell you whether the planner reset the duration, a queue worker dropped the timestamp, the adapter ignored the signal, or Playwright reached its own watchdog first.

Emit one structured event when a tool is queued, started, completed, or aborted. Derive remainingMs at the moment of the event. Keep both the absolute timestamp and the derived value because each answers a different question. The timestamp proves lineage across hops; the remaining value shows how much usable budget reached this hop.

// src/deadline-events.ts
import { DeadlineContext, DeadlineExceeded } from "./deadline";

type DeadlineEventName =
  | "tool.queued"
  | "tool.started"
  | "tool.completed"
  | "tool.aborted";

export function writeDeadlineEvent(
  event: DeadlineEventName,
  context: DeadlineContext,
  details: {
    toolCallId: string;
    parentCallId: string | null;
    attempt: number;
  },
): void {
  const observedAtMs = Date.now();
  const reason = context.signal.aborted ? context.signal.reason : undefined;

  process.stdout.write(
    `${JSON.stringify({
      event,
      requestId: context.requestId,
      toolCallId: details.toolCallId,
      parentCallId: details.parentCallId,
      operation: context.operation,
      attempt: details.attempt,
      deadlineAtMs: context.expiresAtMs,
      observedAtMs,
      remainingMs: context.remainingMs(observedAtMs),
      abortCode:
        reason instanceof DeadlineExceeded ? reason.code : undefined,
      abortReason:
        reason instanceof Error ? reason.message : String(reason ?? ""),
    })}\n`,
  );
}

Do not add prompts, credentials, arguments, or bodies to these events. IDs and timing fields reconstruct the deadline path without copying tool data.

Several near-misses produce similar logs but require different fixes:

• An upstream HTTP 504 is not proof that the local deadline fired. Check signal.aborted and its reason. If the signal is still active and the response status is 504, the dependency or an intermediary ended the request first. Investigate that boundary instead of widening every child budget.
• Playwright reports Test timeout of 30000ms exceeded. for a test that reaches its default test timeout. That documented runner message does not prove your custom deadline propagated. Compare the test failure time with the attached deadline events and see whether a tool.aborted event exists.
• A blocked JavaScript event loop can delay delivery of the timer callback. The absolute timestamp may already be in the past even though the abort listener has not run. Check remainingMs() before each new side effect rather than relying only on receipt of the event.
• A server may finish a write after Fetch rejects locally. Client cancellation stops waiting and can abort Fetch work, but it is not a distributed rollback protocol. Query the server fixture by idempotency key before classifying the late commit as a propagation failure.
• A short tool-local cutoff can abort one branch while the root still has time. If sibling calls remain active and the abort reason names that tool, the hierarchy is working. The investigation belongs with that tool's local policy or dependency latency.

Playwright has multiple timeout layers. The test timeout includes the test function, fixture setup, and beforeEach work. Auto-retrying assertions have their own timeout. Action and navigation timeouts can differ again. Keep the application deadline smaller than the test timeout so the test has time to assert the reason, attach evidence, and dispose resources. Do not derive the application's remaining budget from testInfo.timeout; that property is the configured test timeout, not a documented countdown of time left.

Use tracing for runner and browser evidence, then attach the Node-side deadline tree explicitly. A Node fetch() is outside the browser context, so its tool budget will not appear as a browser network operation.

pnpm exec playwright test e2e/agent-deadline.spec.ts \
  --trace retain-on-failure \
  --reporter line

trace_file="$(find test-results -type f -name trace.zip -print -quit)"
test -n "$trace_file"
pnpm exec playwright show-trace "$trace_file"

If the test fails before attachment, inspect the last structured event. tool.queued without tool.started points to admission. tool.started without completion or abortion points to a signal-blind adapter, blocked event loop, or earlier runner failure. A server commit after tool.aborted shifts the investigation to idempotency or compensation.

Find work that escaped the deadline tree entirely

A second failure looks like lost cancellation from the outside but has no broken parent-child edge to inspect. A tool adapter can start an unawaited task, schedule a follow-up after returning, or publish a job without copying the deadline context. The registered tool call aborts correctly, yet the detached work continues and reaches the dependency. Extending AbortSignal handling on the registered call does not affect work that never inherited its signal.

Compare three event shapes. A healthy cancelled child has the same request and absolute cutoff as its parent, a non-null parent call, and a final aborted event whose reason matches the cancellation path. A propagation defect still has a visible parent-child relationship, but the child cutoff is later than the parent, the child signal stays active, or its adapter continues after the abort. Detached work differs earlier. The late dependency request has no corresponding queued and started events under the parent, or its job record begins a new lineage with no declared durable-job boundary. Sometimes the only clue is a downstream request after the root's final event.

The request id is a misleading value here. A developer may copy it into the detached task for logging while omitting the cutoff and cancellation relationship. Matching request ids make the timeline look connected, but they do not prove inheritance. Read the parent call id and absolute cutoff together, then confirm that every dependency call can be traced to a started tool event. Likewise, remainingMs equal to zero on a late event only says the observer calculated lateness. It does not show that admission was checked before the side effect began.

Build a fixture around the dependency, not only the promise returned by the adapter. Let the adapter appear to finish, cancel the root, and then release the scheduled follow-up. The assertion should inspect the fake dependency or queue and prove that no undeclared call arrived. Keep a positive control for an intentionally durable job: it should be acknowledged as a separate lifecycle with its own persisted deadline and cancellation policy. Otherwise a test that forbids every post-response action will break legitimate exports, notifications, or audit delivery.

An existing suite will first expose helpers that fire and forget telemetry, cache warming, or cleanup. Classify those tasks before gating. Pure telemetry may be allowed to outlive the user result under a bounded operational lifecycle, while a follow-up that can create an external effect must not be disguised as cleanup. Land spawn and queue-edge instrumentation first, then update test doubles to preserve lineage. Add the dependency-level negative assertion next. Only after known background tasks have declared owners should an unmatched downstream call fail the suite.

Roll production adoption from observation to enforcement. During observation, compare root completion with later child and dependency events and retain only sanitized timing metadata. Classify every unmatched call as missing instrumentation, an explicitly separate lifecycle, or unauthorized detachment, with evidence for the choice. Upgrade event producers before enabling rejection so an absent event cannot masquerade as forbidden work. Then inject a detached follow-up in a sandbox and prove both the admission boundary and the reconciliation report identify it before applying the rule to real traffic.

The runtime team owns creation of the root cutoff and registration of child work. Tool owners must pass the context and stop starting side effects after expiry. Queue owners preserve the absolute timestamp and define durable-job admission. Downstream service owners expose enough status to distinguish cancellation from commit and own idempotency or compensation. The test owner maintains the fake dependency and runner margin. A handoff should contain the request and call ids, parent relationship, root and child cutoffs, observed times, attempt, abort reason, queue record if present, downstream arrival and commit state, and the first event missing lineage.

The rollout is working when three controls remain distinguishable. A registered child cancelled by its parent emits its final event and creates no new dependency request. An intentionally durable job starts a declared lifecycle with its own persisted cutoff. An injected detached follow-up is reported because the downstream fixture receives a call with no matching lineage. A falling timeout count proves none of these. Reconcile emitted edges with fake or sandbox dependency arrivals, and keep the injected failure red until the unmatched-call detector names it.

Tracking children has a concrete runtime cost. Awaiting bounded cleanup consumes part of the caller's tail budget, so that reserve is unavailable for useful tool work. A registry retains metadata for every child until it settles, which turns forgotten cleanup into process memory growth. Persisting a detached operation as a durable job adds storage, admission, retry, deduplication, and cancellation state. Teams should choose which background work is necessary rather than hiding it behind an unawaited promise.

Deadline propagation tests cannot catch work they never observe. If an adapter opens an uninstrumented side channel, every registered cutoff can pass while the side effect continues. Dependency fakes, queue inspection, and reconciliation of external effects are required to close that blind spot.

Roll the contract into an existing suite without hiding regressions

Inventory every timeout, retry, queue boundary, and promise race. Record which layer owns the root promise and which adapters actually accept an AbortSignal.

Add the cutoff to the internal tool context first. Temporarily keep local limits while logging proposed child expiries. This reveals duration resets and separates durable jobs that need their own lifecycle.

Enforce the invariant on read-only tools, then add parent-cancel, child-clamp, sibling-isolation, queue-expiry, and no-doomed-retry cases. Move writes only after owners confirm idempotency or compensation. A write test that asserts only a rejected promise is incomplete.

Keep the application deadline below Playwright's test watchdog. Identical limits let the runner stop before assertions, attachments, and cleanup, producing an ambiguous failure.

The following job shows the wiring for a repository that already uses pnpm and Playwright. The test command retains a trace on failure, while the test itself attaches the deadline contract. The artifact step runs even after a failed test so the caller can inspect both forms of evidence.

Step order matters here for a reason that has nothing to do with deadlines. pnpm/action-setup has to run before actions/setup-node, because cache: pnpm makes the setup-node step shell out to pnpm store path while it is still running. If pnpm is not on PATH at that moment, the job stops with "Unable to locate executable file: pnpm" before a single test runs. This is the ordering documented in the official setup-node advanced usage guide.

name: agent-deadline-contract

on:
  pull_request:

jobs:
  deadline-contract:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
        with:
          version: 9
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - name: Run deadline propagation cases
        run: >-
          pnpm exec playwright test e2e/agent-deadline.spec.ts
          --trace retain-on-failure
          --reporter line
      - name: Preserve failure evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: deadline-test-results
          path: test-results
          if-no-files-found: ignore

Gate deterministic violations first: a child later than its parent, a missing queue timestamp, a retry after failed admission, or a parent abort lost by an abort-aware adapter. Choose latency-based admission rules from production-safe data and version the policy. Compare late server commits with client aborts, since correct propagation can still expose missing server cooperation or idempotency.

Know the cost and when a deadline is the wrong boundary

The contract adds context to tool signatures, timestamps to queue payloads, events to logs, and timers to process lifecycle. It can also lower apparent success by stopping work that previously finished after the caller's budget. Product owners must choose between a simpler plan, partial result, durable job, or longer root budget.

Cross-service timestamps introduce clock risk. Record received and observed times, monitor clock health, and use a reviewed safety reserve if needed. Never let each service extend the cutoff, because those additions recreate the reset bug.

Every adapter must pass the signal to a supported API or check it between safe steps. Promise.race() only changes which promise wins; it does not stop a signal-blind operation. Use a worker or process boundary when hard termination is safe and required.

Equal child limits can waste a parallel plan's critical path. Allocate local cutoffs by dependency while preserving the parent ceiling. Test the hierarchy and admission decision, not identical durations.

Durable exports, batch evaluations, and long reports may correctly outlive HTTP requests. Acknowledge and persist the job, then give it a separate execution deadline. Do not pretend the original request signal governs it.

Do not treat a deadline as transaction safety. Payments, provisioning, email sends, and other external side effects need idempotency, status lookup, or compensation. The deadline tells the client when to stop waiting or starting more work. It cannot prove whether a remote commit happened.

Synchronous CPU-heavy code is also a poor candidate for signal-only cancellation. If the event loop is blocked, neither timer callbacks nor abort listeners can run promptly. Break the computation into cooperative chunks, move it to a worker, or enforce a process-level limit. Test the chosen isolation boundary rather than asserting that an AbortController preempts JavaScript execution.

Streaming may need an idle timeout plus a maximum duration. Resetting the idle timer after progress is valid only if the maximum lifetime remains fixed and clearly named.

Finally, do not use aggressive deadlines while benchmarking raw dependency latency. The cutoff censors slow observations and changes the workload through cancellation and retries. Run performance experiments with an appropriate measurement design, then use those results to choose operational budgets. Contract tests answer whether descendants obey the chosen boundary; they do not discover the right boundary by themselves.