Agent Token Budget and Tail-Latency Interview Scenarios

Agent performance is a constrained quality problem, not a race to minimize one number. Tokens are consumed across prompts, history, retrieval, tool results, output, and retries. Latency accumulates through model calls, tools, orchestration, queueing, and the slowest branches. A senior QA candidate must show which budget was spent, which path created the tail, and what quality evidence permits a reduction.

These 18 scenarios ask for workload models, trace boundaries, paired evaluations, and release controls. Strong answers keep time to first token, completion time, and end-to-end task latency distinct, and they refuse to declare an optimization successful when it merely truncates difficult users.

Build the Performance Evidence Chain

The OpenAI Agents SDK tracing guide describes timed spans for generations, tools, guardrails, and handoffs, with generation and tool data available for diagnosis. LangSmith's evaluation guidance provides the complementary quality-evaluation layer. Performance changes should be read through both: execution traces explain time and usage, while evals test what the optimization damaged or preserved.

An interviewer should reject any p95 claim that lacks a population and time boundary. A percentile for cached FAQ traffic does not describe multi-tool account recovery, even if both routes share one agent name.

Define Token Budgets by Component

1. Why should a token budget be allocated by workflow component?

One total does not reveal whether history growth, retrieved noise, verbose tool output, repeated planning, or final generation consumed the capacity. I would record usage by generation attempt and attribute input segments where the application can measure them. Set component guardrails tied to behavior, not arbitrary equal shares. Fine-grained accounting adds instrumentation work, but it identifies the safe optimization lever and catches cost shifted from one step to another.

2. How would you test conversation-history trimming?

Build conversations where early turns contain identity, constraints, resolved decisions, and irrelevant chatter. Compare baseline with trimming on the same cases, then evaluate task completion, policy adherence, memory facts, token use, and latency. Inspect what the summarizer or retention rule removed. A smaller prompt is not a pass if the agent asks users to repeat information or violates an earlier restriction. Test long-tail histories, not only the median turn count.

3. What would you do when retrieved context consumes most of the budget?

Measure relevant-fact coverage, duplicate chunks, ranking, and context position before lowering top-k. Try deduplication, metadata filters, reranking, chunk repair, or query decomposition one change at a time. Evaluate recall, faithfulness, answer completeness, and token use on the same RAG set. Cutting context can improve speed while creating silent evidence omissions, so critical multi-hop and conflicting-document cases must remain protected.

Measure the Right Latency Boundaries

4. Why separate time to first token from end-to-end task latency?

Streaming can make the interface feel responsive while tools continue, the final answer stalls, or the task remains incomplete. Record request acceptance, first useful output, final output, and durable task completion as distinct milestones. For non-streamed or tool-first paths, first token may have little product meaning. The correct service objective follows the user journey. Optimizing presentation latency alone can hide a slower or less reliable workflow.

5. How would you attribute latency in a trace with nested model and tool spans?

Build the critical path from parent-child timing rather than adding every span duration. Parallel children overlap, retries create repeated work, and gaps may represent orchestration or queueing. Annotate each attempt and external dependency, then compare successful fast and slow traces in the same workload slice. Span boundaries must be validated with injected delays. A timeline is evidence only if the team knows what each start and end actually surrounds.

6. What test reveals whether streaming backpressure is part of completion latency?

Run the same controlled output with a normal consumer and an intentionally slow consumer, recording provider completion when available, server emission, client receipt, and trace closure. If the generation span includes blocked delivery, its duration will change with consumption. Decide which boundary belongs in user latency and which belongs in model diagnostics. Ignoring backpressure can blame generation; excluding it from end-to-end measurement can hide a real client experience problem.

Diagnose Tail Latency

7. Why should slow traces be grouped by execution path before optimization?

Tail requests may contain different mechanisms: a rare tool, larger context, safety review, retries, cold dependency, or fan-out. Group by intent, tool sequence, context band, retry status, and outcome, then compare within groups. One global p95 can move because traffic mix changed. Path grouping may create sparse samples, so report evidence sufficiency, but it prevents optimizing the common fast path while the actual tail remains untouched.

8. How would you investigate latency caused by parallel tool fan-out?

Inspect sibling span start and end times, concurrency limits, the slowest required branch, cancellation of unnecessary branches, and merge time. Inject controlled delays into each tool to verify the critical-path model. More parallelism can reduce serial time but increase contention, rate limiting, and wasted calls. The candidate should ask whether every branch is required before launch and whether partial results can satisfy the task safely.

9. What evidence distinguishes provider queueing from application queueing?

Instrument admission, worker dequeue, outbound request start, response milestones, and provider-reported timing when available. Correlate with concurrency, rate-limit responses, and host resource metrics. A long generation span that starts before the network request can include application waiting; a gap before the span may indicate orchestration. Without boundary tests, assigning ownership is speculation. The fix may be capacity, concurrency control, request shaping, or provider escalation.

Trade Tokens and Latency Against Quality

10. How would you evaluate a lower output-token limit?

Use paired runs on tasks requiring short, medium, and genuinely long answers. Detect truncation structurally, evaluate required-fact coverage and task completion, and inspect whether the model wastes budget before the answer. Compare latency and usage by slice. A lower cap can prevent runaway output, but it may cut citations, safety instructions, or final steps. The release rule should distinguish concise success from abrupt termination.

11. Why can fewer planning steps increase total latency?

An overly tight planning budget may choose the wrong tool, require recovery, or generate repeated clarification. Compare trajectory length, retries, tool errors, and completion, not just first-pass tokens. Hold workload constant and inspect whether the optimized agent shifts work downstream. A small amount of deliberate planning can reduce total effort; unlimited planning can also loop. The operating point must be measured end to end with quality constraints.

12. How would you test a context summarizer introduced for performance?

Create source histories with must-retain facts, revoked instructions, uncertainty, and adversarial content. Compare summarized state with a fact and policy checklist, then run downstream tasks using only that summary. Measure summary generation cost, added latency, and savings in later turns across realistic conversation lengths. Summarization pays an upfront cost and can create irreversible information loss, so the break-even and quality evidence must be slice-specific.

{
  "performance_policy": "illustrative_only",
  "latency_objectives": "derive_from_user_journey_and_baseline",
  "token_budgets": "allocate_by_component_after_measurement",
  "quality_vetoes": ["unsafe_action", "missing_required_fact", "truncated_contract"],
  "release_method": "paired_eval_plus_reversible_cohort"
}

Design Load and Reliability Tests

13. How would you create a representative agent load model?

Use production-informed proportions for intents, context-size bands, conversation depth, tool paths, streaming behavior, and failure modes, with a separate risk mix for rare expensive journeys. Model arrival patterns and concurrency rather than only a fixed loop. Protect privacy by replaying sanitized or synthetic equivalents. The representative mix estimates user experience; stress and worst-case mixes answer different capacity questions and should not be merged into one benchmark claim.

14. What should a load test assert besides latency?

Assert task success, tool correctness, error and timeout classification, trace completeness, token-budget adherence, cancellation cleanup, and absence of duplicate effects. Monitor queue depth and dependency saturation. Under load, an agent may appear faster because it returns early errors or shorter degraded answers. Quality sampling must remain active, while deterministic invariants cover every run where feasible. Performance without valid work is not throughput.

15. How would you test retry storms under dependency failure?

Inject a bounded tool or model failure, then observe retry eligibility, backoff, concurrency, token multiplication, queue growth, cancellation, and recovery after the dependency returns. Assert one logical side effect through idempotency controls. Use a safe environment and explicit stop limits. Retries improve isolated resilience but can amplify shared outages; the test should prove the system sheds or delays work predictably instead of synchronizing more demand onto the failing service.

Make the Release Decision

16. How would you define a performance gate for an agent release?

Declare workload revision, latency milestones, percentile populations, token components, error handling, and critical quality vetoes. Compare baseline and candidate on paired cases and a realistic load profile. Numeric targets are derived from product objectives and measured behavior; any interview number is explicitly illustrative. Require no severe slice regression and preserve trace exemplars for the tail. One aggregate speedup cannot compensate for unsafe or incomplete tasks.

17. What would you do when median latency improves but p95 worsens?

Confirm sample size and traffic mix, then isolate which paths entered the tail. Inspect retries, context size, tools, fan-out, and queueing against the baseline. If the affected slice is important, hold or constrain the candidate despite the median win. A limited rollout may be acceptable with route-specific fallback and stop conditions. The decision follows user impact and severity, not a preference for one percentile.

18. How would you score a candidate who proposes a universal token cap for every request?

Score below senior level until they segment workloads and define behavior when a budget is exhausted. Ask how the cap covers retrieval, tools, retries, and output, and which quality tests protect long or multi-hop tasks. Shared ceilings can provide a final safety bound, but per-path budgets and graceful degradation are needed for useful control. A universal cap is simple operationally and often arbitrary semantically.

Optimize the Critical Path, Then Prove the Task

The performance story must reconcile four views: representative demand, critical-path timing, component token use, and task quality. Traces expose where work happened; evaluations determine whether reducing that work preserved the product contract.

The decisive QA close is to reject optimizations that win by dropping evidence, truncating difficult cases, or failing early. Change one budget or path, replay the same workload, inspect the tail, and enforce critical quality vetoes. Release only when faster execution still completes the intended task safely across the users most likely to suffer the slow path.