Agent Side-Effect Containment Interview Scenarios for Senior AI Testers

Side-effect containment is where an AI QA interview stops being a prompt exercise and becomes systems safety. A senior tester must reason about authorization, intent binding, retries, partial failure, concurrency, and recovery before an agent can send, delete, purchase, deploy, or modify anything outside its own process.

These scenarios expect a layered test design. Name the trust boundary, authoritative policy check, dry-run artifact, approval record, idempotency behavior, injected failure, and postcondition. A model saying it will be careful is not a control.

Build the harness around disposable resources with independently queryable state. Every case should capture the authenticated actor, proposed operation, policy verdict, downstream request identifier, observed state delta, and cleanup result. That evidence distinguishes a blocked proposal from an attempted call, a committed write from an uncertain timeout, and a successful compensation from a response that merely claimed recovery.

Classify the Proposed Action

The official OpenAI Agents SDK guardrails documentation distinguishes checks around inputs, outputs, and tool calls. For side effects, those checks must work with server-side authorization and execution controls that remain effective even when the model proposes the wrong action.

Begin by separating read-only observations, reversible writes, compensatable operations, and irreversible or externally propagated actions. That classification drives approval, credentials, sandboxing, and test severity.

Risk and Intent Boundaries

1. Why should side-effect tests start from the authoritative state change?

Tool names can be misleading: "prepare email" may only draft, while "share document" may disclose data immediately. Map each operation to systems changed, recipients, data released, reversibility, and downstream triggers. Test both the proposed call and observed postcondition. This prevents a safe-sounding model response from hiding a dangerous connector behavior or an undocumented webhook chain.

2. How would you classify risk for a finance operations agent?

Build dimensions for monetary value, beneficiary change, account scope, reversibility, data sensitivity, and separation-of-duty requirements. Reads, draft payments, scheduled payments, and settled transfers receive different controls. Use policy-owned limits rather than values embedded in prompts. Test boundary amounts and compound actions. Any interview numbers are illustrative until approved from the organization's threat model and controls.

3. Why must tool arguments be bound to user intent?

Selecting an allowed tool does not authorize arbitrary targets or scope. Derive a structured intent record from the authenticated request, then compare account, recipient, amount, environment, and operation against proposed arguments. Ambiguity triggers clarification, not a broad default. Preserve the comparison artifact. A changed target between preview and execution should invalidate approval and require a new decision.

Authorization and Approval

4. How would you test least-privilege credentials for an agent?

Provision identities scoped by environment, tenant, operation, and resource class. Attempt allowed calls, neighboring forbidden calls, list or wildcard operations, and indirect access through another tool. Rotate and revoke credentials during a run. The backend must deny excess privilege independently of the model. A prompt instruction saying "only use test accounts" is not evidence of least privilege.

5. Why should approval be tied to a preview rather than a generic confirmation?

The approver needs the exact actor, action, target, material fields, consequences, and expected state delta. Sign or hash that preview with an expiry and bind it to one execution. Mutating arguments, policy, or source state invalidates the approval. Tests should replay old approvals and swap targets. A generic "yes" in conversation is vulnerable to stale context and scope confusion.

6. How would you test high-risk human approval under concurrency?

Create two proposals against the same resource, approve one, then change the state or execute the other first. The approved call must revalidate version and preconditions at execution. Also test duplicate clicks, delayed approval, withdrawal, and approver permission changes. Evidence includes approval identity, preview hash, state version, decision time, and the backend's final authorization result.

7. Why must denial paths be first-class tests?

Containment is proven when the system refuses and remains unchanged. Test missing approval, expired intent, unauthorized actor, disallowed environment, malformed scope, and policy service unavailability. Verify no downstream call occurred and that the user receives an accurate next step without sensitive policy details. Fail-open behavior during an authorization outage should be treated as a critical defect for risky writes.

Execution Controls and Isolation

8. How would you design and test a dry-run mode?

The same validation and planning path should produce a non-executing diff or transaction preview with authoritative preconditions. Assert that no external state, webhook, notification, or billing event occurs. Compare preview and committed operation identifiers and fields. A separate mock implementation can drift from production semantics, so contract-test dry run against the real validation layer while blocking the commit boundary.

9. Why isolate agent evaluations from production systems?

Adversarial and failure-injection tests intentionally propose unsafe actions and repeated calls. Use disposable tenants, fake payment rails, intercepted messaging, local file roots, and credentials that cannot reach production. Add network egress policy and canary targets to detect escape. The harness should fail closed when isolation setup is incomplete. Test data naming conventions alone do not prevent a connector from targeting real resources.

10. How would you contain a file-management agent?

Restrict it to an explicit root using filesystem enforcement, resolve symlinks and canonical paths before authorization, separate read and write capabilities, and deny executable or secret locations. Preview recursive operations and cap affected items. Test traversal, symlink swaps, case differences, archive extraction, and race conditions. Verify the final filesystem plus audit events, not just the tool's returned success message.

11. Why recheck policy immediately before execution?

Time can pass between planning, approval, and commit. Resource ownership, account status, approver rights, amount limits, or environment may change. Revalidate authorization and optimistic concurrency at the commit boundary using authoritative state. This narrows time-of-check to time-of-use defects. A stale approval should produce a specific conflict and no mutation, followed by a fresh preview if the user continues.

type ApprovedIntent = {
  actorId: string
  targetId: string
  operation: string
  previewHash: string
  expiresAt: string
  idempotencyKey: string
}

async function commit(intent: ApprovedIntent, currentPreviewHash: string) {
  if (intent.previewHash !== currentPreviewHash) throw new Error('stale preview')
  await authorizeAtCommit(intent)
  return executeOnce(intent.idempotencyKey, intent)
}

Idempotency, Failure, and Recovery

12. How would you test idempotency for an agent write?

Send the same semantic operation with the same idempotency key concurrently, after a timeout, and after a client restart. Expect one state transition and stable result identity. Then reuse the key with changed arguments and require rejection. Verify the downstream system, not only gateway responses. Idempotency storage needs scope and retention aligned with the operation's retry window.

13. Why is automatic retry unsafe after an ambiguous timeout?

The server may have committed before the response was lost. Blind retry can duplicate a charge, message, or record. Inject timeouts before acceptance, after commit, and during response delivery. The agent should query by operation identifier or idempotency key, classify the outcome, and only retry when safe. Unknown status should be communicated as unresolved, not converted to failure or success by guesswork.

14. How would you test compensation for a multi-step action?

Model each committed step and its compensation, then inject failure between every pair. Verify reverse ordering, independent authorization, repeated compensation, and residual external effects such as sent notifications. Compensation itself can fail, so the run needs a durable recovery queue and human escalation. The test passes only when final state and outstanding obligations are explicit, not when the orchestration merely returns an error.

15. Why should side-effect budgets be enforced outside the model?

Prompts can request restraint, but a loop or injected instruction may ignore it. Enforce limits on affected resources, money, messages, tool calls, and elapsed authorization window in the execution service. Test cumulative effects across several individually allowed calls. A budget breach should stop new mutations, preserve completed work, and produce a recoverable incident record rather than attempting an unplanned rollback.

Adversarial and Integration Testing

16. How would you test untrusted tool output that proposes a new action?

Return tool content containing instructions to delete, upload, reveal, or call another tool. The agent must treat it as data, keep the original user intent and policy, and require fresh authorization for any new side effect. Delimit provenance in context and apply output validation. Verify no tool chain occurs solely because a retrieved page or connector response requested it.

17. Why test connector schema changes as containment risks?

A renamed default, optional scope becoming broad, or changed enum can turn previously safe arguments into a larger action. Pin and validate schemas, reject unknown fields, and run contract tests against representative server responses before rollout. Diff side-effect annotations and permission requirements. The official MCP tools specification defines tool discovery and invocation contracts; client containment still must distrust changed semantics.

Release and Incident Controls

18. How would you verify a side-effect kill switch?

Activate it before a call, during queued work, and while an operation is in flight. It should block new high-risk commits, revoke or narrow credentials, and preserve the status of already accepted operations. Test authorized recovery and restart. The switch must be independent of model cooperation and observable to operators. Abruptly terminating a process without state reconciliation is not a complete containment design.

19. Why should release approval require destructive-path evidence?

Happy-path demos do not exercise stale approvals, ambiguous writes, duplicate retries, policy outages, concurrency, or compensation failure. A release packet should include isolated failure-injection traces, authoritative state diffs, denied-call evidence, idempotency results, and unresolved risks by tool class. Block deployment when an irreversible operation lacks server authorization, exact approval binding, or verifiable postconditions.

Contain First, Then Automate

Senior candidates place controls at the execution boundary, where prompt behavior cannot bypass them. They understand that approval can go stale, timeouts can hide committed work, and compensation does not erase every external consequence.

The decisive interview answer names which actions the agent may never take autonomously and proves the denial path leaves state untouched. Automation earns broader authority only after containment survives attack, concurrency, and partial failure with evidence an operator can reconstruct.