The human approved one action, but the agent executed another

An operator approves a $240 invoice payment to one supplier, but while the run is paused, the pending tool arguments change to $2,400 and a different bank account. The execution service still sees a valid green approval, even though the reviewer did not authorize the action that happened.

That gap is a time-of-check, time-of-use defect at the human boundary. Effective agent human approval scope mutation testing freezes what the person saw, changes one meaningful field after the decision, and proves the executor rejects the new scope through deterministic authorization code, not a model grader that decides whether two actions sound similar.

Find the field that moved after approval

Most approval flows hold at least three representations of one proposed action. The agent produces tool arguments. The interface turns them into a human-readable card. The execution service later receives a request. A scope mutation exists when an authorization-relevant value differs across those representations and the system proceeds without a new decision.

The obvious example is an amount increase. Less obvious mutations are often more dangerous:

• the amount stays fixed while the currency changes
• a file deletion moves from one generated file to its parent directory
• one email recipient becomes a distribution list
• draft_message becomes send_message
• a read operation gains a write flag
• the tenant stays the same while the customer or account changes
• an attachment is added after the message body was reviewed
• one deployment environment changes from staging to production

The first testing job is to define which fields carry authorization meaning. Do not hash an arbitrary request object and assume coverage. Requests contain timestamps, trace ids, retry counters, display labels, and server defaults. If those volatile values are included, legitimate retries will need another approval. If a dangerous option is omitted, the digest remains unchanged while behavior changes.

Build a typed scope for each tool family. A payment scope might contain tenant, invoice, destination, amount in minor currency units, currency, and operation. A file scope needs a normalized resource identifier, operation, and recursive flag. An outbound-message scope needs channel, recipients, attachment identifiers, and whether the action sends or drafts. The scope is a security contract, not a copy of whatever JSON the agent happened to produce.

There are two useful approval models.

Exact approval binds the decision to one fully specified action. Any meaningful change needs another review. Use it for payments, destructive operations, privilege changes, public messages, and other effects where the person expects to approve the final value.

Bounded approval grants a constrained set of actions. A reviewer might permit refunds for one order up to $100, or read access to one repository for an hour. The executor checks the requested action against the constraints. It does not compare one fixed digest. This model reduces repeated prompts but asks the reviewer to understand a policy, which is harder than reviewing one concrete action.

Avoid a third, accidental model: approval by semantic resemblance. A model grader may say that "$240 USD to supplier A" and "$2,400 USD to supplier A" have the same intent. That can be a useful analysis signal, but it is an unacceptable authorization oracle. Values with operational meaning need typed comparisons.

Scope mutation is related to token replay but not identical. Replay uses an approval more than once. Mutation uses a decision for a different action, and it can happen on the first use. Keep separate reason codes such as replayed and scope_mismatch. Otherwise an investigator sees "invalid approval" and cannot tell whether the token, the action, or the lifecycle failed.

The review screen matters as much as the backend object. If the stored scope contains a recipient hidden behind "1 contact," the server may enforce it perfectly while the human remains uninformed. The durable record should connect three things: the typed approved scope, the exact view model sent to the interface, and the approved decision. A screenshot is useful supporting evidence, but a versioned structured view model is easier to compare, redact, retain, and test.

Represent the approved action as typed scope

The example below defines one domain instead of pretending to canonicalize every possible JSON value. It accepts a payment request only when the key set and types are exact. Money uses integer minor units. Booleans are rejected where integers are expected because Python treats bool as a subclass of int. Unknown fields fail closed so a newly introduced execution option cannot bypass review unnoticed.

from __future__ import annotations

from dataclasses import asdict, dataclass
from hashlib import sha256
from json import dumps
from typing import Any


class InvalidScope(ValueError):
    pass


@dataclass(frozen=True)
class PaymentScope:
    tenant_id: str
    operation: str
    invoice_id: str
    destination_account_id: str
    amount_minor: int
    currency: str

    @classmethod
    def parse(cls, value: dict[str, Any]) -> "PaymentScope":
        expected_keys = {
            "tenant_id",
            "operation",
            "invoice_id",
            "destination_account_id",
            "amount_minor",
            "currency",
        }
        if set(value) != expected_keys:
            missing = sorted(expected_keys - set(value))
            extra = sorted(set(value) - expected_keys)
            raise InvalidScope(f"missing={missing}, extra={extra}")

        text_fields = expected_keys - {"amount_minor"}
        for field in text_fields:
            if not isinstance(value[field], str) or not value[field]:
                raise InvalidScope(f"{field} must be a non-empty string")

        amount = value["amount_minor"]
        if isinstance(amount, bool) or not isinstance(amount, int):
            raise InvalidScope("amount_minor must be an integer")
        if amount <= 0:
            raise InvalidScope("amount_minor must be positive")
        if value["operation"] != "pay_invoice":
            raise InvalidScope("operation must be pay_invoice")
        if len(value["currency"]) != 3 or not value["currency"].isupper():
            raise InvalidScope("currency must be a three-letter uppercase code")

        return cls(**value)

    def canonical_bytes(self) -> bytes:
        encoded = dumps(
            asdict(self),
            sort_keys=True,
            separators=(",", ":"),
            ensure_ascii=False,
        )
        return encoded.encode("utf-8")

    def digest(self) -> str:
        return sha256(self.canonical_bytes()).hexdigest()


@dataclass(frozen=True)
class ApprovalEnvelope:
    approval_id: str
    scope_version: int
    approved_scope: PaymentScope
    approved_digest: str


def authorize_execution(
    envelope: ApprovalEnvelope,
    requested_value: dict[str, Any],
) -> tuple[bool, str]:
    if envelope.scope_version != 1:
        return False, "unsupported_scope_version"
    try:
        requested_scope = PaymentScope.parse(requested_value)
    except InvalidScope:
        return False, "invalid_requested_scope"
    if envelope.approved_scope.digest() != envelope.approved_digest:
        return False, "corrupt_approval_record"
    if requested_scope != envelope.approved_scope:
        return False, "scope_mismatch"
    return True, "approved"

This code stores both the typed value and its digest. The typed value supports field-level diagnostics. The digest gives services a compact identifier and detects accidental storage corruption. A digest is not a secret and does not prove who approved the data. Protect the approval record with your normal authenticated service and storage controls. If the scope travels through untrusted clients, use an authenticated construction designed by your security team rather than treating a plain hash as a signature.

The version field is not decoration. When a new option changes the meaning of a tool, introduce a new scope version and teach the executor how to validate it. Do not quietly add the option outside the old digest. Old approvals should retain old semantics or be invalidated deliberately.

Normalization must happen before the person reviews the action. Suppose the interface shows reports/weekly.csv, but the executor later resolves it relative to a working directory. The real scope is the resolved resource identifier, and that is what the approval view must explain. Likewise, do not let one service convert a decimal amount after approval. Parse and validate at the proposal boundary, then carry the typed value forward.

Sets require a documented choice. Recipient order may not matter, so the proposal service can deduplicate and sort recipient identifiers before display. A sequence of migration steps does matter, so it must keep order. Normalizing everything as a sorted list can erase a real behavior change.

Mutate one dimension at a time

A mutation suite should make the failure easy to name. Begin with one valid approved object. Change exactly one field per case, pass the changed request to the executor, and assert both denial and reason. Include extra and missing fields, not only changed values. Those cases catch schema drift where a new flag appears without entering the approval contract.

from copy import deepcopy

import pytest


BASE_PAYMENT = {
    "tenant_id": "tenant-a",
    "operation": "pay_invoice",
    "invoice_id": "inv-104",
    "destination_account_id": "acct-supplier-7",
    "amount_minor": 24_000,
    "currency": "USD",
}


def approved_envelope() -> ApprovalEnvelope:
    scope = PaymentScope.parse(BASE_PAYMENT)
    return ApprovalEnvelope(
        approval_id="approval-81",
        scope_version=1,
        approved_scope=scope,
        approved_digest=scope.digest(),
    )


@pytest.mark.parametrize(
    ("field", "new_value"),
    [
        ("tenant_id", "tenant-b"),
        ("operation", "schedule_payment"),
        ("invoice_id", "inv-999"),
        ("destination_account_id", "acct-personal-2"),
        ("amount_minor", 240_000),
        ("currency", "EUR"),
    ],
    ids=["tenant", "operation", "invoice", "destination", "amount", "currency"],
)
def test_each_authorized_field_is_immutable(
    field: str,
    new_value: object,
) -> None:
    requested = deepcopy(BASE_PAYMENT)
    requested[field] = new_value

    allowed, reason = authorize_execution(approved_envelope(), requested)

    assert allowed is False
    assert reason in {"scope_mismatch", "invalid_requested_scope"}


def test_unreviewed_option_fails_closed() -> None:
    requested = deepcopy(BASE_PAYMENT)
    requested["expedite"] = True

    allowed, reason = authorize_execution(approved_envelope(), requested)

    assert (allowed, reason) == (False, "invalid_requested_scope")


def test_unchanged_scope_is_accepted() -> None:
    allowed, reason = authorize_execution(approved_envelope(), BASE_PAYMENT)

    assert (allowed, reason) == (True, "approved")

The broad reason assertion in the parameterized test reflects two valid denial layers. A changed operation fails domain parsing, while a valid but different amount produces scope_mismatch. In a production test suite, put the expected reason in each parameter row if reason stability is part of the service contract.

Worked example one is a payment destination swap. Keep amount, invoice, tenant, and currency fixed. Change only destination_account_id. The expected evidence is an approved digest for the supplier account, a requested digest for the new account, and a field diff naming the destination. If the test fails because the request is unauthenticated, it has not exercised scope binding. Use a valid test identity and make the mutation the first rejected condition.

Worked example two is path expansion. A reviewer approves deletion of /exports/run-81/output.tmp; the executor receives /exports/run-81. A text-prefix policy is unsafe because both paths share a prefix. Parse the path into the resource identifier used by the storage service, and include the recursive option. Tests should cover .. segments, symlink behavior at the storage boundary, encoded separators, case rules for the target filesystem, and a switch from one object to a container. Avoid claiming a string normalizer can model every filesystem. The owning service should resolve the resource and return the identifier that enters approval.

Worked example three is an outbound message. The human reviews one recipient, no attachments, and draft mode. After approval, add a hidden recipient, attach a file, or switch the operation to send. Each mutation has a different consequence and deserves its own case id. Recipient order can be normalized if it carries no meaning, but membership cannot. An attachment's display name is not enough; bind to the immutable attachment object id and, when content can change under that id, a content version.

Also test removal. Teams often focus on escalation, but deleting a qualifier can broaden behavior. Changing delete only generated files to delete files may appear as fewer arguments if the qualifier was represented by a filter. A missing restriction must fail, not fall back to a permissive default.

Use audit evidence to separate look-alike failures

The fastest diagnosis compares typed scope values at named boundaries. Preserve proposed_scope, view_scope, approved_scope, and requested_scope as redacted structured records or stable digests. Give every record a schema version and approval id. Record which trusted service produced it.

If proposed and approved scopes differ, inspect the approval presentation path. The UI may have applied a default, dropped a field, or rendered stale state. If approved and requested scopes differ, inspect checkpoint restoration, agent replanning, tool argument assembly, and client tampering. If all scopes match but the tool performs something else, the adapter or downstream service violated its contract.

The following diagnostic compares two saved scope objects and prints field paths without needing an agent framework. Save sanitized fixtures, not live payment data.

from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any


def differences(left: Any, right: Any, path: str = "$") -> list[str]:
    if type(left) is not type(right):
        return [f"{path}: type {type(left).__name__} -> {type(right).__name__}"]
    if isinstance(left, dict):
        lines: list[str] = []
        for key in sorted(set(left) | set(right)):
            child = f"{path}.{key}"
            if key not in left:
                lines.append(f"{child}: added")
            elif key not in right:
                lines.append(f"{child}: removed")
            else:
                lines.extend(differences(left[key], right[key], child))
        return lines
    if isinstance(left, list):
        if left == right:
            return []
        return [f"{path}: sequence changed"]
    if left != right:
        return [f"{path}: {left!r} -> {right!r}"]
    return []


if __name__ == "__main__":
    approved = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    requested = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))
    changes = differences(approved, requested)
    if changes:
        print("\n".join(changes))
        raise SystemExit(1)
    print("Scopes are identical")

python tools/diff_approval_scope.py \
  artifacts/approved-scope.json \
  artifacts/requested-scope.json
python -m pytest tests/agent/test_approval_scope.py -vv

A useful failure prints $.destination_account_id, followed by the old and new redacted identifiers. It should not dump a bearer token, full account number, message body, or attachment contents. Configure the scope serializer to emit safe identifiers and category-level summaries.

Several failures look like scope mutation in a screenshot but have different causes.

A stale interface can show the old proposal while the backend records a newer proposal before the click. Compare the view-model version submitted with the click to the server's current version. The fix is optimistic concurrency at approval, not a more tolerant digest.

A display formatter can round $240.49 to $240. The stored and executed minor-unit values match, so the mutation test passes, but the human saw misleading data. Add presentation contract tests that render boundary values and assert the view model contains the same precision as the typed scope.

An adapter may ignore a validated field. For example, authorization verifies draft, but the mail adapter always sends. Approved and requested scopes match. An adapter contract test with a fake transport should prove each operation maps to the expected downstream call.

Finally, a server may add a trace id or retry counter after approval. A raw-object digest changes even though authorization meaning does not. Move that metadata outside the typed scope. Do not add an ignore-on-mismatch fallback, because it will eventually hide a dangerous new field too.

Distinguish a real mutation from an audit-event misjoin

One more failure can produce an almost perfect scope-mutation trace even when the executor compared the right objects. An observability pipeline may join an approval event from one attempt to an execution event from another. This happens when a run is retried, an approval identifier is only unique inside one workflow, or a dashboard groups events by a mutable pending-action identifier. The screen then places one approved digest beside a different request digest and appears to prove mutation. Tightening the executor comparison will not repair a false join.

Read the executor's own decision event before reading the dashboard's assembled timeline. The useful fields are the approval id, pending-action id, attempt id, scope version, approved digest, requested digest, decision reason, and command id. In a healthy fixture, the approval and execution events name the same immutable approval record and attempt, both digests are equal, and the command id exists only after an allowed decision. In a real mutation fixture, the record and attempt still match, the digests differ, the reason is scope_mismatch, and no adapter command is created. In an audit misjoin, the two displayed digests differ, but the approval event and executor event name different attempts or different approval records. The executor's local event may show two equal digests while the dashboard has borrowed the approved digest from an earlier row.

The most misleading value is a shared run id. It proves that events belong to the same broad workflow, not that they belong to the same approval attempt. A green approval status is equally weak because it says some proposal was approved. Neither value connects that decision to the command under investigation. Follow the immutable approval record through the executor, then use the command id to locate the adapter call. If the adapter reports a command that has no preceding allowed decision, that is a separate enforcement defect even when the digest display looks correct.

Add this distinction to an existing suite before making digest mismatch a release gate. Older fixture builders usually fabricate a boolean approval and omit the record version, view-model version, or attempt identity. Those helpers will break first when the production-shaped envelope becomes mandatory. Land the typed fixture builder and executor evidence assertions first. Next, update service fakes to return a command only after authorization. Then add a retry fixture with two proposals under one workflow and prove that the audit query does not cross their events. Only after those cases pass should a dashboard-level mismatch fail CI.

During deployment, compare the executor's local verdict with the joined audit verdict. Disagreement is an observability incident, not permission to allow the action. Keep denial authoritative for high-impact tools while the audit owner repairs joins. For low-impact sandbox cases, the parallel comparison reveals old events that lack attempt identity and tells the team which producers must be upgraded before historical dashboards can be trusted.

Ownership splits at the boundary. The approval platform owns creation and immutability of the approval record. Each tool team owns the list of effect-bearing fields and their normalization. The interface team owns faithful display of that typed scope. The executor team owns the final comparison and the rule that denial creates no command. The observability team owns event identity and joins. A handoff should contain the sanitized proposed, viewed, approved, and requested scopes, their versions and digests, the exact executor reason, all attempt and event identifiers, the adapter command id if one exists, and a statement of whether an external effect occurred. Without that last statement, a team can close the comparison bug while leaving remediation of a completed action unassigned.

This evidence has a maintenance cost. Keeping both the typed scope and the view model increases retained audit data and redaction work. Attempt-level identifiers increase trace cardinality, and strict fixture builders require updates whenever a tool gains an effect-bearing field. Audit indexes and queries must join on compound, immutable identity instead of a convenient run id. Historical events that never recorded attempt identity may remain impossible to join confidently, so dashboards need to display that evidence gap rather than manufacture continuity.

Scope binding does not prove that the approver was entitled, attentive, or uncompromised. If a stolen operator session approves the exact harmful payment later executed, every digest can match. Authentication, separation of duties, transaction policy, and anomaly review must catch that case. This technique proves sameness between review and execution, nothing more.

Roll out strict binding without freezing every workflow

Inventory tools before enforcing one generic rule. For each tool, list fields shown to the human, fields consumed by the adapter, defaults added by services, and values that can change while paused. Any execution field absent from the approval model needs an owner to classify it as scoped or operational metadata.

Introduce typed scope builders at the proposal boundary. Store the versioned scope beside the pending action and use that stored object to build the approval view. Do not regenerate the card from a later agent message. On approval, record the view-model version and scope digest. At execution, build the requested typed scope from the actual adapter request and compare it with the record.

Shadow comparison is useful for read-only and sandbox traffic. Calculate both digests, log mismatches, and allow the existing behavior while owners remove volatile metadata and fix normalization. Do not use shadow mode to permit known payment, deletion, privilege, or external-message mutations in production. For those tools, validate with recorded fixtures and a sandbox before switching enforcement.

Add tests in layers. Scope parser tests cover types, unknown fields, missing fields, and normalization. Mutation tests cover each authorization dimension. Service tests prove a mismatch stops command creation. Adapter tests prove an accepted typed operation maps to the intended effect. A small browser test can verify the card receives the correct server view model, but it should not be the only oracle.

The CI job below separates the fast contract suite and retains only sanitized mismatch artifacts on failure.

name: approval-scope-contract

on:
  pull_request:
    paths:
      - "approval_scope/**"
      - "tests/agent/test_approval_scope.py"

jobs:
  mutation-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
          cache-dependency-path: requirements-test.txt
      - run: python -m pip install -r requirements-test.txt
      - run: >-
          python -m pytest
          tests/agent/test_approval_scope.py
          -vv
          --junitxml=artifacts/approval-scope.xml
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: approval-scope-evidence
          path: artifacts/
          retention-days: 7

That cache-dependency-path line has to name requirements-test.txt, because the pip cache resolves **/requirements.txt and **/pyproject.toml by default and aborts the setup step when neither pattern matches the checkout.

Plan schema changes like API changes. Adding a scoped field can invalidate pending approvals. Choose whether to let version-one actions finish under their original adapter semantics or cancel them and request review again. Never reinterpret an old approval with a broader version-two schema. The safer default for high-impact actions is to expire pending approvals and explain why a new review is required.

Strict binding adds prompts. An address corrected by one character, a regenerated attachment, or a price updated by the source system may require another click. Measure prompt frequency using actual product telemetry, not illustrative claims. Where interruptions are excessive, consider a bounded approval with explicit constraints, but do not quietly weaken exact matching.

It also adds versioning work. Every field needs a stable type, normalization rule, safe display, and migration plan. That cost buys an auditable answer to a critical incident question: what did the person authorize? Without the typed record, teams end up comparing screenshots to logs and guessing which representation was authoritative.

Choose bounded approval when exact matching is wrong

Exact equality is the wrong model for a deliberately delegated policy. A support lead may approve refunds up to $100 for one order during a shift. A repository owner may approve read-only searches under one project. Model these as constraints with a subject, tenant, resource boundary, operations, limit, expiry, and revocation state. Test values just inside and just outside every boundary.

Do not use bounded approval when the interface showed one concrete action. A card saying "Pay $240" cannot secretly authorize any payment below $500 because the backend policy would have allowed it. The words and controls presented to the reviewer must describe the actual grant.

Leave operational metadata outside scope only if changing it cannot alter authorization or effect. Trace ids, queue attempt counts, and worker names usually qualify. Downstream endpoint, environment, timeout behavior that changes partial execution, and retry policy for non-idempotent tools may not. Make the classification per tool instead of relying on a universal ignore list.

Avoid hashing prose as the primary contract. Human summaries can change punctuation, localization, or display order without changing the action, and they can omit a dangerous field. Hash the typed scope. Store a versioned view model separately and test that it faithfully exposes the scope.

Do not ask a model to decide whether a mismatch is safe at runtime. A grader can cluster failures after the deterministic control has denied them. It can help reviewers understand that two account labels look similar or that an extra recipient is suspicious. It must not override typed scope enforcement.

Finally, do not test scope mutation by editing production approvals or invoking real destructive tools. Construct valid test approvals in an isolated environment, inject one mutation, and stop at a fake adapter. Keep a few sandbox end-to-end cases for wiring. The evidence you need is the denied command and its field-level reason, not a damaged account that proves the guard was absent.