A green browser agent run is not release evidence

The agent says it placed the order, the checkout page flashes “Thanks,” and CI turns green. An hour later, the test tenant contains no order. The run left a screenshot, but it cannot answer whether the click happened, whether the server rejected it, or whether the agent simply declared success too early.

Decide what the evidence must prove

A browser agent creates several kinds of records during one task. They are related, but they are not interchangeable. The final answer is the agent's account of what happened. A tool-call log records what the runtime asked a browser tool to do and what that tool returned. A Playwright trace records Playwright activity in the browser context. An application oracle reports the state owned by the system under test. Release confidence comes from joining those records, not picking the most convincing screenshot.

Start by making the claim narrow. “The agent completed checkout” hides too much. A useful claim might be: the approved test user submitted one order for the seeded cart, the application stored that order as committed, and no second order was created. That sentence identifies the actor, action, quantity, expected state, and duplicate rule. Each part can fail independently.

Use that narrow claim as the unit of review for browser agent release evidence testing. It gives every artifact a limited job and prevents a fluent final answer from carrying more weight than the underlying records.

The mission itself belongs in the bundle. Record a stable mission identifier and a version, not only the natural-language prompt. A small copy edit can change the path an agent chooses. Tool schemas, allowlists, system instructions, agent code, and browser project matter for the same reason. Save identifiers or hashes for those inputs. Do not save private hidden reasoning. Release review needs observable inputs, tool requests, tool results, and claims, not a model's concealed chain of thought.

The tool-call record answers whether the agent requested an action. It should carry a call identifier, ordered position, exact tool name, sanitized arguments, result category, and a link to the attempt. That record is owned by the agent runtime or tool gateway. Playwright does not automatically manufacture it just because the selected tool eventually uses Playwright.

Browser evidence answers a different set of questions. The official Trace Viewer documentation says the Actions view shows the locator used and the duration of each Playwright action. Its before, action, and after snapshots help a reviewer see which DOM node received input. The Network view lists browser requests and lets a reviewer inspect status, headers, and bodies. Console output and test errors have their own views. Those facts are excellent for diagnosis, but a trace remains a record of browser-side execution.

A trace showing a POST response does not automatically prove that a downstream workflow finished. A 202 response may mean the application accepted work for later processing. A 200 response can still precede a later rollback or compensating action. The correct interpretation depends on the product's documented contract. Do not turn an HTTP status into a stronger business promise than the application team made.

Visible assertions have a similarly limited job. Playwright's auto-retrying assertions repeatedly check a locator until the condition passes or its assertion timeout expires. They are appropriate for waiting until a stable user-visible state appears. They do not make the text truthful. A page can render “Order confirmed” from stale client state, an optimistic update, or the wrong account.

The durable oracle should be owned outside the agent's narrative. For an order, it could be a test-support API that returns a sanitized projection from the order service. For a published message, it could be the resulting test-tenant record. For a settings change, it could be a fresh API read under the same user. Playwright's API testing support allows the request fixture to make these checks in the same test, but the endpoint and its semantics belong to your application.

A good gate has three possible engineering outcomes. Pass means all required records agree with the narrow claim. Fail means evidence shows a contract violation, such as a denied action executing or two orders existing. Incomplete means a required artifact or oracle is unavailable. Incomplete still blocks a required release case. Relabeling it as a product failure sends work to the wrong team, while relabeling it as a pass rewards missing proof.

Write the ownership beside the evidence source. The agent team owns tool-call capture. The browser harness owns trace configuration and web assertions. The product team owns the business oracle contract. The CI platform owns retention and access. When the bundle breaks, this split tells the reviewer who can fix the first missing link.

Build one bundle around one attempt

Parallel workers and retries make casual filenames dangerous. “checkout-trace.zip” says nothing about which account, browser, test, or attempt produced it. Use one run identifier for the mission and one attempt number for each execution. Playwright exposes testInfo.retry, where the first run is zero and the first retry is one. Put that value into the manifest even if the release project currently disables retries.

Keep the business identity separate from the attempt identity. Suppose the mission owns order key release-4821. Attempt zero and attempt one are execution records. The order key is the application's deduplication identity, if the application supports such a key. Generating a new order key on every retry could hide a duplicate side effect. Reusing a key against an application that has no idempotency contract could produce a different failure. Document the choice rather than guessing.

A compact manifest should include the mission version, run ID, attempt number, browser project, agent build, tool schema version, ordered tool calls, final claim, visible assertion, business oracle, artifact paths, and verdict. Every artifact path must resolve before the gate can pass. Every tool-call identifier must be unique inside the attempt. The oracle should compare a deliberately small expected projection with a deliberately small observed projection.

Do not dump a whole customer record into the manifest because it is convenient. Select only fields needed by the assertion, such as test run ID, resource ID, state, and count. Passwords, cookies, authorization headers, addresses, payment details, and raw model prompts do not become safe merely because a CI system stores them. Trace capture deserves separate access control because its Network view may contain request and response content.

The following validator is a complete TypeScript command-line program. It checks structure, duplicate call IDs, required artifact types, file existence, and consistency between a passing verdict and the business oracle. A malformed or incomplete bundle exits nonzero. A real change to the manifest can therefore make the oracle fail.

Its awaits live inside an async main() function rather than at the top level, and that is a portability fix rather than a style preference. A plain .ts file in a package without "type": "module" is treated as CommonJS by the usual runners, and top-level await fails there with a build error before the program reads a single byte of the manifest. That is a realistic shape for the project described in this article, which is a pnpm and Playwright repository rather than an ESM-first one. Wrapping the body keeps tsx validate-evidence.ts <manifest.json> working under either module setting, and the trailing catch turns a usage mistake or an unreadable file into a nonzero exit instead of an unhandled rejection. Renaming the file to .mts is the other valid fix, but it forces every caller and CI step to learn the new extension.

import { access, readFile } from "node:fs/promises";
import { isDeepStrictEqual } from "node:util";
import process from "node:process";

type ToolCall = {
  id: string;
  name: string;
  outcome: "succeeded" | "failed" | "denied";
};

type Artifact = {
  kind: "trace" | "agent-log" | "oracle";
  path: string;
};

type EvidenceManifest = {
  schemaVersion: 1;
  runId: string;
  attempt: number;
  browserProject: string;
  agentBuild: string;
  missionVersion: string;
  toolCalls: ToolCall[];
  businessOracle: {
    expected: unknown;
    observed: unknown;
    passed: boolean;
  };
  artifacts: Artifact[];
  verdict: "passed" | "failed" | "incomplete";
};

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === "object" && value !== null && !Array.isArray(value);
}

function parseManifest(value: unknown): EvidenceManifest {
  if (!isRecord(value)) throw new Error("manifest must be an object");
  if (value.schemaVersion !== 1) throw new Error("schemaVersion must be 1");
  if (typeof value.runId !== "string" || value.runId.length === 0)
    throw new Error("runId is required");
  if (
    typeof value.attempt !== "number" ||
    !Number.isInteger(value.attempt) ||
    value.attempt < 0
  )
    throw new Error("attempt must be a non-negative integer");
  if (typeof value.browserProject !== "string" || value.browserProject.length === 0)
    throw new Error("browserProject is required");
  if (typeof value.agentBuild !== "string" || value.agentBuild.length === 0)
    throw new Error("agentBuild is required");
  if (typeof value.missionVersion !== "string" || value.missionVersion.length === 0)
    throw new Error("missionVersion is required");
  if (!Array.isArray(value.toolCalls)) throw new Error("toolCalls must be an array");
  for (const call of value.toolCalls) {
    if (!isRecord(call)) throw new Error("each tool call must be an object");
    if (typeof call.id !== "string" || call.id.length === 0)
      throw new Error("each tool call needs an id");
    if (typeof call.name !== "string" || call.name.length === 0)
      throw new Error("each tool call needs a name");
    if (!["succeeded", "failed", "denied"].includes(String(call.outcome)))
      throw new Error("each tool call needs a valid outcome");
  }
  if (!Array.isArray(value.artifacts)) throw new Error("artifacts must be an array");
  for (const artifact of value.artifacts) {
    if (!isRecord(artifact)) throw new Error("each artifact must be an object");
    if (!["trace", "agent-log", "oracle"].includes(String(artifact.kind)))
      throw new Error("each artifact needs a valid kind");
    if (typeof artifact.path !== "string" || artifact.path.length === 0)
      throw new Error("each artifact needs a path");
  }
  if (!isRecord(value.businessOracle))
    throw new Error("businessOracle must be an object");
  if (!("expected" in value.businessOracle) || !("observed" in value.businessOracle))
    throw new Error("businessOracle needs expected and observed values");
  if (typeof value.businessOracle.passed !== "boolean")
    throw new Error("businessOracle.passed must be boolean");
  if (!["passed", "failed", "incomplete"].includes(String(value.verdict)))
    throw new Error("verdict is invalid");
  return value as EvidenceManifest;
}

async function main(): Promise<void> {
  const filename = process.argv[2];
  if (!filename) {
    throw new Error("usage: tsx validate-evidence.ts <manifest.json>");
  }

  const manifest = parseManifest(JSON.parse(await readFile(filename, "utf8")));
  const problems: string[] = [];

  const callIds = manifest.toolCalls.map((call) => call.id);
  if (new Set(callIds).size !== callIds.length)
    problems.push("duplicate tool-call id");

  for (const kind of ["trace", "agent-log", "oracle"] as const) {
    if (!manifest.artifacts.some((artifact) => artifact.kind === kind))
      problems.push("missing " + kind + " artifact");
  }

  for (const artifact of manifest.artifacts) {
    try {
      await access(artifact.path);
    } catch {
      problems.push("artifact does not exist: " + artifact.path);
    }
  }

  const oracleMatches = isDeepStrictEqual(
    manifest.businessOracle.expected,
    manifest.businessOracle.observed,
  );
  if (manifest.businessOracle.passed !== oracleMatches)
    problems.push("businessOracle.passed disagrees with expected and observed");
  if (manifest.verdict !== "passed")
    problems.push("manifest verdict is " + manifest.verdict);
  if (manifest.verdict === "passed" && !oracleMatches)
    problems.push("passing verdict has a failed business oracle");
  if (manifest.verdict === "passed" && manifest.toolCalls.length === 0)
    problems.push("passing verdict has no recorded tool calls");

  if (problems.length > 0) {
    for (const problem of problems) process.stderr.write(problem + "\n");
    process.exitCode = 1;
  } else {
    process.stdout.write("evidence bundle is internally consistent\n");
  }
}

main().catch((error: unknown) => {
  const message = error instanceof Error ? error.message : String(error);
  process.stderr.write(message + "\n");
  process.exitCode = 1;
});

Internal consistency is not proof that the product behaved correctly. The validator cannot know whether “committed” is the right expected order state. It can prove that the gate did not silently mark a mismatched projection as passed and that required files exist. Product-specific tests still establish the meaning of the projection.

Attach the sanitized oracle result to the Playwright test so the HTML report and trace-adjacent artifacts use the same attempt. The official TestInfo attachment API accepts either a body or a file path, and reporters can present those attachments. Await the call so Playwright finishes copying the attachment before temporary data is removed.

This helper expects a test-only endpoint that returns a JSON array. The endpoint is not a Playwright feature. Your product team must define it, authenticate it, and guarantee what “committed” means. The helper preserves only three safe fields before attaching them.

import {
  expect,
  type APIRequestContext,
  type TestInfo,
} from "@playwright/test";

type OrderProjection = {
  id: string;
  runId: string;
  status: string;
};

function toOrderProjection(value: unknown): OrderProjection[] {
  if (!Array.isArray(value)) return [];
  return value.flatMap((item) => {
    if (typeof item !== "object" || item === null) return [];
    const record = item as Record<string, unknown>;
    if (
      typeof record.id !== "string" ||
      typeof record.runId !== "string" ||
      typeof record.status !== "string"
    ) return [];
    return [{
      id: record.id,
      runId: record.runId,
      status: record.status,
    }];
  });
}

export async function expectOneCommittedOrder(
  request: APIRequestContext,
  testInfo: TestInfo,
  runId: string,
): Promise<OrderProjection> {
  const response = await request.get(
    "/test-support/orders?runId=" + encodeURIComponent(runId),
  );
  const responseText = await response.text();
  let raw: unknown = null;
  let parseError: string | null = null;
  try {
    raw = JSON.parse(responseText);
  } catch {
    parseError = "response was not valid JSON";
  }
  const orders = toOrderProjection(raw);

  await testInfo.attach("order-oracle", {
    body: JSON.stringify({
      httpStatus: response.status(),
      contentType: response.headers()["content-type"] ?? null,
      runId,
      parseError,
      orders,
    }, null, 2),
    contentType: "application/json",
  });

  expect(response, "test-support order query").toBeOK();
  expect(parseError, "test-support order response format").toBeNull();
  expect(orders, "exactly one order must belong to this run").toHaveLength(1);
  expect(orders[0]).toMatchObject({ runId, status: "committed" });
  return orders[0];
}

The exact comparison in the manifest validator is intentionally strict. That makes drift visible, but it also creates maintenance when the projection changes. Version the manifest schema and migrate producers and consumers together. Do not loosen the validator to “contains some truthy fields” just to avoid a coordinated change.

Work through three green-looking failures

False success is easier to understand when the evidence sources disagree. Three common failures can produce a polished agent answer and a plausible screenshot. The first is optimistic UI, the second is a repeated side effect, and the third is correct behavior in the wrong account. Their fixes are different, so a generic “agent failed checkout” label is not useful.

An optimistic banner appears before the application knows the final result. The agent clicks Submit, sees “Order submitted,” and ends its task. A screenshot taken at that moment looks persuasive. The browser request later returns an error and the page changes to “Order failed.” If the harness grades only the first visible text, it approves a result the application explicitly rejected.

This self-contained Playwright test demonstrates that timing shape. Read it as an executable illustration, not as a test of anything. The application is an HTML string the test fulfills through page.route, and the assertions are the test's own final expect lines, so the page and the check are written in the same function. No change to a product could make it fail, and there is no separate oracle here to change. Its value is that a reviewer can run it and watch the status text pass through “Order submitted” before settling on “Order failed,” which is the sequence a screenshot taken at the wrong moment would have frozen into evidence.

import { test, expect } from "@playwright/test";

test("optimistic text is not a completed order", async ({ page }) => {
  let releaseResponse: () => void = () => {};
  const responseGate = new Promise<void>((resolve) => {
    releaseResponse = () => resolve();
  });

  await page.route("https://shop.example.test/**", async (route) => {
    const request = route.request();
    const pathname = new URL(request.url()).pathname;

    if (pathname === "/checkout") {
      await route.fulfill({
        status: 200,
        contentType: "text/html",
        body:
          "<button>Submit order</button>" +
          "<p role='status'>Ready</p>" +
          "<script>" +
          "document.querySelector('button').addEventListener('click', async () => {" +
          "const status = document.querySelector('[role=status]');" +
          "status.textContent = 'Order submitted';" +
          "const response = await fetch('/orders', { method: 'POST' });" +
          "status.textContent = response.ok ? 'Order confirmed' : 'Order failed';" +
          "});" +
          "</script>",
      });
      return;
    }

    if (pathname === "/orders" && request.method() === "POST") {
      await responseGate;
      await route.fulfill({
        status: 503,
        contentType: "application/json",
        body: JSON.stringify({ error: "test service unavailable" }),
      });
      return;
    }

    await route.abort();
  });

  await page.goto("https://shop.example.test/checkout");
  await page.getByRole("button", { name: "Submit order" }).click();
  await expect(page.getByRole("status")).toHaveText("Order submitted");

  releaseResponse();

  await expect(page.getByRole("status")).toHaveText("Order failed");
  await expect(page.getByRole("status")).not.toHaveText("Order confirmed");
});

In a real release case, the browser agent performs the interaction against the running product and the harness observes it. The final assertion should match the product's final user state, while a separate application query checks commitment. That query is the part the fixture above cannot supply. The reusable oracle in this article is expectOneCommittedOrder, and the release test is the one that calls it against a real test-support endpoint, where zero orders, two orders, or one order in a state other than committed each fail for a reason the product controls.

Keep the two artifacts in separate roles. The fixture is documentation you can execute, useful for explaining the failure mode to a reviewer or for exercising a harness that grades page text, and it runs even when the model and tool gateway are unavailable. The release test is the thing that decides a verdict. A suite that contains only the fixture has proved that the author understands optimistic UI, not that the application avoids it.

A repeated side effect often begins with an ambiguous timeout. Attempt zero submits the order. The server commits it, but the response never reaches the browser. The agent reports uncertainty or the test times out. A retry starts with a clean browser context and submits again. If the second attempt uses a new business identity, both requests can succeed and the final attempt looks perfectly green.

The evidence bundle must ask what attempt zero changed before attempt one acts. Query by the mission's business key, not merely by the retry's new run directory. Preserve the failed trace. Its Network view may show a request with no completed response, while the order oracle may already return one committed record. That combination points to an uncertain response after a possible commit. It does not justify another unguarded submission.

Idempotency is an application contract, not a testing incantation. If the order API accepts a documented idempotency key, generate it once for the mission and record it across attempts. Assert that both attempts still map to exactly one order. If no such contract exists, automatic retries are unsafe for this consequential case. Disable them and route the incomplete attempt to review or cleanup. The cost is slower recovery from infrastructure flakes and more manual triage. The benefit is avoiding a test harness that creates the duplicate it was meant to catch.

Do not “fix” the problem by deleting attempt zero's order before retrying. Cleanup would erase the evidence that the first action may have succeeded. It also changes the user-level semantics from at-most-once submission to test-managed compensation. Cleanup belongs after the verdict unless the scenario explicitly tests recovery.

The wrong-account failure has different evidence. A reused storage state authenticates the page as tenant B while the mission names tenant A. The agent finds a matching settings control, makes the requested change, and receives a success banner. Every browser action can be correct relative to the loaded page. The product may also persist the change correctly, but under the wrong actor.

Catch this before the consequential action. Add the expected synthetic tenant and user IDs to the fixture. Read an application-owned identity endpoint or an unambiguous account marker, then compare it with the mission. After the action, include actor and tenant in the sanitized durable projection. A visual check of a common avatar or first name is too weak when test accounts share labels.

This case is sometimes blamed on model reasoning because the final answer names the wrong tenant. The first broken link is usually setup identity. If the agent never received an account identifier, the mission contract is incomplete. If it received the right identifier but the browser context contained another login, the fixture is contaminated. If both were right and the agent navigated to a different tenant, the agent path is responsible. The joined bundle separates those cases without interpreting prose style.

Account isolation costs setup time. Creating a fresh user for every test may be too slow, while one global user creates ambiguous state. A practical compromise is one synthetic tenant per parallel worker and one unique business resource per test, provided the suite verifies the current identity at the start of every mission. Record the worker or shard identity as metadata, but never treat a worker number as proof of the logged-in account.

These examples also show why screenshots should remain supporting artifacts. The optimistic screenshot is captured too soon. The duplicate screenshot cannot show how many orders exist. The wrong-account screenshot may omit the identity entirely. Each image helps reconstruct a moment, but the durable oracle decides the limited business claim.

Diagnose the first broken link

Open the manifest before opening the trace. Confirm the mission version, run ID, attempt, browser project, agent build, account, and business key. A beautiful trace attached to the wrong retry is worse than no trace because it encourages a confident diagnosis of another execution. Check that the artifact filenames and the contents agree, not merely that the directory name looks right.

Read the ordered tool-call log next. Find the last confirmed observation before the disputed action. Then locate the exact call ID for that action and its returned result. “The model intended to click” is not enough. The runtime needs to show that it emitted the browser tool request. If the call is missing, browser debugging will not explain why it never happened.

Now open the Playwright trace. For a local artifact, the documented command is playwright show-trace with the path to the zip. The following shell snippet finds one trace under a test result directory and refuses to continue when none exists.

set -euo pipefail

TRACE_FILE="$(find test-results/agent-release -type f -name trace.zip -print -quit)"

if [ -z "$TRACE_FILE" ]; then
  printf '%s\n' "No trace.zip found under test-results/agent-release" >&2
  exit 1
fi

pnpm exec playwright show-trace "$TRACE_FILE"

Use the Actions view to locate the disputed click or fill. The Call panel identifies the locator and action details. Before, action, and after DOM snapshots show what changed around that operation. The action log explains Playwright's waiting and interaction work. This is the place to distinguish “the runtime asked for a click” from “Playwright completed the click.”

Move to Network only after tying the browser action to the tool call. Look for the request the product contract says should follow. Check method, URL path, status, and a safe correlation identifier. A missing request can mean the click never triggered the application handler, but it can also mean the product uses another transport. Confirm the application's actual architecture before declaring the request mandatory.

Four evidence shapes deserve different owners. When the tool call is absent and the trace has no browser action, inspect agent planning, tool availability, or policy denial. When the tool call exists but the trace action ends in an interaction error, inspect page state and the locator produced by the adapter. When the action completes and the relevant request receives a product error, assign the application or dependency response. When the request appears successful but the durable record is absent, investigate the endpoint contract, asynchronous completion, rollback, and the oracle itself.

The last shape is where teams overclaim. A network response can be successful for the request it represents while the user's broader goal remains unfinished. If completion is asynchronous, poll the documented status endpoint with a bounded assertion. Do not invent a sleep duration and assume the work must finish within it. If the product offers no query for final state, narrow the release claim to the accepted state that can actually be proved.

A nearby failure can look identical in the test summary. A navigation failure, expired test credential, unavailable oracle endpoint, and genuine agent refusal may all end with no order record. Their first evidence differs. Navigation trouble occurs before a mission action. Expired credentials usually leave an authentication response or identity mismatch. An unavailable oracle produces an incomplete bundle even if the browser flow succeeded. A refusal leaves an agent result and no action. Keep these categories separate in reporting.

Playwright's testInfo.status is available after the test finishes in afterEach hooks and fixtures. Its documented values include passed, failed, timedOut, skipped, and interrupted. Compare it with testInfo.expectedStatus when producing a harness summary. Do not parse human-readable console text to infer a status that the API already exposes. Record testInfo.retry beside it so “passed” cannot hide that the first attempt failed.

Web assertion failures are evidence about the asserted page condition, not a universal root cause. A timed-out text assertion tells you the expected state did not appear within its assertion window. It does not say whether the agent skipped the action, the page rejected it, the backend failed, or the locator targeted the wrong status element. Use the trace and oracle to locate the earlier divergence.

Preserve conflicts instead of flattening them. “Visible state confirmed, durable state absent” is a valuable outcome. “Agent claimed success, tool call denied” is another. A report generator that keeps only one final boolean destroys the most diagnostic part of an agent test. Store each evidence verdict and derive the release verdict from an explicit rule.

Artifact review can expose secrets. Trace Viewer can display request headers and bodies, so do not upload an unrestricted trace to a public location. Use synthetic data, remove secrets at their source where possible, restrict artifact readers, and test redaction with deliberately seeded canary values. A redaction check must scan the artifact that will actually be retained, not a hand-built sample that cannot contain the secret.

Roll the gate into CI without hiding failures

Begin with one consequential workflow and one browser project. Choose a case with a stable test tenant, a documented approval boundary, and a durable oracle the product team trusts. Running a large mission set before the manifest and ownership model settle creates hundreds of incomplete bundles without teaching the team how to judge one.

Keep trace policy explicit. Playwright's recording options support screenshots, video, and trace modes. The retain-on-failure trace mode records each run and keeps the trace for failed runs. That is useful when retries are disabled because it preserves the first failure. Recording every trace with on gives more data, but the official documentation notes the performance cost. Measure runtime and storage in your own suite before expanding retention.

This release config disables retries for the side-effecting project, retains failure artifacts, and gives its output a dedicated directory. The HTML reporter makes attachments visible to reviewers. A team can add other browser projects after the first case is reliable.

import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
  testDir: "./tests/agent-release",
  outputDir: "test-results/agent-release",
  retries: 0,
  reporter: [
    ["line"],
    ["html", { outputFolder: "playwright-report/agent-release", open: "never" }],
  ],
  use: {
    baseURL: process.env.AGENT_TEST_BASE_URL,
    trace: "retain-on-failure",
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
  projects: [
    {
      name: "agent-release-chromium",
      use: { ...devices["Desktop Chrome"] },
    },
  ],
});

Disabling retries is not a blanket recommendation for every browser test. Read-only navigation checks can often retry without causing product state. A purchase, deletion, publication, or invitation deserves stricter treatment until its idempotency and cleanup contracts are proven. Split these cases into a separate project rather than forcing one retry policy across the suite.

Wire CI so artifact publication runs even when the test command fails. The example below is a runnable GitHub Actions workflow for a pnpm project. It intentionally enables Corepack before invoking pnpm and does not ask setup-node to initialize a pnpm cache. The artifact step uses always() so a failed release case still uploads the trace and report.

name: Browser agent release evidence

on:
  pull_request:
  workflow_dispatch:

jobs:
  agent-release:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v6

      - uses: actions/setup-node@v7
        with:
          node-version: 22

      - name: Enable pnpm
        run: corepack enable

      - name: Install dependencies
        run: pnpm install --frozen-lockfile

      - name: Install Chromium
        run: pnpm exec playwright install --with-deps chromium

      - name: Run the release evidence project
        env:
          AGENT_TEST_BASE_URL: ${{ secrets.AGENT_TEST_BASE_URL }}
        run: pnpm exec playwright test --config=playwright.agent-release.config.ts

      - name: Publish evidence
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: agent-release-evidence-${{ github.run_attempt }}
          path: |
            test-results/agent-release
            playwright-report/agent-release
          if-no-files-found: error
          retention-days: 14

The retention value in that workflow is a policy example, not a measured optimum. Choose it from incident response needs, artifact sensitivity, storage limits, and organizational rules. A longer period gives reviewers more history and creates more exposure. A shorter period reduces exposure and may erase the only useful failure before an intermittent incident is investigated.

Rollout works better in stages. First, generate manifests in report-only mode and validate them against stored failed and passed runs. Seed intentional defects: remove a trace, duplicate a tool-call ID, change the observed order state, and point an artifact to a missing file. The validator must reject each mutation. This proves the gate can fail for changes that matter.

Next, run one read-only browser mission in CI. It exercises identity joins, agent logs, traces, and report attachments without risking duplicate product state. Once that path is stable, add one side-effecting test with a synthetic tenant and application oracle. Keep it nonblocking briefly while teams fix evidence plumbing, but display incomplete runs prominently. A silent shadow gate provides no migration pressure.

Turn blocking on only after failure ownership is clear. Agent policy failures, browser harness failures, application failures, oracle failures, and CI artifact failures need distinct routing. The release decision can still be one block, but the diagnostic category should tell the first owner where to look.

Expand the browser matrix from product support, not ambition. A second engine adds useful compatibility coverage and also multiplies agent calls, fixture setup, artifacts, and triage. Stabilize the business oracle once, then reuse the same claim across projects. Do not accept different durable outcomes merely because browser rendering differs.

Version the manifest before changing required fields. During migration, allow the validator to read the old version and the new version, but let only the new version satisfy the new gate. Remove the compatibility branch after historical artifacts no longer need active validation. An unversioned schema encourages producers to add fields while reporters silently ignore them.

The main costs are concrete. Traces and videos consume storage. Durable queries add application test-support work. Synthetic account creation increases setup time. Redaction reduces diagnostic detail and requires maintenance. Side-effect protection may remove automatic retries. Cross-browser runs spend more agent and infrastructure capacity. Those costs are justified for high-consequence workflows, not every click in the product.

Know when this gate is the wrong tool

Not every agent behavior needs an end-to-end browser release case. Tool schema validation, argument normalization, approval matching, and manifest parsing belong in fast contract or unit tests. Running a browser and a model to discover that a required JSON field is absent adds latency without adding a stronger oracle.

Exploratory missions also resist binary release grading. “Find anything confusing on this page” has no single expected path or complete answer set. Capture its observations for review, but do not pretend a durable business oracle can decide whether the exploration was insightful. Use a curated evaluation rubric or human review for the subjective part, and reserve browser assertions for factual claims such as the URL visited or an error encountered.

Avoid consequential tests against third-party production systems when you lack a dedicated tenant, permission, cleanup, or side-effect query. A trace of an email-send click does not make unsolicited email acceptable. A screenshot of a payment page does not authorize a real charge. Replace the dependency with an approved sandbox or stop the test before the external commitment.

Purely visual regressions need a visual comparison strategy. Agent tool logs add little when the question is whether a component shifted or a theme broke. Conversely, a visual snapshot cannot replace the business oracle for an order. Choose the evidence source that directly answers the defect risk.

Some applications expose no durable test-facing state. That is a product observability limitation, not permission to fabricate certainty. You can still prove that the agent selected the intended control, that the browser sent a request, and that the page reached a documented state. Phrase the claim at that boundary. Ask the product team for a safer oracle before making committed backend state a release requirement.

Do not retain traces when the test cannot avoid sensitive data and the artifact system cannot protect it. Run narrower checks, turn off body capture where the tooling and contract allow, or use a synthetic environment. Redacting an attachment does not redact the independent browser trace. Security review must cover every retained source.

A full gate is excessive for low-risk, easily reversible interactions such as opening a help panel. A small Playwright assertion may be enough. Save the joined agent, browser, and durable bundle for flows where false success, wrong-account action, duplicate execution, or unauthorized side effects would affect users.

Manual review remains useful when a new tool, approval type, or product workflow has no stable automated oracle. The reviewer should receive the same attempt identity and evidence sources, then record a reasoned decision. Human review should not repair missing artifacts by imagining what probably happened.

Finally, do not use this gate to compare prose elegance. Two agent runs can explain themselves differently while taking the same permitted actions and reaching the same proved result. Release criteria should tolerate harmless wording variation. They should remain strict about identity, control, tool execution, visible state, durable effect, and artifact completeness.