Keep a shared Playwright platform from becoming everyone’s problem

Friday’s release job is green, but its report contains only the Chromium smoke project. A CI cleanup changed the command to playwright test --project=chromium-release, and no check noticed that Firefox and WebKit had disappeared. The tests did exactly what the command requested, so this is a platform-governance failure, not a Playwright failure.

Decide which rules belong to the platform

A useful internal developer platform gives teams a maintained route to production. For browser testing, that route usually includes a Playwright version, a configuration baseline, browser installation, CI commands, artifact handling, and a place to find the report. Governance is the small set of rules that keeps that route honest. It is not a committee that approves selectors, and it is not a giant shared helper library that every test must import.

Playwright knows how to load configuration and run the tests selected by that configuration plus the command line. It does not know which browsers your company promised to cover, whether a flaky checkout test should block a release, or which team must investigate a missing shard. Those are organizational decisions. A platform owner has to turn them into code or CI checks if they are meant to survive staff changes and repository growth.

Start by separating four contracts that teams often mix together.

The selection contract says what is supposed to run. It covers test directories, matching rules, projects, command-line filters, and release lanes. A green Chromium project cannot satisfy a contract that requires Chromium, Firefox, and WebKit. Likewise, three configured projects do not prove that CI selected all three. The command that launches the run is part of the contract.

The execution contract says under what conditions tests run. It includes retries, workers, environment identity, setup projects, and any dependency relationship between projects. A retry policy changes the meaning of green. Worker count changes pressure on the application and test data. A setup dependency changes which tests must pass before dependent projects begin. These settings are not harmless performance knobs.

The evidence contract says what a reviewer receives after the run. A terminal exit code is too thin for a shared platform. At minimum, the result needs project names, test identities, statuses, errors, and enough retained artifacts to investigate the failure mode the policy cares about. The reporter documentation describes the built-in report formats, while the trace viewer guide explains what a recorded trace can show. Neither page decides how long your organization retains those files or who may read them.

The ownership contract says who changes the baseline and who responds when it fails. Platform engineers should own the runner integration and shared defaults. Product teams should own assertions about their product. CI administrators may own artifact permissions and retention. Security teams may set rules for secrets and sensitive test data. Putting all four responsibilities into a “QA team” label makes failures bounce between queues.

Consider the missing-browser incident from the opening. There are at least three ways to create the same visible symptom. Someone can remove a project from playwright.config.ts. Someone can keep the project but add --project=chromium-release to the CI command. Someone can change testMatch or testIgnore so that a project collects no relevant files. One check cannot catch all three because the changes happen at different layers.

The configuration below makes the first failure loud. It keeps the release project list stable, rejects duplicate or unexpected names, and requires ownership metadata. Project names are visible during execution and in reports. Project metadata is serialized into report data, which gives downstream tooling an owner field without inventing a Playwright option.

import { defineConfig, devices } from '@playwright/test';

const requiredProjectNames = [
  'chromium-release',
  'firefox-release',
  'webkit-release',
] as const;

const governedProjects = [
  {
    name: 'chromium-release',
    owner: 'quality-platform',
    use: { ...devices['Desktop Chrome'] },
  },
  {
    name: 'firefox-release',
    owner: 'quality-platform',
    use: { ...devices['Desktop Firefox'] },
  },
  {
    name: 'webkit-release',
    owner: 'quality-platform',
    use: { ...devices['Desktop Safari'] },
  },
] as const;

function assertProjectPolicy(
  projects: readonly { name: string; owner: string }[],
): void {
  const names = projects.map(project => project.name);
  const required = new Set<string>(requiredProjectNames);
  const duplicates = names.filter(
    (name, index) => names.indexOf(name) !== index,
  );
  const missing = requiredProjectNames.filter(name => !names.includes(name));
  const unexpected = names.filter(name => !required.has(name));
  const ownerless = projects
    .filter(project => project.owner.trim() === '')
    .map(project => project.name);

  if (duplicates.length > 0) {
    throw new Error(`Duplicate Playwright projects: ${duplicates.join(', ')}`);
  }
  if (missing.length > 0 || unexpected.length > 0) {
    throw new Error(
      `Release project policy mismatch. Missing: ${missing.join(', ') || 'none'}; ` +
      `unexpected: ${unexpected.join(', ') || 'none'}`,
    );
  }
  if (ownerless.length > 0) {
    throw new Error(`Projects without owners: ${ownerless.join(', ')}`);
  }
}

assertProjectPolicy(governedProjects);

export default defineConfig({
  testDir: './tests/e2e',
  outputDir: 'test-results',
  forbidOnly: Boolean(process.env.CI),
  failOnFlakyTests: Boolean(process.env.CI),
  retries: process.env.CI ? 1 : 0,
  reporter: process.env.CI
    ? 'blob'
    : [['html', { open: 'never' }]],
  use: {
    trace: 'on-first-retry',
  },
  projects: governedProjects.map(({ owner, ...project }) => ({
    ...project,
    metadata: { owner },
  })),
});

This oracle can fail. Delete firefox-release from governedProjects without an approved policy change, duplicate a name, or blank an owner, and configuration loading throws before a browser starts. Changing both the required list and the project definition is still possible, as it should be. That becomes an explicit policy change for code review instead of an accidental coverage reduction hidden in a refactor.

The validation is deliberately deterministic. Playwright may evaluate its configuration more than once, so configuration code should not call a remote service, generate a random project name, or write mutable state. If the source of truth lives in a central package, publish a versioned, deterministic policy function and pin that package through the repository lockfile. A network lookup during config loading turns a governance check into another availability dependency.

This configuration does not protect the CI command. Running it with --project=chromium-release still selects only that project. The appropriate control is a reviewed release workflow whose command runs the whole matrix, plus diagnostics that compare the planned projects with the report. Pretending the config can see an organization’s intent would create exactly the false confidence governance is meant to remove.

Make the release contract executable

Policy prose goes stale fastest where a setting has a pleasant default. A team writes “no focused tests in CI,” but nobody enables forbidOnly. Another team permits one retry for diagnosis, then treats a retry pass as equivalent to a first-attempt pass. Both policies sound clear in a handbook and disappear under deadline pressure.

Two Playwright settings address those specific cases. With forbidOnly enabled, focused tests declared with test.only() or test.describe.only() cause an error. With failOnFlakyTests enabled, a run fails when Playwright classifies a test as flaky. According to the retry documentation, a test is flaky when it fails initially and passes on a retry. Retrying and accepting flakiness are separate decisions.

That distinction matters in a release lane. A retry can provide a second attempt and a trace without laundering the first failure. The config above allows one CI retry, records a trace on the first retry, and still returns a failing outcome for a flaky classification. A development lane may reasonably use the same retry count without failing on flakiness, but it should not publish the same status label as the release gate.

Worked example two starts with an intermittent entitlement test. Its first attempt submits a role change and times out waiting for the new role. The retry starts in a fresh worker process and finds the role already present because the server completed the first request after the client timed out. The second attempt passes. Without the flaky failure policy, the job can be green even though the test exposed ambiguous product state, a weak oracle, or both.

A trace captured with on-first-retry shows the retry, not the original failed attempt. That can be useful when the retry fails again, but it may show only the already-updated role in this example. If the first failure is the evidence your team needs, use retain-on-failure instead. That mode records the initial run and keeps its trace when it fails. The cost is more trace recording during first attempts, even though successful-run traces are discarded.

Project ownership should also be executable. The following small test runs under every selected release project. It reads the actual resolved project metadata, rejects an empty owner, checks the naming rule, and attaches the resolved context to the test result. Removing the owner or renaming a project can make it fail. This is a policy probe, not an application test, so keep it in a clearly named governance folder.

import { test, expect } from '@playwright/test';

test('release project publishes its governed identity', async ({}, testInfo) => {
  const owner = testInfo.project.metadata.owner;

  if (typeof owner !== 'string' || owner.trim() === '') {
    throw new Error(
      `Project "${testInfo.project.name}" has no metadata.owner value`,
    );
  }

  expect(
    testInfo.project.name,
    'release project names must end with -release',
  ).toMatch(/-release$/);

  await testInfo.attach('project-governance.json', {
    body: Buffer.from(
      JSON.stringify(
        {
          project: testInfo.project.name,
          owner,
          retry: testInfo.retry,
        },
        null,
        2,
      ),
    ),
    contentType: 'application/json',
  });
});

The probe has two boundaries. First, a project excluded by the CLI never runs the probe, so its absence cannot prove itself. Second, attaching metadata does not prove product behavior. A checkout assertion must still verify checkout, and an accessibility assertion must still verify the expected accessibility condition. Platform tests should fail on platform contract changes, while product tests fail on product changes.

Avoid encoding every preference as a release rule. Locator style, fixture naming, and page-object structure usually belong in library guidance, linting, or team review. They are poor reasons to stop an unrelated release unless the organization can name a concrete risk. Good gates protect promised coverage, truthful status, reproducibility, or evidence availability. Style debates make a shared platform feel arbitrary.

Project dependencies deserve the same restraint. Playwright supports one project depending on another, and dependent projects wait for their dependencies to pass. Setup tests then appear in reports and can have traces. Use that feature when the setup is genuinely part of the tested run, such as creating an authenticated state through observable setup steps. Do not turn every infrastructure prerequisite into a setup project. A database outage or unavailable test environment may be better represented as an environment readiness failure before the suite consumes browser minutes.

Diagnose the missing evidence before changing tests

The first useful artifact is often the CI invocation, not the trace. Copy the exact command from the job log. Check for --project, file filters, --grep, --shard, --no-deps, and an alternate --config path. Then check whether the job set CI, because the example config deliberately changes retries, flaky handling, reporters, and focused-test enforcement when that variable is present.

Use collection mode to inspect selection without spending time on browser execution. Playwright’s --list option collects and lists tests. Run it with the normal configuration, then repeat the suspicious filter. The project prefix in the listing makes a narrowed project command visible before a release run.

set -euo pipefail

pnpm exec playwright test --list
pnpm exec playwright test --project=chromium-release --list

rg -n --glob '*.yml' --glob '*.yaml' --glob 'package.json' \
  'playwright test|--project|--shard|--config' .github package.json

trace_file="$(find test-results -type f -name trace.zip -print -quit 2>/dev/null || true)"
if [[ -n "$trace_file" ]]; then
  pnpm exec playwright show-trace "$trace_file"
else
  printf '%s\n' \
    'No trace.zip was found. Check the trace mode, retry status, and artifact upload.'
fi

Treat the two listings as different questions. The first shows everything collected by the config. The second shows what the explicit Chromium filter selects. If the first listing includes all three release projects and the CI report contains only Chromium, the workflow command is the leading suspect. If Firefox and WebKit appear as project names but collect fewer tests, inspect their testMatch and testIgnore rules. If a spec is absent from every project, inspect testDir, file naming, and the path passed on the command line.

The report answers a later question: what actually ran and how Playwright classified it. The HTML report can be filtered by passed, failed, flaky, and skipped statuses and shows the projects used for tests. A blob report contains run details and attachments so reports from shards can be merged. Neither format can describe a project that the process never selected. An omitted test produces no failure stack, screenshot, or trace because no test attempt existed.

That limitation is why “check the trace” is weak first advice for governance incidents. Trace Viewer is excellent for examining actions, DOM snapshots, network activity, console output, and assertion timing within a recorded test attempt. It cannot explain why another project was filtered out before execution. Start with selection evidence, then move to per-attempt evidence.

Trace absence has several meanings. With trace: 'on-first-retry', a test that fails with retries disabled has no first-retry trace. A test that passes on its initial attempt also has no retry trace. A CI upload step can lose a trace that existed on the worker. Finally, a filtered-out test never created one. Check the configured trace mode, the result’s retry classification, the worker filesystem before upload if available, and the uploaded artifact contents in that order.

The output directories are another common near-miss. outputDir stores per-test output such as screenshots and traces, with unique test output directories. The blob reporter writes its report to blob-report by default. Uploading only test-results does not upload the blob report, and uploading only blob-report may omit artifacts if the chosen reporter or workflow does not include them as expected. Follow the paths produced by the configuration actually used, rather than assuming every Playwright artifact lives under one directory.

Worked example three is illustrative and looks like a sharding defect. A merge job produces an HTML report, but one quarter of the expected tests is absent. The test files are balanced and every visible result is green. The team initially changes fullyParallel because the problem appeared after sharding.

The decisive evidence is the CI artifact set. Three blob archives arrived for a four-job matrix because one job was cancelled before its upload step. merge-reports read the three files available and generated a report from them. Changing parallelism cannot recreate a report that never arrived. The fix is to make uploads run for failed jobs where possible, verify expected artifact cardinality before merging, and keep the matrix job status visible beside the merged report.

A similar-looking imbalance has a different cause. Without fullyParallel: true, Playwright shards at file granularity. One large spec file can make one shard take much longer, while another gets fewer files or none. In that case, all planned shard artifacts can exist and the merged report can contain the full test set. The evidence points to distribution, not loss. Splitting oversized spec files may improve balance without changing isolation semantics. Turning on full parallelism offers test-level granularity, but only suites safe for independent parallel execution should pay that trade-off.

Browser launch failures form another near-miss. When a worker cannot find or start the browser binary, the log fails before application navigation and the report identifies a launch problem. That is not evidence that a browser project was omitted. Check the installed Playwright package version, the browser installation step, and the runner image. The CI guide documents installing browsers and operating-system dependencies with playwright install --with-deps. Do not “fix” a launch error by removing the affected project from the release command.

Application failures remain application failures even on a governed platform. If all required projects ran, the report is complete, and a trace shows the same product assertion failing against a valid environment, send it to the product owner named for that suite. Governance should shorten routing, not relabel a real regression as infrastructure.

Scale shards without losing the release record

Sharding adds a second unit of completeness. Before sharding, the platform asks whether every required project ran. After sharding, it must also ask whether every planned shard contributed a result. A merged report is useful presentation, but the merge job needs its own oracle for the expected inputs.

The workflow below uses four shards as an illustrative policy choice, not as a claim about the best shard count. Each shard installs the repository’s locked dependencies and matching Playwright browsers, runs the same unfiltered release command with a different shard index, and uploads its blob report even when tests fail. The merge job downloads those reports, demands exactly four zip files, and only then generates an HTML report.

name: Governed Playwright release

on:
  pull_request:

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    env:
      CI: "true"
    steps:
      - uses: actions/checkout@v6

      - uses: pnpm/action-setup@v4
        with:
          version: 10

      - uses: actions/setup-node@v6
        with:
          node-version: 22
          cache: pnpm

      - run: pnpm install --frozen-lockfile

      - run: pnpm exec playwright install --with-deps

      - name: Run shard
        run: >-
          pnpm exec playwright test
          --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

      - name: Upload blob report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 7

  merge-reports:
    if: ${{ !cancelled() }}
    needs: [e2e]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6

      - uses: pnpm/action-setup@v4
        with:
          version: 10

      - uses: actions/setup-node@v6
        with:
          node-version: 22
          cache: pnpm

      - run: pnpm install --frozen-lockfile

      - name: Download shard reports
        uses: actions/download-artifact@v5
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true

      - name: Verify every shard reported
        shell: bash
        run: |
          expected=4
          actual="$(find all-blob-reports -maxdepth 1 -type f -name '*.zip' |
            wc -l | tr -d ' ')"
          if [[ "$actual" -ne "$expected" ]]; then
            printf 'Expected %s blob reports, found %s\n' \
              "$expected" "$actual" >&2
            exit 1
          fi

      - name: Merge HTML report
        run: >-
          pnpm exec playwright merge-reports
          --reporter=html
          ./all-blob-reports

      - name: Upload HTML report
        uses: actions/upload-artifact@v4
        with:
          name: playwright-html-report
          path: playwright-report
          retention-days: 7

The setup order is intentional. The pnpm action puts pnpm on the path before actions/setup-node enables its pnpm cache. This avoids asking the Node setup action to call a package manager that is not available yet. The merge job installs dependencies because the Playwright CLI used for merge-reports comes from the repository dependency, but it does not install browser binaries because merging reports does not launch a browser.

The zip-count assertion can fail when an artifact is missing. It is valid for this workflow because each matrix job runs one Playwright command with the blob reporter, producing one blob archive, and the download step merges the artifact contents into one directory. If your workflow runs multiple Playwright commands per shard, runs multiple environments, or changes blob output naming, replace the simple count with a manifest keyed by shard and environment. Do not keep an oracle whose assumptions no longer match the producer.

Artifact retention is an explicit cost and security decision. Seven days in the example is a chosen value, not a measured recommendation. Set it from the real investigation window, storage budget, and data classification for your organization. Traces and screenshots can include page content. Custom attachments can include anything the test adds. Restrict artifact access and avoid attaching secrets instead of assuming a short retention period makes unsafe evidence harmless.

Shard count also needs evidence from your own runs. More shards add job startup, dependency installation, browser installation, artifact transfer, and merge work. They also increase simultaneous traffic to the test environment. Measure queue time, execution time, failure rate, and environment saturation from actual CI data before changing the matrix. A fast local run does not prove that eight concurrent CI jobs will respect rate limits or test-data isolation.

Workers multiply that pressure inside each shard. A four-shard matrix with each process using several workers can create much more concurrency than the matrix size suggests. Do not copy a worker count from another repository. Start from available CPU and environment capacity, then observe. The Playwright CI guidance specifically warns against configuring more workers than the agent can support because that can introduce timeouts and failures.

Report completeness and test completeness remain separate. Four zip files prove four reporter processes contributed. They do not prove the expected tests were collected. Keep a collection manifest or reviewable --list output for the release configuration, and compare project and test identities when coverage changes. An artifact count is a transport oracle, not a product-coverage oracle.

Roll the policy into an existing suite

A large repository already has a platform, even if nobody designed it. Package scripts, copied workflows, wiki commands, team-specific configs, and habitual reruns form the real operating model. Replacing only playwright.config.ts leaves those paths intact and creates two competing platforms.

Inventory every runner entry point before enforcing anything. Search package scripts, workflow files, reusable workflows, Docker entry points, and developer documentation for playwright test, --project, --config, and --shard. Record which command blocks a merge, which runs nightly, which targets production, and which is only a local convenience. The same flags can be correct in one lane and dangerous in another.

Next, capture a baseline from real reports. List the projects and tests currently collected. Record which tests are passed, flaky, failed, or skipped under the existing retry policy. Note which artifacts exist for each class. Do not manufacture percentages or choose a flaky-test budget before seeing the data. The purpose of the baseline is to expose the migration cost, not to make the old platform look healthy.

Introduce project identity before project enforcement. Give each release project a stable name and owner metadata, then publish reports without changing browser coverage or parallelism. Teams can update dashboards and ownership routing while execution stays familiar. Once every project has an owner, add the deterministic config validation. A missing project then fails at configuration load with a message that points to the policy, instead of surfacing as a mysterious drop in report volume.

Move CI to one canonical command next. Local scripts may still select a single browser for fast feedback, but name them accordingly, such as test:e2e:chromium. The release job should call the unfiltered command owned by the platform workflow. If a product team needs a narrower release lane, document the reduced coverage as a different contract instead of quietly passing --project.

Flakiness enforcement usually needs a staged rollout. First enable retries and preserve the classification in reports. Assign existing flaky tests to their product owners with the failing attempt and retry evidence. Then enable failOnFlakyTests for the release gate when the team is prepared for those classifications to block it. Leaving the setting permanently non-blocking teaches everyone that “flaky” is a cosmetic label. Enabling it without an ownership path can stop delivery for old debt that nobody has capacity to fix.

Temporary exceptions should be visible and expiring. A release manager may accept a known flaky test for a specific release, but the exception needs a named issue, owner, and removal condition in whatever governance system the organization uses. Do not hide the exception by increasing retries until the test happens to pass. More attempts can increase latency and side effects while making the report harder to interpret.

Add sharding only after the selected test set and failure semantics are stable. Otherwise, a coverage change, new retry policy, and new distribution model land together. When the first red build arrives, the team cannot tell which boundary changed. Start with a small matrix chosen from actual run time and environment capacity. Verify report cardinality and merge behavior before making the merged report the release record.

Keep a rollback path for each control. If artifact merging breaks, preserve individual blob reports and shard statuses while repairing the merge job. If new parallelism overloads the environment, reduce shards or workers without removing browser projects. If flaky enforcement reveals an unmanageable backlog, route a temporary, reviewed exception at the release-policy layer rather than weakening assertions across the suite.

Version upgrades deserve their own change. The Playwright package and installed browser binaries are coupled through the version in the repository. Let the lockfile select the package used by both test and merge jobs, and install browsers after dependencies. Test the new version with the governed project matrix before making it the default. A central platform package that upgrades every repository at once saves maintenance work, but it also expands the blast radius of a bad default.

Ownership becomes useful during migration when it routes a concrete failure. The platform team handles a config-policy exception, missing artifact, incompatible runner image, or report merge failure. The product team handles an assertion against its behavior. The environment owner handles capacity and availability. A failure can cross boundaries, but the first responder should not be chosen by who happens to recognize the stack trace.

The final migration check is adversarial. Ask a reviewer to remove one required project, leave a focused test, force a test to pass only on retry, cancel one shard before upload, and rename a release project. Each change should trip a different control with a useful message. If two failures produce the same generic red job, the policy may be enforceable but still expensive to operate.

Account for the cost, and know when to stop

Every shared default takes autonomy from somebody. The right question is whether the protected release claim is worth that cost. Three browser projects multiply executions for tests that run in every project. That may be justified for a cross-browser product promise, but it is wasteful for a backend-only contract test that never uses a browser. Put API-only work in a suitable project or suite instead of making it impersonate browser coverage.

Failing on flaky tests improves the honesty of a release status and creates immediate delivery friction. Teams with a large existing flaky backlog will see more red builds until they fix or explicitly quarantine the causes. That friction is the point of a gate, but it needs product ownership and capacity. A platform team should not declare victory because it made the dashboard red.

Trace policy trades evidence against runtime, storage, and privacy. on-first-retry reduces first-attempt recording but can miss the state that caused the initial failure. retain-on-failure captures that first failed attempt but records traces during initial runs so they are available to retain. Recording every trace gives broad evidence and a broad artifact bill. Pick modes per lane and failure risk instead of choosing one global value because it appears in a starter config.

Central project validation narrows experimentation. A developer who wants to try a new device or browser channel will hit the unexpected-project rule in the release config. Give experiments a separate config or a clearly non-release lane. Do not weaken the release project oracle to accommodate a temporary local experiment. Conversely, do not force every experiment through the platform review process when it cannot affect release evidence.

Full parallelism is not a governance badge. It can improve shard balance by distributing individual tests, but it also exposes suites that share accounts, files, mutable server state, or order assumptions. If those tests are not isolated, keep file-level sharding while you repair the data model. Faster distribution is not worth nondeterministic product state.

A small repository with one team, one browser target, and no release gate may not need a central package, ownership metadata, a merge job, and a policy probe. A short checked-in config, one CI command, and an uploaded HTML report can be enough. Governance should grow when coordination cost appears, not before.

A single-project local debugging command should not be blocked either. Selecting Chromium while developing one locator is efficient and honest when the command is labeled local feedback. The problem begins when that result is promoted as the multi-browser release record. Keep development feedback fast, and reserve strict completeness checks for lanes that make a completeness claim.

Do not use project names as a substitute for product ownership if the mapping is false. Browser projects usually express execution configurations, not business domains. The owner metadata in the example belongs to the platform because the matrix itself is a platform promise. Individual checkout or billing tests still need their real product owners in the repository’s ownership system.

Finally, skip artifact collection that creates more risk than evidence. A test handling regulated or highly sensitive data may need redacted fixtures, restricted storage, or no screenshots at specific steps. Playwright can retain what the test and trace capture, but governance must decide what should never be captured. The safest report is not the largest one.