A green retry should still fail your Playwright build

Your suite is green, but one checkout test needed its second attempt. The pull request can merge because the retry repaired the result, while the HTML report quietly labels the test flaky. That is a release policy bug, not a successful run.

Teams often search for a Playwright reporter switch to fix this. The switch exists, but it is not a reporter option. failOnFlakyTests belongs to the test-runner configuration, and its job is narrow: if any test is classified as flaky, the overall run exits with an error. Reporters still provide the evidence that explains why the gate fired.

Why a passing retry changes the run result

Playwright classifies an ordinary test by looking across its attempts. A test that passes on the first run is passed. A test that fails first and later passes within its retry allowance is flaky. A test that never recovers is failed. Those labels describe different evidence, even when the last browser page looked healthy.

Without the flaky-test gate, a run containing passed and flaky tests can finish successfully. This default is practical for teams that use retries as a temporary shock absorber. It is dangerous when a green CI badge is interpreted as proof that the tested behavior was repeatable. One attempt saw a real assertion, timeout, hook, or infrastructure error. A later attempt did not erase it.

The basic configuration is deliberately small:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  failOnFlakyTests: Boolean(process.env.CI),
  reporter: process.env.CI
    ? [['line'], ['html', { open: 'never' }]]
    : [['list'], ['html', { open: 'never' }]],
});

retries and failOnFlakyTests do separate work. The retry produces another attempt. The gate evaluates the resulting flaky classification. Setting the gate while leaving retries at zero is valid, but it cannot turn an unretried failure into a flaky result. That failure already makes the run fail for the usual reason.

The reporter entry above also does not enforce the policy. The line reporter gives CI a compact log, and the HTML reporter preserves a navigable record. You could replace both and the gate would behave the same way. This matters during review because placing failOnFlakyTests inside an HTML reporter options object is not an alternative spelling. It is the wrong configuration level.

The command-line form is useful when configuration ownership is complicated:

npx playwright test --retries=2 --fail-on-flaky-tests

Run that in a non-required CI job first. If it exits non-zero while the existing job stays green, you have found tests that your current release signal normalizes. You have not yet proved why those tests are unstable.

The following timings are illustrative. A common line or list reporter sequence for a recovered test has this shape:

x  1 tests/checkout.spec.ts:18:5 › card checkout (1.2s)
✓  2 tests/checkout.spec.ts:18:5 › card checkout (retry #1) (846ms)

1 flaky
  tests/checkout.spec.ts:18:5 › card checkout

Exact symbols and timing text can vary with the reporter and Playwright version. The decisive evidence is the failed attempt followed by a passing retry and the final flaky classification. A log containing only the final check mark has usually been truncated, reformatted by a CI integration, or read without opening the test's attempts in the report.

Prove the gate before changing release policy

Do not validate this option by waiting for a production flake. Add a disposable test that deterministically fails on attempt zero and passes on retry one. It proves the classification and the exit behavior without depending on network timing or a race you cannot reproduce.

Place this file in the configured test directory on a branch used only for the trial:

import { test, expect } from '@playwright/test';

test('flaky gate smoke check', async ({}, testInfo) => {
  expect(
    testInfo.retry,
    'attempt zero fails; retry one proves the flaky gate',
  ).toBeGreaterThan(0);
});

With one retry and no gate, the test fails once, passes once, is reported as flaky, and the run can succeed. Add --fail-on-flaky-tests, and the same attempt history should produce an unsuccessful process result. Delete the scratch test after the check. Leaving an intentionally flaky test in the suite trains people to ignore exactly the signal you are trying to protect.

Capture the result without letting shell error handling hide which command failed:

set +e
npx playwright test tests/flaky-gate.smoke.spec.ts \
  --retries=1 \
  --fail-on-flaky-tests \
  --reporter=line
playwright_status=$?
set -e

if [ "$playwright_status" -eq 0 ]; then
  echo "Expected the deterministic flaky test to fail the run" >&2
  exit 1
fi

echo "Flaky-test gate returned status $playwright_status as expected"

This script checks only the boundary you care about: a recovered test must not yield a zero process status. It does not assert a particular non-zero number because CI wrappers and signal handling can affect process codes. It also avoids invented timing thresholds. The smoke test is deterministic by construction.

There are three easy ways to misread this trial.

First, a test that fails every attempt proves normal failure handling, not the flaky gate. Open the report and confirm that at least one retry passed. Second, a test run with --retries=0 never gets the second attempt required for a flaky classification. Third, a CI step with continue-on-error or a shell suffix that discards the command status can paint the job green even though Playwright returned an error. Check the step configuration as well as the test log.

Use the HTML report to corroborate the terminal. Filter for flaky tests, open the smoke test, and inspect both attempts. Attempt zero should contain the assertion failure. Retry one should contain a pass. If the report shows only one attempt, fix the retry configuration before investigating the gate.

Also confirm which configuration the command actually loaded. Monorepos frequently have a root config, a package config, and a CI command that changes the working directory. npx playwright test --list is useful for checking collection, while the opening lines of the run show the selected projects and worker count. If a developer sees retries locally but the trial job does not, inspect the config path and environment before questioning the classification. The option cannot evaluate an attempt that the invoked project never scheduled.

Filters matter for the same reason. A --project, --grep, file argument, or shard limits the cases that participate in that invocation. The gate does not consult a previous HTML report or remember a flaky test from another job. If smoke tests and regression tests run as separate commands, each command makes its own decision. Make both required if both are part of the release contract, or merge their status explicitly in the CI layer.

Distinguish a Playwright retry from a CI rerun

Two logs can show the same test fail and later pass even though only one qualifies for the built-in flaky gate. In a Playwright retry, both attempts belong to one invocation. Attempt zero has testInfo.retry equal to zero, the retry has it equal to one, and the report groups both under one case before classifying that case as flaky. With the gate enabled, that invocation returns an error.

Read three identities before calling the behavior equivalent. The Playwright attempt identity comes from the test case and retry. The process identity comes from the job command and its raw status. The pipeline identity comes from the workflow run and rerun attempt used in artifact names. A healthy in-process retry has one pipeline attempt, one report, and at least two Playwright attempt records for the case. The cross-run near-miss has separate reports and process results, with retry zero in each. A CI interface that labels the later job as another attempt can be misleading because that label does not populate testInfo.retry.

This difference also explains a common evidence gap. Teams retain only the successful rerun's HTML report, then see a first-run pass with no retry and assume the earlier red job was a transient platform error. The failed report contains the only first error, trace, and worker state. Preserve artifacts under run-attempt-specific names before allowing reruns to replace the visible release result.

Land that evidence path before changing rerun acceptance. First make report names unique by pipeline attempt and verify that a failed attempt uploads even when its test step is red. Next preserve the raw Playwright process result beside each report. Only then add a policy that correlates separate invocations. The first rollout failure is usually destructive replacement: the green rerun publishes to the same artifact name or summary slot and makes the red run difficult to retrieve. A correlation rule built after that loss has no trustworthy first sample to compare.

The CI policy owner decides whether a successful rerun can satisfy a required check. The Playwright or suite owner decides how retries operate inside one invocation. When handing off a cross-run recovery, include both process statuses, both report artifact names, commit, config path, selected projects, worker count, and the first failure evidence. Without those fields, the receiving team cannot tell whether code, selection, or environment changed between runs.

Catching cross-run recovery requires pipeline-level correlation or historical test analysis, not another Playwright reporter setting. That broader gate costs storage for multiple reports, stable identity across runs, and rules for reruns made after an infrastructure repair or configuration change. It can also block a release because two runs were not comparable. Keep that cost explicit instead of claiming the current-run option enforces historical stability.

Read expected status before routing a red line

An acknowledged expected failure can look like an ordinary failed attempt in low-detail terminal output. The separating evidence is expectedStatus. For an ordinary case that starts a flaky chain, attempt zero has actual status failed while the test expected passed; a later attempt passes and the case outcome becomes flaky. A test marked as expected to fail has expected status failed. When it actually fails, the case outcome is expected, not flaky.

This distinction prevents the gate from being blamed for honoring an explicit test contract. A failure symbol or nonempty error array is misleading when read without expected status and case outcome. Conversely, an expected-failure marker is not a general quarantine mechanism. If the behavior unexpectedly starts passing, Playwright treats that disagreement as unexpected rather than silently certifying the known defect as fixed.

The suite owner controls the expected-failure annotation and must supply its reason, product owner, and removal condition. The release-policy owner controls the flaky gate. Their handoff should contain the annotation visible in the report, actual and expected statuses, case outcome, and every attempt. The concrete maintenance cost is a second class of accepted red evidence that must be reviewed separately. failOnFlakyTests does not turn an intentionally expected failure into flaky debt, so a stale annotation can continue accepting the known defect until its owner deletes it.

Find the cause instead of blaming the retry

Once the policy works, the difficult part starts. failOnFlakyTests tells you that an outcome changed. It cannot tell you whether the cause was the application, test data, test code, or the CI machine. Treat the first failed attempt as primary evidence and the retry as a comparison sample.

Start with the error that ended attempt zero. A locator timeout and an HTTP 503 can lead to the same final flaky label, but they demand different owners. Read the assertion message, call log, attachments, and trace before increasing a timeout. Then compare the retry at the same point. The useful question is not "why did the retry pass?" in the abstract. It is "what observable input or state differed immediately before the first failing operation?"

Worked example: a read-after-write race

An order test creates a record through an API, opens the order page, and expects the new row immediately. The first read reaches a replica before the write is visible. By the time Playwright starts a fresh worker and retries, replication has caught up. The gate correctly calls the test flaky, but adding another retry only lengthens the disguise.

The original test often looks innocent:

import { test, expect } from '@playwright/test';

test('new order appears in history', async ({ page, request }) => {
  const created = await request.post('/api/orders', {
    data: { sku: 'QA-BOOK', quantity: 1 },
  });
  expect(created.ok()).toBeTruthy();

  const order = await created.json() as { id: string };
  await page.goto('/orders');
  await expect(page.getByTestId(`order-${order.id}`)).toBeVisible();
});

The failure evidence that identifies this race is specific. The create response is successful and contains the new id. The first page request or API response used by the page does not contain that id. The retry's corresponding response does. A screenshot showing an empty table is compatible with this explanation, but it is not sufficient by itself because a bad filter, wrong tenant, or failed JavaScript request produces the same pixels.

The durable fix belongs at the contract boundary. If the product promises immediate read-after-write behavior, the backend should satisfy it and the test should stay strict. If the product explicitly exposes eventual consistency, poll the authoritative user-visible condition within a bounded product allowance. Do not sleep for an arbitrary duration.

import { test, expect } from '@playwright/test';

test('new order becomes visible within the product allowance', async ({ page, request }) => {
  const created = await request.post('/api/orders', {
    data: { sku: 'QA-BOOK', quantity: 1 },
  });
  expect(created.ok()).toBeTruthy();

  const order = await created.json() as { id: string };
  await page.goto('/orders');

  await expect
    .poll(async () => {
      const response = await request.get(`/api/orders/${order.id}`);
      return response.status();
    }, {
      message: `order ${order.id} should become readable`,
      timeout: 10_000,
      intervals: [250, 500, 1_000],
    })
    .toBe(200);

  await page.reload();
  await expect(page.getByTestId(`order-${order.id}`)).toBeVisible();
});

The cost is explicit. This version can wait up to the product's consistency allowance and makes extra reads. It is justified only if eventual visibility is the documented behavior. If ten seconds was chosen merely because it made CI green, the test now hides a latency regression.

Worked example: shared credentials across workers

Two parallel tests sign in as the same customer. One changes the locale or empties the cart while the other asserts the previous state. The failed attempt and retry run in different scheduling conditions, so the collision disappears. A longer expect timeout does not solve shared ownership.

Look for the same account id, cart id, workspace, or feature-flag key in failures that overlap in wall-clock time. Compare testInfo.parallelIndex, project name, and request payloads attached to each test. The strongest evidence is not that tests pass with one worker. It is that conflicting writes use the same resource while their execution intervals overlap.

Allocate state per test or per parallel slot. A test-scoped fixture is usually easier to clean up because its lifetime matches one case:

import { randomUUID } from 'node:crypto';
import { test as base, expect } from '@playwright/test';

type Fixtures = {
  account: { email: string; password: string };
};

export const test = base.extend<Fixtures>({
  account: async ({ request }, use) => {
    const email = `pw-${randomUUID()}@example.test`;
    const response = await request.post('/api/test-support/accounts', {
      data: { email, password: 'local-test-password' },
    });
    expect(response.ok()).toBeTruthy();

    await use({ email, password: 'local-test-password' });

    await request.delete(`/api/test-support/accounts/${encodeURIComponent(email)}`);
  },
});

This is an integration pattern, so the support endpoint must be an authenticated test-only facility in your own environment. The important property is unique ownership, not the sample URL. Per-test provisioning adds API traffic and cleanup code. In return, the result no longer depends on which worker reaches a shared account first.

Near-miss: a worker crash is not a flaky assertion

A browser process exit, out-of-memory kill, or unhandled worker error can also disappear on retry. The report may still end with a flaky test because the later attempt passes, but the first attempt will not show the expected assertion mismatch. It may end abruptly, show a closed page or browser message, or contain a global error associated with the worker.

That distinction changes the owner. Application assertions go to the product or test team. Repeated browser launch failures, host pressure, and killed processes go to the CI platform investigation first. Capture machine-level logs and container termination reasons. Do not rewrite the locator merely because its test title received the flaky label.

Worker replacement can also explain a recovery without a crash. Playwright discards the worker process after a test failure and continues in a new one; with retries enabled, the failed test is retried from that fresh worker. File-level beforeAll work runs again. Worker-scoped fixtures are rebuilt. Browser and in-process module state do not carry over from the failed process.

Compare workerIndex, setup logs, and fixture-created resource ids between attempts. If retry one passes only after beforeAll reseeds data or a worker fixture obtains a different account, the evidence points to state ownership rather than a slow locator. Reproducing with one worker does not remove this lifecycle because a failure still causes replacement.

Serial suites add another complication. Playwright retries the serial group together, so earlier cases can run again before the originally failing case. A passing retry may depend on those repeated side effects. The preferred repair is independent tests with isolated setup. If the business scenario must remain serial, make the shared state transition explicit and inspect every case in the retried group, not only the final flaky title.

Wire the gate into CI without losing the evidence

A strict exit code with no retained report creates a queue of untriageable failures. The CI job should upload the HTML report even when Playwright fails. Use the CI system's unconditional or not-cancelled condition for artifact collection, while leaving the test step itself allowed to fail the job.

name: End-to-end tests

on:
  pull_request:

jobs:
  playwright:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: lts/*
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Run Playwright with flaky tests rejected
        run: npx playwright test --fail-on-flaky-tests
      - name: Upload HTML report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v5
        with:
          name: playwright-report-${{ github.run_attempt }}
          path: playwright-report/
          retention-days: 14

The job above assumes the config already enables retries and the HTML reporter. Keeping the flag in the workflow makes the release policy conspicuous. Keeping it in playwright.config.ts avoids drift between CI providers. Either is defensible, but choose one owner and document it. Duplicating the same true value is harmless technically and confusing operationally.

Retain enough evidence to compare attempts. An HTML report records classifications and attachments, but a video alone rarely explains a timing failure. For suites where flakes are actively investigated, a trace mode that preserves failed attempts is often more useful than recording every pass. Artifact policy deserves its own budget review because traces, screenshots, and videos consume storage and upload time.

Do not let a notification layer reinterpret the result. Some CI wrappers parse JUnit and decide job status independently. Others mark the test step as allowed to fail and rely on a later summary step. During rollout, record the raw Playwright process status beside the published report. The source of truth should be obvious when the UI badge and test summary disagree.

Roll out the rule without freezing delivery

Turning the gate on for a large suite can expose years of normalized instability in one morning. That is useful data, but making every existing flake a surprise release blocker encourages teams to disable the option. Roll it out as an engineering policy change, not a one-line config cleanup.

Begin with a baseline job that uses the same tests, projects, workers, retries, and environment as the required job. Change only the flaky gate. Run it often enough to identify recurring offenders. The goal is an inventory with evidence, owner, and failure signature, not a percentage invented from a handful of builds.

Next, remove causes that affect many tests. Shared accounts, mutable global fixtures, unstable test-support APIs, and overloaded runners usually create clusters. Fixing one resource boundary can stabilize dozens of cases. Raising every timeout separately creates dozens of slower cases and leaves the boundary broken.

Then decide how known flakes are handled while the gate becomes required. A narrowly filtered quarantine project can keep a release moving, but the excluded tests no longer protect that release. Make the loss visible. Give each quarantine entry an owner and removal condition. Avoid logic that silently retries until green outside Playwright because those attempts will not participate in Playwright's flaky classification.

Finally, switch the job from advisory to required and watch both failure evidence and runtime. A gate with two retries executes work that it will reject anyway when attempt one recovers. That extra time is not wasted if it supplies a comparison trace that speeds diagnosis. It is waste if nobody opens the report and the same test blocks builds for weeks.

The central trade-off is release availability versus signal honesty. Rejecting every flake can delay a safe change because of a test defect. Allowing every recovered test can ship a product race because the retry hid it. Mature teams do not pretend one side is free. They set the gate according to suite ownership, response time, and the consequence of a false green.

When this gate is the wrong tool

Do not enable it as a substitute for investigating failures. It classifies outcomes; it does not find root causes. If the team lacks access to first-attempt traces, service logs, or test data, improve observability before making the gate the only release signal.

Avoid using it to evaluate a diagnostic stress run built around --repeat-each. Repetition asks how often independent executions fail. Retry classification asks whether a failed test recovered within its configured retry chain. Keep stress results separate from the required acceptance job so the meaning of a red build remains clear.

Do not expect value from the option in a no-retry suite. Every unexpected failure already returns an error, and no failed test can recover into the flaky category. Add retries only if the extra attempt supplies useful diagnostic evidence or reflects an intentional policy. Adding them solely to make failOnFlakyTests relevant creates work without improving coverage.

Be careful with third-party systems that define flaky from historical runs rather than attempts in one Playwright invocation. Historical instability is valuable, but it is a different calculation. The built-in gate acts on Playwright's classification for the current run. Do not claim that it enforces a thirty-day flake threshold or reads yesterday's report.

Skip the gate for local interactive debugging unless a developer is reproducing CI policy. Local work commonly uses one file, headed mode, breakpoints, or zero retries. A non-zero status after an intentional first-attempt experiment adds noise. Keep CI strict and give developers a documented command that recreates the exact gate when needed.

Most importantly, do not turn the option off because the first rollout is red. A recovered failure is evidence that the suite made two different claims about the same behavior. Fix the product race, isolate the data, repair the test, or explicitly remove that check from the release contract. A green badge earned by ignoring the disagreement is the one outcome this setting was designed to prevent.