A green retry is still a CI failure

Your pipeline is green, but the HTML report shows one test failed before it passed on retry. If the job publishes only the final exit status, that instability disappears from the team's view. The application may have a race, the test may share data, or the runner may be unhealthy, and the next commit gets blamed when the same fault finally exhausts its retries.

Retries are useful for collecting another observation. They are not a repair. A dependable CI policy keeps the failed attempt, classifies the recovered test as flaky, and makes that classification affect the gate.

Understand what a retry changes

Playwright starts with zero retries unless the project or command line enables them. When a test fails, the runner discards that worker process and its browser. A retry runs in a new worker, so worker setup, beforeAll hooks, test fixtures, browser context, and page state may all be created again.

That fresh process is an important diagnostic fact. A pass on retry can mean the original assertion was merely early. It can also mean the retry escaped contaminated worker state, received different test data, hit a healthier backend instance, or ran after a cache became warm. The two attempts share a test identity, but they do not share every condition.

Playwright reports three outcomes after retries are considered:

• passed means the test passed on its first attempt.
• flaky means an earlier attempt failed and a retry passed.
• failed means every allowed attempt failed.

By default, a run containing only passed and flaky tests can still exit successfully. That behavior is convenient when a team first introduces retries, but dangerous when a green check is interpreted as reliable product evidence. failOnFlakyTests changes the exit decision without erasing the recovered result.

Do not normalize a "flake rate" by counting only final failures. Ten tests that each pass on their second attempt are ten reproducibility incidents, not a perfect run. Track first-attempt failures by test, project, browser, commit, and error signature. A single percentage without those dimensions mixes application races, infrastructure outages, and weak tests into one number no one can act on.

Configure the gate to preserve every attempt

A practical CI configuration uses a small retry count, fails on recovered tests, and retains enough evidence to compare attempts. This file is valid Playwright TypeScript:

import { defineConfig } from '@playwright/test';

const inCI = Boolean(process.env.CI);

export default defineConfig({
  testDir: './tests',
  retries: inCI ? 1 : 0,
  failOnFlakyTests: inCI,
  outputDir: 'test-results',
  reporter: inCI
    ? [
        ['line'],
        ['html', { open: 'never', outputFolder: 'playwright-report' }],
        ['json', { outputFile: 'test-results/results.json' }],
      ]
    : [['list'], ['html', { open: 'never' }]],
  use: {
    trace: inCI ? 'retain-on-failure-and-retries' : 'off',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
});

The trace choice is deliberate. The commonly copied on-first-retry setting records the first retry, not the original failed attempt. That is economical and often enough to debug a failure that repeats. It is weaker evidence for a flaky test because the recorded retry may pass cleanly. retain-on-failure-and-retries costs more disk and runtime, but preserves the comparison this policy depends on.

The JSON report is useful for trend ingestion. The HTML report is better for a person working one incident. Upload both playwright-report and test-results after the test command regardless of its exit code. Artifact upload belongs in the CI platform's always-run cleanup step; otherwise the most valuable directory disappears precisely when the test command returns nonzero.

Keep retention finite. Videos and traces across a large matrix can consume significant storage. A reasonable policy retains merge-request evidence for days, release evidence for longer, and aggregated flake metrics beyond the raw files. Measure the added test duration and artifact volume before enabling video for every browser project.

Prove that the flaky-test gate works

Teams often set failOnFlakyTests and assume it is active. A tiny canary can verify the runner, retry setting, reporter, artifact upload, and job exit behavior together.

Save this temporary test as tests/flaky-canary.spec.ts:

import { expect, test } from '@playwright/test';

test('CI exposes a recovered attempt', async ({ page }, testInfo) => {
  await page.setContent(`
    <output data-testid="attempt">${testInfo.retry}</output>
  `);

  await testInfo.attach('attempt.json', {
    body: Buffer.from(
      JSON.stringify({
        retry: testInfo.retry,
        project: testInfo.project.name,
      }),
    ),
    contentType: 'application/json',
  });

  await expect(page.getByTestId('attempt')).toHaveText(
    String(testInfo.retry),
  );

  // Attempt 0 fails. Attempt 1 passes.
  expect(testInfo.retry).toBe(1);
});

Run it through the same configuration CI uses:

CI=1 npx playwright test tests/flaky-canary.spec.ts

The expected sequence is one failed attempt, one passed retry, a final flaky classification, and a nonzero process exit because failOnFlakyTests is enabled. The report should expose both attempt.json attachments and the retained attempt artifacts. If the job is green, the CI environment variable, config selection, or command-line overrides are not what the team thinks they are.

This canary is intentionally flaky and should not live in the normal product suite. Run it in a dedicated pipeline validation job or remove it after proving the integration. A permanent known flake corrupts the same metrics the policy is meant to protect.

Use the failed attempt to identify the owner

Open the first failed attempt before the retry. The failure message and stack tell you which assertion or action noticed the problem. The trace then shows the sequence that produced it: locator resolution, DOM snapshots, network activity, console messages, and timing between actions.

Use npx playwright show-report playwright-report to navigate attempts. A trace file can be opened directly with:

npx playwright show-trace path/to/trace.zip

Look for a specific split in the evidence:

• If the expected element appears later in the same stable DOM state, the test probably asserted before the product's real readiness signal.
• If the response is a 500, reset, or timeout, inspect the service and environment before changing the locator.
• If two workers use the same account, order, or filename, repair data isolation.
• If the first attempt starts with state left by another test, find the shared worker fixture, backend record, or storage file.
• If the retry runs against different feature flags or a different deployment, the CI environment is not pinned tightly enough.

A trace cannot see everything. Database rows, message queues, server logs, and deployment identifiers live outside the browser. Attach the smallest safe correlation data to the test: generated entity IDs, response status, build identifier, or sanitized API body. Never attach access tokens, cookies, or full customer records just because the report is private today.

testInfo.retry is useful in those attachments. It lets you align an application log with attempt zero or one. It should not select a weaker assertion, add an arbitrary sleep, or skip setup on the retry. Once the retry follows a different contract, the result no longer tells you whether the original scenario recovered.

Compare attempts at the same checkpoint. If attempt zero received a pending response while attempt one received a completed response, investigate the product's readiness contract. If both responses match but locator resolution differs, inspect rendering and selector scope. If the body assertion passes in both attempts and only fixture teardown fails, the owner is resource cleanup, not the page action named in the test title. A browser crash or worker exit before the first action belongs to the runner or host until application evidence says otherwise.

Write that classification into the defect or quarantine record with links to both attempts. A screenshot from the passing retry is not evidence for the failure, and a stack trace from attempt zero is not evidence that the retry exercised the same backend state. Keeping the pair together prevents those two common triage mistakes.

Reproduce the failure without disguising it

Turn retries off while diagnosing so every bad observation remains bad:

npx playwright test tests/orders.spec.ts \
  --retries=0 \
  --repeat-each=20 \
  --workers=1 \
  --trace=retain-on-failure

A serial repetition helps expose an in-test race without cross-worker competition. If it remains stable, repeat with the normal CI worker count. Failures that appear only in parallel usually point to shared accounts, fixed filenames, rate limits, ports, or backend capacity. Failures that follow one browser project suggest rendering, browser behavior, or project-specific configuration.

Once the cause is known, fix the matching layer:

• Replace fixed sleeps with a web-first assertion on the state the user needs.
• Give each worker unique test data and make cleanup idempotent.
• Wait for the relevant response or UI transition, not generic "network idle."
• Use a role, label, or test ID that describes one element instead of whichever CSS node appears first.
• Repair the application race when the UI announces completion before the operation is durable.
• Pin browser, application build, flags, locale, and timezone when those variables are part of the result.

Every remedy has a cost. Stronger isolation creates more data and cleanup work. Better readiness signals may require product code. Lower worker counts reduce contention but make the pipeline slower and can conceal production concurrency bugs. Longer timeouts increase feedback time and rarely fix a condition with no reliable completion signal.

When retries and a flaky gate are the wrong model

Do not retry an operation that cannot be repeated safely. A payment capture, irreversible deletion, email send, or one-time migration needs an idempotency design or a different test boundary before automatic retries are enabled. A second attempt can change the system again and produce a misleading pass.

Avoid using retries to assess performance. A latency threshold that passes after warming caches is not flaky in the same sense; the retry has changed the workload. Record repeated measurements under a defined performance protocol instead.

Order-dependent suites also need repair before a flaky gate adds much value. Because a failed test restarts its worker, the retry may pass only because preceding state vanished. Treat that as isolation evidence, not proof that the test is acceptable.

There are periods when a team may report flakes without blocking every change, such as the first week of instrumenting a large inherited suite. Make that a time-limited policy with an owner and threshold. Keep the artifacts and visible classification from day one. Otherwise "temporary" retries quietly become the mechanism that turns intermittent failures into permanent green checks.