Classify Flaky, Expected, and Failed Tests with Playwright Retries

Retries are useful only when the first failure remains visible. If a pipeline converts "failed, then passed" into an ordinary green check, the suite loses diagnostic value and intermittent defects accumulate. Playwright provides the mechanics to rerun in a fresh worker and classify outcomes, but the team must connect each classification to an operational decision.

There are three separate ideas to keep straight. A flaky test changes outcome across attempts. An expected failure is a deliberately declared mismatch for a known condition. A failed test remains unexpected after its allowed attempts. Treating these as one bucket produces confused dashboards, stale annotations, and retries that function as concealment.

Read retries as classification evidence

The official Playwright retries guide defines a first-attempt pass as passed, a failure followed by a retry pass as flaky, and a test that exhausts retries as failed. That taxonomy describes execution history; it does not identify root cause. A flaky outcome may come from the product, test, environment, or shared data.

Configure retries to create another controlled observation, not to manufacture a preferred result. One retry often answers the most important first question: does the same test pass in a clean worker immediately after failure? The appropriate maximum depends on suite cost and incident policy, but every additional attempt spends resources and can reduce the urgency of a weak signal.

Account for the fresh worker after failure

Playwright Test executes files in independent operating-system worker processes. After any test failure, the runner discards the worker and its browser. Subsequent tests continue in a new worker, and an enabled retry begins there. beforeAll and worker-scoped fixture setup can therefore run again.

This behavior contains damage from a failed test, but it also provides a clue. If the retry passes only after process restart, inspect process-level caches, worker fixtures, server state, reused accounts, and setup order. The browser page from the first attempt is not being "tried again" in place.

Make setup idempotent and cleanup tolerant of partial execution. A failed test may leave a remote record even though its local fixture is gone. The next worker must either create an isolated record or reconcile that residue explicitly. Depending on worker restart as the cleanup mechanism leaves external systems dirty.

Configure a policy that preserves the failed attempt

A practical configuration can keep local feedback immediate while enabling one evidence-producing retry in CI. Capture a trace on the retry and retain failure screenshots. The resulting artifacts should be uploaded even when the test command fails.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
  reporter: process.env.CI
    ? [['line'], ['html', { open: 'never' }]]
    : [['list']],
});

Do not automatically raise retries when flakiness grows. That increases the probability of an eventual pass without improving the underlying contract. Track the first-attempt failure rate or flaky outcome count separately from terminal failures, and alert when either violates team policy.

Distinguish attempt status from test expectation

Each TestResult describes one attempt and can report passed, failed, timedOut, skipped, or interrupted, along with its retry number, errors, attachments, and timing. The test also has an expected status. Classification emerges from the relationship between actual attempts and that declared expectation.

An ordinary test expects to pass. A first failure followed by a pass is flaky. Repeated failures remain unexpected. A test marked with test.fail() expects to fail; Playwright runs it and checks that it actually does. If it suddenly passes, the expectation is stale and the run should draw attention.

This is why "expected failure" is not a polite label applied by a dashboard after the run. It is executable policy in the test and must have a reason, ownership, and removal condition.

Use expected-failure annotations narrowly

Mark a failure expected only when the behavior is known, scoped, and still worth executing. Include an issue identifier or precise reason, and apply the condition only to the affected project or environment.

import { expect, test } from '@playwright/test';

test('downloads an invoice in the supported browser', async ({ page, browserName }) => {
  test.fail(
    browserName === 'webkit',
    'QA-482: invoice download response is not handled in the WebKit project',
  );

  await page.goto('/billing/invoices');
  const download = page.waitForEvent('download');
  await page.getByRole('button', { name: 'Download invoice' }).click();
  await expect((await download).suggestedFilename()).toMatch(/invoice-\d+\.pdf/);
});

The test still exercises the broken behavior and will object when the condition unexpectedly passes. After the product fix lands, remove the annotation and verify the ordinary expectation across the intended projects.

Use test.fixme() when executing the scenario is currently unsafe, crashes, or consumes unacceptable resources. Use test.skip() when a scenario genuinely does not apply to a configuration. Neither one supplies runtime evidence, so neither should become a permanent substitute for repair.

Keep retry attempts comparable

Playwright exposes testInfo.retry, starting at zero for the initial attempt. It is tempting to use it to make the second attempt more forgiving: extend a timeout, choose a different account, skip setup, or bypass an assertion. That creates two different tests and makes a passing retry meaningless.

Retry-aware code is defensible for extra logging, targeted attachments, or idempotent cleanup of environmental residue. It should not alter business data or the expected user outcome. If a cache must be cleared only on retry, investigate why the first attempt begins with contaminated state and make the cleanup part of deterministic setup when possible.

import { test } from '@playwright/test';

test.beforeEach(async ({ page }, testInfo) => {
  if (testInfo.retry > 0) {
    await testInfo.attach('retry-context', {
      body: Buffer.from(JSON.stringify({
        retry: testInfo.retry,
        project: testInfo.project.name,
      })),
      contentType: 'application/json',
    });
  }

  await page.goto('/dashboard');
});

The scenario remains identical. The later attempt simply carries an explicit diagnostic marker.

Classify the failure domain after a flaky result

A flaky label starts investigation; it does not finish it. Compare the first and passing attempt at their earliest divergence.

Product race: the same input produces different event ordering or UI state. Keep the regression visible and repair the readiness or concurrency defect.

Test synchronization: the assertion observes an intermediate state or waits on a proxy condition. Replace sleeps and weak checks with an observable product outcome.

Data collision: parallel workers mutate the same account, row, file, or queue. Allocate a namespace by worker or lease independent records.

Environment pressure: the browser or service becomes starved, crashes, or exceeds a realistic timeout. Measure worker resources and service health before increasing budgets.

External instability: an uncontrolled dependency varies or is unavailable. Decide whether the suite should virtualize that boundary, retry at a lower integration layer, or surface the dependency incident directly.

Quarantine with an exit condition

Quarantine is appropriate when a known intermittent test blocks unrelated delivery and cannot be repaired immediately. It should preserve visibility through a separate project, tag, or scheduled job. Record owner, issue, evidence, impact, and a date or condition for reevaluation.

Do not quarantine by deleting the test or applying an unexplained global skip. That erases both coverage and accountability. Avoid marking a genuinely flaky test as expected to fail: it may pass, which contradicts test.fail(), and the annotation misrepresents a variable outcome as a stable known defect.

A quarantine dashboard should distinguish not run, expected failure, flaky, and failed. Combining them into "non-passing" prevents sensible release policy. A critical product regression that fails consistently needs a different response from an irrelevant browser scenario that was correctly skipped.

Diagnose misleading retry patterns

Pass on retry after a worker restart often indicates leaked worker state or an external record left by setup. Failure on every retry with the same first error is probably deterministic and gains little from more attempts. Different errors across attempts suggest an unstable prerequisite, broad timeout cascade, or environment pressure.

If the first attempt times out and the retry passes quickly, compare network and fixture chronology rather than assuming a slow page. If retries fail on different tests after a common beforeAll, inspect that shared setup and remember it runs again in the replacement worker.

Serial groups require extra care because the group is retried together and later tests may be skipped after an earlier failure. Prefer isolated tests so each result maps to one scenario and can be retried independently.

Operational retry-classification checklist

• Preserve first-attempt errors and attachments.
• Use retries as another observation, not a success converter.
• Remember that failure causes a fresh worker and browser.
• Keep setup idempotent across worker restarts.
• Separate passed, flaky, expected, skipped, and failed reporting.
• Scope test.fail() to a known condition with an issue reference.
• Remove expected-failure annotations after the verified fix.
• Keep retry-aware behavior diagnostic and scenario-neutral.
• Give quarantined tests ownership and an exit condition.
• Compare attempts at the earliest divergent event.

Convert every classification into action

Adopt a simple response contract: passed tests need no incident work; flaky tests enter a triage queue with both attempts attached; expected failures are checked against their issue and removal condition; terminal failures block or escalate according to product risk; skips are reviewed for continued applicability. With that contract, retries add evidence and worker isolation without diluting the suite's signal. The objective is not the highest eventual pass count. It is a result history precise enough to tell the team what must happen next.