Record every retry without filming every Playwright test

A test fails twice and passes on its third run, but CI keeps only one video. The missing clip is not random if the suite still uses on-first-retry. When each retry can fail differently, one recording is not enough to compare the attempts.

on-all-retries solves that narrow evidence gap. It records and keeps every retry, not every run. That last distinction explains most surprises with the setting.

Know exactly which attempts are recorded

Playwright calls the initial execution the first run. Any later execution caused by the retry allowance is a retry. With retries: 2, a test can therefore have up to three attempts: first run, retry one, and retry two.

The on-all-retries video mode records retry one and retry two if they occur. It never records the first run. It also keeps each retry recording regardless of whether that retry passes or fails. The policy is based on attempt position, not final classification.

Here is the practical matrix for two configured retries:

The distinction from on-first-retry appears only when a test reaches another retry. Both modes keep retry one. Only on-all-retries also records retry two and any later retries allowed by configuration.

The distinction from on is larger. on records every run, including first-attempt passes, so a stable suite can create video for every test. on-all-retries leaves first-attempt passes alone and begins only after failure triggers retry execution.

The distinction from retain-on-failure is about both recording and retention. retain-on-failure records every run but keeps a particular run's video only when that run fails. If the final retry passes, its recording is discarded. on-all-retries keeps that passing retry because it was a retry. This is useful when you need to compare the last failing retry with the recovery that followed.

Two newer alternatives fill gaps on either side. retain-on-first-failure records only the initial run and keeps it when that run fails. retain-on-failure-and-retries records every run, then keeps failed runs and all retries. The second option is the closest thing to a complete retry narrative without retaining video for first-attempt passes. It is also more expensive because a test that fails first and later passes keeps both sides of the comparison.

Choose from the investigation question. If the question is "what did the original failure look like?", record the first run. If it is "how did retry one differ from retry two?", use on-all-retries. If both questions routinely matter, accept the cost of the broader retention mode. A mode name should describe evidence policy, not become a standard copied into every project.

The normal configuration belongs under use:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [['line'], ['html', { open: 'never' }]],
  use: {
    video: 'on-all-retries',
    trace: 'retain-on-failure-and-retries',
  },
});

Video received the extended modes, including on-all-retries, in Playwright 1.61. If TypeScript rejects the value or the runner reports an invalid mode, check npx playwright --version and the package lock. Do not silence the type error with a cast. A cast cannot teach an older runner a new option.

This example pairs video with a broader trace policy on purpose. The first run is absent from on-all-retries video, while retain-on-failure-and-retries keeps a failed first run as well as retries. That combination gives a visual comparison for retries and trace evidence for the trigger. It costs more storage than either setting alone.

Verify the policy with a disposable three-attempt test

Artifact settings are easy to review incorrectly because a naturally flaky test may recover at a different point on each run. Use a deterministic probe. It should fail on the first run, fail on retry one, and pass on retry two. Then the expected video count is unambiguous: two retry videos and no first-run video.

import { test, expect } from '@playwright/test';

test('on-all-retries video probe', async ({ page }, testInfo) => {
  await page.setContent(`
    <main>
      <h1>Artifact policy probe</h1>
      <p data-testid="attempt">Attempt ${testInfo.retry}</p>
      <button type="button">Continue</button>
    </main>
  `);

  await page.getByRole('button', { name: 'Continue' }).click();
  await expect(page.getByTestId('attempt')).toHaveText('Attempt 2');
});

Run only this scratch file with exactly two retries and the HTML reporter enabled. Attempt zero displays Attempt 0 and fails. Retry one displays Attempt 1 and fails while being recorded. Retry two displays Attempt 2 and passes while being recorded. Playwright classifies the whole test as flaky because a later attempt recovered.

npx playwright test tests/video-policy.probe.spec.ts \
  --retries=2 \
  --reporter=line,html

find test-results -type f -name '*.webm' -print | sort
npx playwright show-report

Do not rely solely on file count in a real suite. A test can open more than one page, output paths are scoped per test attempt, and other tests may create videos. In the probe, isolate the file, use one page, and inspect the attachment list for each retry in the HTML report. The test page itself prints the retry index, so the visible content corroborates the attempt metadata.

Delete the probe after validation. It is deliberately flaky and should not become a permanent acceptance test. If you want a lasting configuration check, put it in a separate tooling project that is not part of release coverage and assert its artifact inventory in that project's own CI job.

The probe answers whether the policy works. It does not prove that CI retains the output after the runner finishes. That is a separate boundary. If videos exist in test-results inside the test step but disappear from the downloadable artifact, fix the upload path or condition rather than changing Playwright modes.

testInfo.retry is zero-based. Zero means the first run, one means the first retry, and two means the second retry. Do not confuse it with repeatEachIndex. --repeat-each creates repeated executions for stress or reproduction, but each repeated case can still have its own retry chain. A repeated execution at retry zero is not recorded by on-all-retries merely because another repetition ran before it.

That distinction is useful when a diagnostic command combines both options. Suppose --repeat-each=10 --retries=2 runs a case many times. Videos appear only for repetitions that fail and enter their retry chain. Count them per reported test result, not against the total number of process executions. Otherwise a correct sparse artifact set looks like data loss.

Diagnose a missing video from the first absent boundary

Start with attempt history. Open the HTML report and count how many times the test ran. A passed first attempt should have no video under this mode. A test with retries configured but never used should also have none. The configuration creates permission to record retries; it does not manufacture retry attempts.

Next, verify the resolved project. use.video can be set globally and overridden in a project or with test.use. The test result page shows the project identity. Read the relevant configuration path rather than assuming the top-level value won. A mobile project set to video: 'off' will not inherit the global mode.

Then inspect when the failure happened. Video recording is associated with a browser context and its pages. A configuration parse error, test collection error, browser launch failure, or worker failure before a page exists cannot produce a useful page video. The absence is expected evidence about the failure stage. Look at runner stderr and global errors instead.

After that, inspect context shutdown. Playwright documents that a video is fully written when the browser context closes. The Playwright Test runner manages its standard fixtures and attaches retained videos after the attempt. Code that creates an additional context manually should close it explicitly:

import { test, expect } from '@playwright/test';

test('manual context closes so its video is flushed', async ({ browser }, testInfo) => {
  const context = await browser.newContext({
    recordVideo: { dir: testInfo.outputPath('manual-video') },
  });

  try {
    const page = await context.newPage();
    await page.goto('https://example.com');
    await expect(page.getByRole('heading', { name: 'Example Domain' })).toBeVisible();
  } finally {
    await context.close();
  }
});

This manual recordVideo example is a library-level recording, not an implementation of the on-all-retries retention policy. It demonstrates the lifecycle rule only. Do not replace the test-runner option with manual contexts throughout a suite. You would inherit file naming, retention decisions, attachment wiring, and cleanup.

Finally, check CI transport. List the .webm files immediately after the test command with a step that runs even when tests fail. Then list the downloaded artifact. If the first list has the file and the second does not, Playwright completed its part. Common causes are uploading only playwright-report/ when video data lives under test output, an artifact condition that runs only on success, or a path rooted in the wrong package of a monorepo.

Do not promise that every .webm will sit at one fixed filename such as video.webm. Use report attachments as the authoritative mapping between test attempt and file. A recursive extension search is a diagnostic inventory, not an API contract.

A second near-miss is an HTML report with working video links on the CI machine but broken links after publication. That points to report packaging or hosting, not recording. Upload the complete report folder and any separately retained test output required by your chosen workflow. Do not copy only index.html and assume linked data followed it. Test the downloaded artifact in the same form reviewers receive it.

When a monorepo runs Playwright from a package directory, resolve paths from that working directory. A root-level upload of test-results/ can be empty while packages/storefront/test-results/ contains every clip. Print the working directory and file inventory in the failing job. This is faster and more reliable than adding a second video mode and hoping files appear somewhere else.

Multi-page tests require one more check. Playwright associates video with pages, so a retry that opens a popup can contribute more than one recording. The main page clip may end at the click that opened the popup while the actual failure appears in the popup clip. Review attachments on that retry before concluding that recording stopped early. If the popup never opened, the single main-page video and the action trace support a different failure: the triggering interaction did not create the expected page.

Close manually managed pages and contexts through normal cleanup so recordings can finish. Do not add a fixed delay at the end of the test to "give video time." Video finalization follows context lifecycle, not an arbitrary sleep. A job killed by an outer CI timeout may still leave incomplete evidence because graceful cleanup and upload never ran.

Compare retries that fail for different reasons

The mode earns its storage when later attempts contain information that the first retry cannot provide. Three examples show where the comparison helps and where it does not.

Worked example: an overlay changes between retries

A checkout button times out on the first retry because a consent overlay covers it. On the second retry, the overlay closes, the click lands, and a payment request returns an error. With only the first retry video, the team sees the overlay and assumes every attempt failed there. With both videos, it sees two separate failure stages.

Attach attempt metadata so the report does not depend on reading a small number rendered in the video:

import { test, expect } from '@playwright/test';

test.beforeEach(async ({ page }, testInfo) => {
  await testInfo.attach('attempt.json', {
    body: Buffer.from(JSON.stringify({
      retry: testInfo.retry,
      workerIndex: testInfo.workerIndex,
      parallelIndex: testInfo.parallelIndex,
      project: testInfo.project.name,
    }, null, 2)),
    contentType: 'application/json',
  });

  await page.goto('/checkout');
});

test('submits card payment', async ({ page }) => {
  await page.getByRole('button', { name: 'Accept cookies' }).click();
  await page.getByRole('button', { name: 'Pay now' }).click();
  await expect(page.getByRole('heading', { name: 'Payment confirmed' })).toBeVisible();
});

This test assumes the consent control is part of the product contract. If it is optional, use a locator check or test fixture appropriate to the actual app rather than copying the exact flow. The attachment APIs and retry fields are real; the application selectors are illustrative integration points.

Compare each retry at the first diverging action. Video can confirm that an overlay, focus state, layout shift, or spinner looked different. Trace should confirm the action log, locator resolution, DOM snapshot, console, and network evidence. A video of a button that appears enabled does not prove the request left the browser.

Worked example: the missing first run is the only useful failure

A cold service returns an error on attempt zero. Both retries pass after the service warms. on-all-retries dutifully records two successful flows and omits the failing response. Nothing is broken in video retention; the selected mode does not cover the first run.

Use retain-on-first-failure if you need only the initial failed run. Use retain-on-failure-and-retries if you need the failing first run plus every retry, including a passing recovery. The latter creates the most complete attempt story and the largest evidence set among these targeted choices.

The identifying evidence is an attempt-zero HTTP or assertion error in the report with videos attached only to retry results. Do not respond by increasing retries or manually copying a retry video. Change the policy to match the investigation question.

Worked example: repeated authentication failures evolve

Retry one reaches the login form but receives an expired test credential. Retry two uses newly provisioned data but lands in a forced password-reset screen. The final result remains failed. Two retry videos show that the setup changed the failure rather than repairing it.

Check request responses and test-data logs before blaming the UI. A screenshot of the reset screen establishes state, while the credential creation response establishes why that state existed. If retry-specific fixture logic provisions a new user, review testInfo.retry branches carefully. Retry code can turn one defect into a different defect and make the sequence harder to reason about.

The fix may be to remove attempt-dependent data mutation and create a valid isolated user before the first run. That costs provisioning time on every test but makes attempts comparable. Alternatively, a documented cache cleanup before retries may be correct for a known environment boundary. The report should make that policy visible instead of hiding it in a helper.

Keep the CI artifact bill and diagnosis value aligned

Video consumes CPU, disk, compression time, upload time, storage, and reviewer attention. on-all-retries controls volume better than on when most tests pass immediately, but a highly unstable suite can still produce many recordings. A test with four retries can retain four videos.

Upload evidence even when the test command fails:

- name: Run Playwright tests
  run: npx playwright test

- name: Inventory retained videos
  if: ${{ !cancelled() }}
  run: find test-results -type f -name '*.webm' -print | sort

- name: Upload Playwright evidence
  if: ${{ !cancelled() }}
  uses: actions/upload-artifact@v5
  with:
    name: playwright-evidence-${{ github.run_attempt }}
    path: |
      playwright-report/
      test-results/
    retention-days: 7

Choose retention from investigation reality and data policy, not from an invented universal number. Seven days above is an explicit example, not a measured recommendation. Teams with weekend queues or regulated audit needs may require longer. Teams whose videos contain sensitive test data may require tighter access and shorter retention.

Control recording size through scope before lowering visual quality. Apply the mode to projects where retry comparison is useful, such as a small end-to-end browser matrix, rather than every fast API-only project. An API test with no meaningful page does not gain much from a browser video.

You can roll the mode out to a tagged diagnostic slice before adopting it across a browser project:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  projects: [
    {
      name: 'chromium',
      grepInvert: /@retry-video/,
      use: { ...devices['Desktop Chrome'], video: 'off' },
    },
    {
      name: 'chromium-retry-video',
      grep: /@retry-video/,
      use: {
        ...devices['Desktop Chrome'],
        video: 'on-all-retries',
        trace: 'retain-on-failure-and-retries',
      },
    },
  ],
});

This split avoids running tagged tests twice because the ordinary project excludes the same tag. Its cost is another project definition and a policy developers must understand. It can also hide a new flaky test if nobody adds the tag. Use it as a migration step with a removal date, not as a permanent substitute for a suite-wide evidence decision.

During the canary, review actual artifacts rather than estimating them. Record how many tests reached retries, how many pages produced video, how much data CI uploaded, and whether reviewers used the second retry clip. Those become measurements only after the jobs run. Until then, describe the expected shape without attaching invented percentages or storage savings.

Watch runner headroom during rollout. Recording can expose a CPU- or disk-constrained agent. Compare equivalent jobs with and without the mode, label any figures as measurements only after collecting them, and inspect whether timeouts cluster under load. If evidence capture changes outcomes, reduce worker concurrency or increase runner capacity before declaring the application flaky.

Give reviewers an order of operations. Read the error and trace first, open video when visual state matters, and compare retry metadata. Without that habit, a folder full of clips becomes expensive theater. With it, the second retry can disprove the story suggested by the first.

Make cleanup ownership explicit as well. Test output directories are often removed at the start of a later command. If a job runs browser projects sequentially into the same configured output location and uploads only at the end, verify that later invocations do not erase earlier evidence. Prefer one runner invocation for projects that share a report, or upload each invocation's evidence to a unique artifact before starting the next. The exact layout is a CI design choice; silent reuse is the failure to avoid.

Separate an application stall from recording pressure

Two retries can end with the same locator timeout and show the same spinner, yet require different fixes. In one case, the application request that should dismiss the spinner remains pending or returns an error. In the other, the request and response complete normally, but the browser and runner fall behind while recording and compressing video on a saturated agent. The terminal's final timeout line does not separate those causes.

Read the retry result before reading filenames. With two retries reached, the healthy artifact shape is a result for retry index one with its video attachment and a result for retry index two with another video attachment. The overall test may be flaky, failed, or recovered, but that final classification does not describe attachment coverage. A broken recording boundary has a retry result with no video attachment and no corresponding file in the runner-side inventory. A broken transport boundary has the attachment and file before upload, then loses the file in the downloaded artifact. A misleading inventory can show three video files for two retries because one retry opened a popup. Count attempt attachments and pages, not bare extensions.

For the identical spinner failures, inspect the first event that should have advanced the page. A request still pending at the attempt deadline, an error response, or an application console failure points toward the product or its service. A completed response followed by unchanged DOM state points toward client behavior. Broad delay across navigation, actions, fixture work, and artifact finalization, coupled with runner CPU or disk pressure, supports an infrastructure-capacity explanation. One slow request alone does not prove recorder pressure. Compare an equivalent diagnostic job with recording disabled before changing worker count or machine size.

This distinction has a concrete cost. Reducing workers to restore headroom lengthens the job because less work runs concurrently. Moving to a larger runner increases compute spend. Keeping current capacity and narrowing video to one project preserves throughput, but it leaves other projects without the retry comparison. Raising locator timeouts spends developer wait time and can conceal the capacity signal, so it is the weakest first response.

Roll the producer change only after downstream consumers can represent it. A custom evidence collector keyed only by test identity may keep one attachment and overwrite the other because retries belong to the same test. Land attempt-aware storage and report rendering first, then enable the new mode on a canary project, then adjust artifact quotas and retention, and only then expand the project set. The first proof is not that CI stays green. It is that a controlled two-retry probe displays two separately labeled retry attachments after download and that an ordinary first-pass success displays none.

Ownership follows the first broken boundary. The automation framework owner handles resolved Playwright policy and attempt mapping. The CI platform owner handles runner capacity, archive paths, and retention. The application owner handles a request or render transition that is unhealthy with adequate runner headroom. A useful handoff includes the run and shard, project, test identity, retry index, project-local Playwright version, attachment list before upload, downloaded inventory, first diverging trace action, relevant response state, and agent resource evidence. Sending only the final timeout and one clip forces the receiving team to recreate the classification work.

This comparison does not detect a server-side data mutation that leaves the same pixels and network status on both retries. If the response is syntactically successful but contains the wrong business state, retain a sanitized response or domain-specific attachment. More retry video cannot reveal data that the page never renders.

When not to record every retry

Do not choose this mode when the initial failure is your main evidence. It never records attempt zero. retain-on-first-failure or retain-on-failure-and-retries fits that question better.

Do not choose it in a suite with retries disabled. No retry means no recording. Use a failure-retention mode if video is required for a single failed run, or leave video off if trace and screenshots already answer the likely failures.

Avoid it for stable, fast checks where a retry video adds no diagnostic signal. API contract tests, setup projects with no page interaction, and failures before browser creation need logs, response bodies, or runner errors instead.

Do not use multiple retry videos as permission to keep unreliable tests indefinitely. More evidence makes a flake easier to study, but every recovered attempt still indicates inconsistent behavior. Pair the artifact policy with ownership and a decision about whether flaky tests fail CI.

Skip video when the visual record would expose secrets, personal data, or privileged admin screens that your artifact store cannot protect. Masking data in the application or using safe test accounts is better than trusting every reviewer to handle downloaded clips correctly.

Most of all, do not infer that a missing first-attempt clip is an upload bug. Read the attempt matrix first. Under on-all-retries, absence of the first run is the configured behavior, and changing CI paths will never create a recording that Playwright was told not to make.