Keep the failure video that a passing retry cannot explain

The checkout test fails on its first run, passes on retry, and leaves you a video of the successful flow. That clip proves the page can work. It says nothing about the overlay that covered the button during the failed attempt.

When the initial failure matters more than the recovery, retain that attempt directly. retain-on-first-failure records the first run only and keeps its video only when that run fails.

Understand the first-run retention rule

Playwright distinguishes the first run from retries. Attempt zero is the first run. Attempt one is the first retry, attempt two is the second retry, and so on. The mode name refers to the failed first run, not to the first failure anywhere in a chain.

With two configured retries, the retained videos follow this matrix:

Later retries are not recorded under this mode. A retry that exposes a new error cannot contribute another video. That limitation is the main storage benefit and the main diagnostic cost.

The mode also works when retries are zero. A failed single run still qualifies because it is the first run and it failed. This makes the setting useful in strict suites that reject failures immediately but still need a visual artifact.

Configuration belongs in the use section:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  reporter: [['line'], ['html', { open: 'never' }]],
  use: {
    video: 'retain-on-first-failure',
    trace: 'retain-on-first-failure',
    screenshot: 'only-on-failure',
  },
});

The video value was added with the expanded video modes in Playwright 1.61. Check the installed package version if an editor or runner rejects it. Do not force the string through TypeScript with as any; an older runtime will not gain support from a cast.

Pairing the same trace policy with video makes the retained evidence line up on attempt zero. The video shows pixels over time. The trace adds actions, DOM snapshots, console and network context. The screenshot captures one failure-state frame. Keeping all three may be excessive for a stable suite, so begin with the evidence your common failure classes need.

The retention rule hides one operational detail. Playwright must record the first run before it knows whether that run will pass. When the run passes, the recording is discarded. Stored artifact volume can be low while browser and temporary disk work still happens across every test. This mode saves retained storage, not all recording overhead.

Compare it with nearby choices before standardizing it:

• on-first-retry records the first retry, so it misses the initial failure but captures one recovery attempt.
• on-all-retries records and keeps every retry, so it compares later attempts but still omits the initial run.
• retain-on-failure records every run and keeps every failed attempt, so an all-failing chain can retain several videos.
• retain-on-failure-and-retries keeps failed runs plus all retries, including a retry that passes.

There is no universally best mode. Decide whether you need the trigger, the recovery, every failed stage, or the whole retry story.

Prove the mode with one controlled failure

A deterministic probe prevents a naturally flaky UI from changing the expected artifact set. The following test renders visible attempt information, performs a real browser interaction, fails at retry index zero, and passes at retry index one.

import { test, expect } from '@playwright/test';

test('first-failure retention probe', async ({ page }, testInfo) => {
  await page.setContent(`
    <main>
      <h1>First-run video probe</h1>
      <p data-testid="attempt">Retry index: ${testInfo.retry}</p>
      <button type="button">Check artifact policy</button>
    </main>
  `);

  await page.getByRole('button', { name: 'Check artifact policy' }).click();
  expect(testInfo.retry, 'attempt zero fails and retry one passes').toBe(1);
});

Run this scratch file with one retry. The terminal should show a failed first attempt and a passing retry. The HTML report should classify the test as flaky. Its first result should have the retained video, and the passing retry should not.

npx playwright test tests/first-failure-video.probe.spec.ts \
  --retries=1 \
  --reporter=line,html

find test-results -type f -name '*.webm' -print | sort
npx playwright show-report

Treat the file search as an inventory, not as attempt mapping. Open the report and follow the attachment from the first result. A test may create more than one page, and Playwright associates a video with each recorded page, so raw counts can exceed one video per attempt.

Run a second control with zero retries. The test remains failed and the first-run video remains eligible for retention. This proves that retention is not contingent on a retry being scheduled:

npx playwright test tests/first-failure-video.probe.spec.ts \
  --retries=0 \
  --reporter=line,html

Delete the probe after the configuration check. Its use of testInfo.retry to decide success is intentionally artificial. A production acceptance test should not become valid merely because it is running for the second time.

There are two boundaries in this verification. The Playwright boundary decides which recording remains in test output and attaches it to the result. The CI boundary uploads that output and the HTML report. A video visible before artifact upload but absent after download is a CI packaging problem, not a retention-mode problem.

Do not inspect page.video().path() inside the test and assume the file is complete. Playwright documents that video is fully written when its browser context closes. The test runner handles the lifecycle for its standard context. Let the result attachment and completed report identify the retained file.

Find why the first-failure clip is missing

Begin with the installed version and resolved config. Run the local Playwright CLI from the same package and lockfile as CI. In a monorepo, a root command and a package command may resolve different dependencies. Confirm that the result belongs to the project whose use.video value is retain-on-first-failure.

Then inspect the attempt result. A passed initial run should retain no video. A skipped test does not exercise the page flow. A test that was never collected cannot create an artifact. These are correct empty outcomes, not reporter failures.

Next, locate the failure stage. A configuration parse error occurs before tests. A browser launch error occurs before a recording context exists. A worker-scoped fixture can fail before the built-in page fixture is requested. None of those situations guarantees a page video. The stack and runner log are the appropriate evidence.

Lazy fixtures make this easy to overlook. A test that accepts only request does not create a page merely because video is configured. API-only failures need response bodies, request logs, and attachments rather than an empty browser recording. Do not add an unused page fixture solely to produce black frames.

If the test creates an extra context manually, its lifecycle is separate from the runner's normal page fixture. Close it explicitly so the video flushes:

import { test, expect } from '@playwright/test';

test('closes a manually recorded support flow', async ({ browser }, testInfo) => {
  const context = await browser.newContext({
    recordVideo: { dir: testInfo.outputPath('support-flow-video') },
  });

  try {
    const page = await context.newPage();
    await page.goto('https://example.com');
    await expect(page.getByRole('heading', { name: 'Example Domain' })).toBeVisible();
  } finally {
    await context.close();
  }
});

This code demonstrates the documented context-close requirement. Manual recordVideo does not inherit the test runner's retain-on-first-failure decision. If you create manual contexts, you own their retention and attachment policy. Prefer the built-in fixture unless the test genuinely needs another context.

After runner-side checks, inspect upload paths. The HTML folder and test output are different paths in common configurations. Uploading only a single HTML file can break attachment access. Upload the complete directories from the working directory where Playwright wrote them, and run the artifact step even when the test step failed.

A final near-miss is an empty or unreadable clip after a forced process termination. Video finalization depends on graceful context closure. If the CI system kills the job for an outer timeout, there may be no opportunity to finish writing or upload artifacts. Compare the CI job status and termination log with an ordinary Playwright test timeout. Increasing the test timeout will not repair a job that the CI platform killed.

Use the initial video for the failures it can actually show

One first-run clip is powerful when the error is visual and transient. It is weak when later attempts diverge or the failure lives outside the page.

Worked example: a startup overlay captures the click

The application shows a maintenance notice while a client-side configuration request is still loading. The test finds the "Create workspace" button, but an overlay intercepts the click on the first run. By retry time, caches are warm and the overlay disappears.

The initial video can show the overlay and cursor interaction. The trace should confirm the action call log and DOM state. Network evidence should show whether the configuration request was pending, slow, or failed. A screenshot taken only at the final timeout may show the overlay, but it cannot show that it appeared after the locator first became visible.

The fix depends on product intent. If users must not interact until configuration is ready, the application should expose a clear disabled or loading state and the test should wait on that user-visible contract. If the overlay should never cover the enabled control, fix the UI. Adding a sleep makes the test less likely to collide without proving either behavior.

A web-first assertion can encode an actual readiness signal when the product exposes one:

import { test, expect } from '@playwright/test';

test('creates a workspace after configuration is ready', async ({ page }) => {
  await page.goto('/workspaces');

  const createButton = page.getByRole('button', { name: 'Create workspace' });
  await expect(page.getByTestId('configuration-status')).toHaveText('Ready');
  await expect(createButton).toBeEnabled();
  await createButton.click();

  await expect(page.getByRole('dialog', { name: 'New workspace' })).toBeVisible();
});

The selectors are application integration points, not claims about a Playwright demo. The trade-off is that the UI must expose a stable readiness contract. That is usually healthier than asking automation to infer readiness from the disappearance of every possible overlay.

Worked example: the first run has bad seed data

A test account starts with an expired subscription. Retry setup mutates the account to active, and the retry passes. The first video shows an upgrade page instead of the expected dashboard. That is useful, but it cannot tell whether the backend seeded the wrong plan or the frontend interpreted a correct response badly.

Attach the account identifier and sanitized seed response before navigation. Inspect the dashboard request in the trace. If the API says expired, repair test-data provisioning. If it says active while the page renders expired, route the defect to the UI or caching layer. The clip is evidence of what a user saw, not proof of the upstream cause.

The durable data fix may provision a fresh account per test. That adds setup latency and cleanup responsibility. Reusing a shared account is faster until parallel workers, previous runs, or retries mutate it. Name that trade-off instead of disguising it with another retry.

Second failure mode: a retry breaks somewhere new

The first run times out on login, so its video is retained. Retry one logs in but fails during checkout because cleanup from the discarded worker removed the cart. Under retain-on-first-failure, no retry video exists. The report contains the retry's error and trace if configured, but the visual sequence is missing.

That signature is not an upload failure. The initial result has video, the retry result does not, and the retry error points to a different step. Switch to retain-on-failure if every failed attempt needs video. Choose retain-on-failure-and-retries if a passing recovery is also important.

The broader policy costs more recording retention and review time. Before changing it suite-wide, determine whether divergent retry failures are common. A single complex test may deserve an override or focused diagnostic job while the rest of the suite keeps the leaner first-failure policy.

Near-miss: the failure is a response contract, not a visual state

An API assertion expects 201 and receives 409. A page video, if any, shows no useful change. Attach the response status and a redacted body, then inspect setup ownership and idempotency. Keeping a video because "all failures need artifacts" adds cost without evidence.

Use artifact types according to the failure. Video is good for motion, focus, overlays, popups, navigation, and layout. Trace is stronger for action and network correlation. Structured attachments are stronger for data contracts. Runner logs are stronger before a browser exists.

Worked example: the failure moves into a popup

An account page opens billing in a new tab. The initial run reaches the click, then times out waiting for the popup's heading. The main-page clip looks complete and gives the impression that recording ended before the failure. In fact, each recorded page has its own video object, so the popup may have a separate attachment.

Wait for the popup event before triggering the action, then assert on the returned page:

import { test, expect } from '@playwright/test';

test('opens the billing portal', async ({ page }) => {
  await page.goto('/account');

  const popupPromise = page.waitForEvent('popup');
  await page.getByRole('link', { name: 'Manage billing' }).click();
  const billingPage = await popupPromise;

  await expect(billingPage).toHaveURL(/\/billing(?:\/|$)/);
  await expect(
    billingPage.getByRole('heading', { name: 'Billing details' }),
  ).toBeVisible();
});

The URL and selectors are application-specific integration points. The event pattern is the important part: start waiting before the click so a fast popup cannot race ahead of the listener. In the retained first attempt, review both page attachments. The popup video can reveal an identity-provider error, blank load, or redirect screen that never appears in the opener's clip.

If the report contains only the opener video, check whether the popup was ever created. The trace action and event history are stronger than assuming the second video was lost. A blocked popup, same-tab navigation, or click that never completed produces a different artifact set. If the popup existed but its context was force-killed, inspect the outer job termination and cleanup path.

Multi-page evidence increases retained size even though the policy keeps only one attempt. Count pages as well as failed tests when estimating storage from a trial. Do not advertise "one video per failed test" as a guarantee.

Separate a stalled service from a stalled renderer

An initial run can end with a locator timeout while the video shows a spinner that never clears. That presentation fits at least two different failures. The service request needed by the page may remain pending or return an error. Alternatively, the request may complete with the expected response while client code fails to commit the next UI state. The final assertion text and the retained pixels can be nearly identical, but the first repair belongs to the service or environment and the second belongs to the client.

Read the attempt record as a set. For a fail-then-pass case, the healthy policy output has a failed result at retry index zero with a video attachment, followed by a passing result at retry index one without a video attachment. The test's overall flaky classification is a misleading shortcut because it says nothing about which result owns the clip. A raw file count is also misleading when the first run opened more than one page or when output from an earlier invocation remains in the directory. Follow the attachment from retry index zero and confirm that the visible attempt marker, when a probe supplies one, agrees with the report result.

For the spinner, find the network operation that should cause the transition. A request still pending at failure time or an error response separates the service path from the rendering path. If the expected response completed before the assertion deadline, inspect the following DOM snapshots and console evidence. A completed response with an unchanged spinner and a client exception supports a rendering defect. A completed response containing the wrong domain state supports a data or service defect even if its transport status looked healthy. The status code alone is therefore a potentially misleading value. Healthy transport is not the same as healthy business data.

Collect only the response detail needed to make that decision. A redacted status and bounded body fragment may be enough. Saving full authentication responses or account payloads in every trace increases privacy exposure and artifact size. The diagnostic gain costs review work, storage, and a sanitization contract; if the failure can be classified from an approved request identifier and server log, prefer that narrower handoff.

An existing report consumer may break before Playwright does. Systems built around on-first-retry often look only at the final result or first retry for video. After the policy changes, they may label a correct first-result attachment as missing. Land per-attempt attachment rendering and retry-index labels before changing the producer configuration. Then run the controlled fail-then-pass probe through the full download path, canary known visual flakes, and expand by project only after reviewers can reach attempt zero without searching the artifact tree.

Ownership should follow the separating evidence. The automation framework owner handles the retention mode, attempt labels, and attachment mapping. The application team owns a completed response that never produces the promised UI state. The service or test-data owner owns a pending, failed, or semantically wrong response. The CI owner takes the issue only when the first-result attachment exists before upload and is absent afterward. A useful handoff includes run, shard, project, test identity, retry index, first failing assertion, the attached video's page identity, the relevant request outcome, sanitized data evidence, and inventories from both sides of upload. A clip of a spinner without that attempt context is a symptom, not a routing packet.

This technique does not catch a state transition that completes correctly in the browser but writes the wrong value to a downstream system. The first-run page can look perfect while an asynchronous service records bad data later. Verify that contract with an API, event, or database-facing assertion owned by the suite. Retaining more pixels cannot validate an effect that never appears on screen.

Migrate CI without trading storage for blind spots

Teams commonly move to this mode from on-first-retry. The behavior change is not simply fewer files. Evidence moves from retry one to attempt zero. Validate the new policy on a deterministic probe and a few known flakes before deleting the old artifact path.

A migration from retain-on-failure has a different risk. A fail-fail-pass sequence previously kept videos for both failed attempts; the new policy keeps only the first. Search historical reports for tests whose retries fail at different steps. Those are the cases most likely to lose useful evidence. Keep the broader mode for that project until the underlying instability is repaired, or accept the loss explicitly.

Run old and new policies in separate, equivalent diagnostic jobs if you need to compare artifact value. Do not run the same stateful tests concurrently against shared accounts, because the comparison itself can create flakes. Use isolated data, the same commit and browser projects, and label any storage or runtime figures as observed results from those jobs.

Upload both report and test output after failed runs:

- name: Run Playwright tests
  run: npx playwright test

- name: Show retained first-failure videos
  if: ${{ !cancelled() }}
  run: find test-results -type f -name '*.webm' -print | sort

- name: Upload Playwright report and output
  if: ${{ !cancelled() }}
  uses: actions/upload-artifact@v5
  with:
    name: playwright-first-failures-${{ github.run_attempt }}
    path: |
      playwright-report/
      test-results/
    retention-days: 7

Seven days is an illustrative policy. Choose retention after considering triage delay, storage cost, and the sensitivity of recorded screens. Do not present an unmeasured reduction as a guaranteed saving. Run comparable CI jobs and report observed data if you need a business case.

Watch runtime as well as retained bytes. Stable first runs are recorded temporarily and then discarded. On CPU- or disk-constrained runners, that work can affect test timing even when the final artifact is small. Reduce workers or improve runner capacity if evidence recording exposes resource starvation. Hiding the symptom by raising every timeout makes the suite slower and less diagnostic.

Roll out by project when browser matrices have different value. A critical desktop checkout project may justify video, while an API project or a browser used only for fast smoke coverage may rely on traces and screenshots. Project overrides are clearer than conditionals hidden in fixtures.

Review privacy before broad adoption. First failures are more likely to contain raw input, partially completed forms, error messages, or privileged states. Use synthetic accounts, keep secrets out of the UI, restrict artifact access, and set a retention period. A useful clip is still production data from the test environment.

Define the review workflow. Start with the first error and trace, then open the video if visual sequence matters. Record the root cause and whether the clip contributed. If nobody uses retained videos across a meaningful trial, remove or narrow the policy rather than collecting evidence by habit.

Sharded CI needs unique artifact identity. Each shard has its own test-results tree, and flattening those trees before upload can overwrite files with similar names. Upload per-shard output under a shard-specific artifact name, or preserve directory boundaries when collecting a combined artifact. The HTML or merged report should remain the mapping from a test attempt to its attachment. A folder of detached .webm files is harder to review and easier to misattribute.

Reruns need the same care. Include the CI run attempt in artifact names so a first execution and a manually retried workflow do not appear to be one evidence set. A video from the earlier commit or attempt can look perfectly relevant while describing different application state. Artifact provenance is part of test evidence, not clerical decoration.

When not to keep only the first failure

Do not use this mode when retries routinely fail at different stages. It deliberately cannot show later retry video. Choose a mode that retains failed attempts or all retries.

Avoid it when stable-test recording overhead is unacceptable. The policy discards passing videos after recording; it does not skip recording first runs. on-all-retries avoids recording stable first attempts but also loses the initial failure, so decide which cost matters.

Do not expect video from collection, configuration, browser-launch, or early fixture failures. Build runner and fixture observability for those boundaries. Creating a page solely to satisfy an artifact checklist wastes time and can obscure the actual stage.

Skip video for API-only and non-visual contract tests unless they also exercise a meaningful page. Attach structured responses and logs instead. Evidence should answer the likely debugging question.

Do not treat one initial clip as proof of root cause. It shows rendered behavior. Confirm state, requests, console output, and test data through the trace and approved attachments before assigning ownership.

Finally, do not choose the setting only because it sounds like "keep the first failure." Its exact meaning is narrower: record the first run, retain it if it fails, and ignore video for all later retries. That policy is excellent when attempt zero is the missing story and incomplete when the story changes afterward.