Stop test evidence from overwriting itself in Playwright

The checkout test fails, and the CI artifact contains one response.json even though the test captured three requests. A retry ran after the failure, then another result exported an attachment with the same label. By the time an engineer opens the evidence, the response that explains the failure has been replaced.

Find the namespace that actually collided

Three names tend to get mixed together in this failure. The first is the label passed to testInfo.attach(). A reporter can show that label to a person. Playwright documents that it also sanitizes the label and uses it as a prefix when saving an attachment, but it does not document the label as a unique key. The second name is the source path where the test creates a file. The third is the destination chosen by a custom reporter, an archive script, or a CI artifact store. Each lives under a different owner.

That distinction changes the diagnosis. TestResult.attachments is an array. Every entry can carry a name, a contentType, and either a path or a body. If two entries both say response, the array can still contain two entries. A home-grown reporter can lose one later by running Object.fromEntries() over that array with attachment.name as the key. JavaScript keeps one value for a repeated object key. Playwright did not collapse the entries in that case. The reporter did.

The source path has a separate boundary. Playwright gives every specific test run an outputDir, and testInfo.outputPath() returns a path inside it. The official guarantee is useful and precise: tests running in parallel do not interfere with one another. It means two tests can each use testInfo.outputPath('response.json') without manually adding a worker number.

That guarantee does not turn outputPath() into a file allocator. Calling it with the same relative segments inside one attempt refers to the same leaf in that attempt's directory. If a loop writes three response bodies to that leaf, normal filesystem behavior applies. A default writeFile() call replaces the existing contents. The report may later contain only the final bytes, even if the test observed three distinct events.

Consider a payment test that records the quote response, the authorization response, and the final order response. A helper named all three files response.json. The test happened to pass most days, so nobody inspected the evidence. On the day authorization returned the wrong currency, the later order request failed too. The helper overwrote the useful authorization body with the final error body. Adding a worker index would not have helped because all three writes came from the same test attempt in the same worker.

The reporter destination is wider again. A custom exporter may copy every attachment to evidence/<attachment-name>. An object-store upload may use only the basename as its object key. A script may unzip every shard into one directory. Those choices discard the test, project, repeat, retry, shard, and run boundaries that existed before the export. Once the hierarchy is flattened, response from one test is indistinguishable from response from another.

Look for the first boundary where two logical records become one physical location. If result.attachments.length is correct but the export contains fewer files, the loss happened after Playwright collected the result. If the attachment already contains the wrong bytes, inspect the source path and the order of writes. If the failed attempt is absent while the retry is present, inspect attempt retention and report merging before changing the in-test filename.

One more timing detail matters. testInfo.attach() is asynchronous. When it receives a path, Playwright copies the file to a location available to reporters. The documentation says the source can be removed safely after the attach call has been awaited. Deleting, renaming, or rewriting the source before that promise finishes creates a missing or inconsistent attachment problem, not a name collision. Await the copy first, then clean up.

This is why a useful incident note names all three values: the human label, the test-owned source path, and the final exported key. A screenshot in chat with only response.json hides the boundary that failed. A manifest showing the attempt and attachment ordinal makes the overwrite reproducible.

Give each captured event its own file identity

A reliable filename describes an artifact event, not merely its media type. response.json, screenshot.png, and log.txt identify formats. They do not identify which response, which screen state, or which slice of the log was captured. Within one attempt, add a controlled semantic kind and a monotonic ordinal. Use the label for human context, and use the path for storage identity.

The ordinal should be owned by the fixture or helper instance for the current test. A module-level counter is unsafe because one worker executes more than one test over its lifetime. A timestamp is also a poor primary key. Two captures can share a coarse timestamp, clock-based names are hard to assert, and sorting them does not necessarily reproduce the business sequence. Random identifiers prevent overwrites, but they make it harder to compare two runs. A small counter plus a controlled kind gives stable order without pretending the test title is a safe path segment.

The following helper writes JSON evidence in create-only mode. It fails on an accidental second write instead of silently replacing the first file. Its test would fail if someone removed the ordinal from artifactFileName(), so the test protects an actual behavior rather than confirming a hard-coded fixture.

import { dirname } from 'node:path';
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import { expect, test, type TestInfo } from '@playwright/test';

type JsonArtifactKind = 'cart' | 'authorization' | 'order';

export function artifactFileName(
  kind: JsonArtifactKind,
  ordinal: number,
): string {
  return `${kind}-${String(ordinal).padStart(3, '0')}.json`;
}

export async function attachJson(
  testInfo: TestInfo,
  kind: JsonArtifactKind,
  ordinal: number,
  value: unknown,
): Promise<string> {
  const target = testInfo.outputPath(
    'evidence',
    artifactFileName(kind, ordinal),
  );

  await mkdir(dirname(target), { recursive: true });
  await writeFile(target, `${JSON.stringify(value, null, 2)}\n`, {
    encoding: 'utf8',
    flag: 'wx',
  });

  await testInfo.attach(`${kind} response [${ordinal}]`, {
    path: target,
    contentType: 'application/json',
  });
  return target;
}

test('keeps every checkout response as a separate artifact', async ({}, testInfo) => {
  const observations = [
    { kind: 'cart' as const, body: { total: 4900, currency: 'INR' } },
    {
      kind: 'authorization' as const,
      body: { status: 'challenge-required' },
    },
    {
      kind: 'authorization' as const,
      body: { status: 'approved' },
    },
    { kind: 'order' as const, body: { orderId: 'order-2048' } },
  ];

  const paths: string[] = [];
  for (const [index, observation] of observations.entries()) {
    paths.push(await attachJson(
      testInfo,
      observation.kind,
      index + 1,
      observation.body,
    ));
  }

  expect(new Set(paths).size).toBe(observations.length);
  const savedBodies = await Promise.all(
    paths.map(async path => JSON.parse(await readFile(path, 'utf8'))),
  );
  expect(savedBodies).toEqual(observations.map(item => item.body));
});

Create-only writes are valuable during a migration because they turn an invisible overwrite into a test failure with a Node filesystem error whose code is EEXIST. Do not copy the full rendered error string into an assertion because wording and absolute paths differ by operating system. Assert the error code in a unit test for the helper, or let the failing write point directly to the duplicate destination.

Screenshots need the same event-level identity. A checkout test may capture before-submit, validation-error, and after-retry. Those stages are controlled vocabulary, not raw page headings. If two screens legitimately share the same stage, the ordinal still separates them. The display names can be longer and friendlier because a manifest retains them without making them responsible for path safety.

Avoid building paths directly from testInfo.title, a URL, an account email, or a localized heading. Besides exposing sensitive data, those strings contain separators and characters that storage systems normalize differently. Playwright documents that attachment names are sanitized when used as filename prefixes, but the exact sanitized spelling is not a durable cross-system contract. A custom exporter may apply a different sanitizer. Keep raw context in structured metadata, and keep storage segments small and controlled.

The cost is more files. Three captures now consume three objects, three manifest entries, and more report space. That cost is intentional when each capture answers a different diagnostic question. If a loop records hundreds of equivalent samples, uniqueness is not the missing design decision. The suite needs sampling, aggregation, or a bounded trace rather than hundreds of individual attachments.

Prove it is a collision before changing retries

Retries make this failure easy to misread. The initial result and each retry are separate TestResult objects. The reporter API exposes result.retry, with zero for the first attempt and increasing values for later attempts. A custom exporter that writes every attempt to checkout/response.json can make a passing retry replace a failed attempt's body. Increasing retries only gives that exporter more opportunities to hide the first failure.

Start with the report result that failed. Count its attachments and record their ordered names, content types, and whether each entry carries a path or a body. Then inspect the retry separately. Do not compare only the final test outcome, because a test that passes after retry is classified differently from a clean first-attempt pass. The attachment question belongs to the individual result.

During rollout, a small custom reporter can enforce a team rule that attachment labels must be unique within one result. Playwright itself does not document this as a requirement. The rule is useful when existing dashboards or people treat the label as an identifier, and it produces a precise diagnostic before any exporter runs.

import type {
  FullResult,
  Reporter,
  TestCase,
  TestResult,
} from '@playwright/test/reporter';

class AttachmentNameAuditReporter implements Reporter {
  private readonly violations: string[] = [];

  onTestEnd(test: TestCase, result: TestResult): void {
    const indexesByName = new Map<string, number[]>();

    result.attachments.forEach((attachment, index) => {
      const indexes = indexesByName.get(attachment.name) ?? [];
      indexes.push(index);
      indexesByName.set(attachment.name, indexes);
    });

    for (const [name, indexes] of indexesByName) {
      if (indexes.length < 2)
        continue;

      this.violations.push(
        `ATTACHMENT_NAME_REUSED test=${JSON.stringify(test.id)} ` +
        `retry=${result.retry} name=${JSON.stringify(name)} ` +
        `indexes=${indexes.join(',')}`,
      );
    }
  }

  async onEnd(_: FullResult): Promise<{ status: 'failed' } | void> {
    if (this.violations.length === 0)
      return;

    for (const violation of this.violations)
      process.stderr.write(`${violation}\n`);

    return { status: 'failed' };
  }
}

export default AttachmentNameAuditReporter;

For two repeated labels at indexes zero and two, that reporter emits a line shaped exactly like ATTACHMENT_NAME_REUSED test="..." retry=0 name="response" indexes=0,2. That is output from this reporter, not a claimed built-in Playwright error. The distinction matters when an engineer searches logs or writes an alert.

Configure the audit alongside the normal reporter. Reporter exceptions are not a dependable CI gate because Playwright documents that errors thrown by custom reporter methods are swallowed. The audit collects violations and returns a failed status from onEnd(), which the reporter contract allows to affect the test runner's exit status.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: process.env.CI
    ? [
        ['blob'],
        ['./reporters/attachment-name-audit.ts'],
      ]
    : [
        ['list'],
        ['html', { open: 'never' }],
      ],
  use: {
    trace: 'retain-on-failure',
  },
});

The audit can prove ambiguous labels, but it cannot prove bytes were overwritten. For that, compare the expected capture count with the source files before attachment, then compare it with result.attachments, then compare it with exported files. A count that drops between source and result points to test or fixture code. A count that drops after onTestEnd() points to reporter or CI export code. Equal counts with repeated content point to a capture bug or a shared source file, not a naming loss.

Trace Viewer adds another useful checkpoint when tracing includes attachments. Open the failed attempt, select the Attachments tab, and compare the listed entries with the custom export. If all expected entries appear there but the downloaded CI folder has one, the test captured them and the flattening happened later. If an expected event is absent from both places, inspect whether execution reached and awaited the corresponding attach() call.

Two near-misses often produce the same “one file left” complaint. First, an assertion can throw before the last attachment call. The missing file was never registered. Put failure evidence in finally or an afterEach hook when it must survive an assertion, while still checking that the page or response object is available. Second, a CI artifact glob can select only playwright-report/** while a custom exporter writes under test-results/**. Nothing was overwritten; the upload omitted the source tree. The result array and job upload log separate these causes.

Keep reporter exports and shard merges collision-safe

The safest custom exporter preserves an attempt directory and treats attachment order as part of identity. testCase.id is useful because Playwright computes it from file, title, and project, but the documented guarantee is only that it is unique within a Playwright session. A durable object key also needs an external run key. Repeat index and retry identify repeated executions of the same case. The attachment ordinal identifies entries whose display names happen to match.

Worker indexes are weak substitutes for those fields. A worker index tells you which process ran an attempt, not which test or attachment the file belongs to. Worker processes can restart after failures, and the same parallel slot can run many tests. A shard number separates machines but does not separate two tests on the same shard. Put scheduler identity in the manifest for diagnosis, not in place of artifact identity.

This reporter exports attachments without trusting their labels as filenames. It hashes the run and attempt identity, uses the attachment's array position as the unique leaf prefix, maps a small set of content types to controlled extensions, and retains the original label in manifest.json. Create-only copies turn an unexpected duplicate attempt key into a visible failure at the end of the run.

import { createHash } from 'node:crypto';
import { constants } from 'node:fs';
import { copyFile, mkdir, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import type {
  FullResult,
  Reporter,
  TestCase,
  TestResult,
} from '@playwright/test/reporter';

const extensionByContentType: Record<string, string> = {
  'application/json': '.json',
  'image/png': '.png',
  'text/plain': '.txt',
  'application/zip': '.zip',
};

function shortHash(...parts: Array<string | number>): string {
  const hash = createHash('sha256');
  for (const part of parts)
    hash.update(`${String(part)}\0`);
  return hash.digest('hex').slice(0, 24);
}

type ExportedAttachment = {
  ordinal: number;
  name: string;
  contentType: string;
  file: string;
};

class EvidenceExporter implements Reporter {
  private readonly failures: string[] = [];
  private readonly pending: Promise<void>[] = [];
  private readonly runKey = process.env.CI_RUN_KEY ?? `local-${process.pid}`;
  private readonly runKeyMissingInCi = Boolean(
    process.env.CI && !process.env.CI_RUN_KEY,
  );

  constructor() {
    if (this.runKeyMissingInCi)
      this.failures.push('CI_RUN_KEY must be set when CI is enabled');
  }

  onTestEnd(test: TestCase, result: TestResult): void {
    if (this.runKeyMissingInCi)
      return;
    this.pending.push(this.exportResult(test, result));
  }

  private async exportResult(test: TestCase, result: TestResult): Promise<void> {
    try {
      const attemptKey = shortHash(
        this.runKey,
        test.id,
        test.repeatEachIndex,
        result.retry,
      );
      const attemptDir = join('evidence-export', attemptKey);

      await mkdir(attemptDir, { recursive: true });
      const manifest: ExportedAttachment[] = [];

      for (const [ordinal, attachment] of result.attachments.entries()) {
        const extension = extensionByContentType[attachment.contentType] ?? '.bin';
        const file = `${String(ordinal).padStart(3, '0')}${extension}`;
        const destination = join(attemptDir, file);

        if (attachment.path) {
          await copyFile(attachment.path, destination, constants.COPYFILE_EXCL);
        } else if (attachment.body) {
          await writeFile(destination, attachment.body, { flag: 'wx' });
        } else {
          continue;
        }

        manifest.push({
          ordinal,
          name: attachment.name,
          contentType: attachment.contentType,
          file,
        });
      }

      await writeFile(
        join(attemptDir, 'manifest.json'),
        `${JSON.stringify({
          runKey: this.runKey,
          testId: test.id,
          repeatEachIndex: test.repeatEachIndex,
          retry: result.retry,
          manifest,
        }, null, 2)}\n`,
        { encoding: 'utf8', flag: 'wx' },
      );
    } catch (error) {
      const message = error instanceof Error ? error.message : String(error);
      this.failures.push(`Evidence export failed for ${test.id}: ${message}`);
    }
  }

  async onEnd(_: FullResult): Promise<{ status: 'failed' } | void> {
    await Promise.all(this.pending);

    if (this.failures.length === 0)
      return;

    for (const failure of this.failures)
      process.stderr.write(`${failure}\n`);
    return { status: 'failed' };
  }
}

export default EvidenceExporter;

The queue in this example is deliberate. The Playwright 1.61 reporter type declares onTestEnd() as a void hook, while onEnd() may return a promise that the runner awaits. Starting each copy in onTestEnd() and awaiting every stored promise in onEnd() keeps the reporter within that contract and prevents the process from exiting with exports still in flight.

Set CI_RUN_KEY to a value unique to the pipeline execution, including a rerun or attempt component. Do not silently fall back to a branch name in CI. Two executions of the same commit on the same branch are still different retention units. The local process fallback in the example is convenient for development, but it is not a promise of cross-process uniqueness.

Most teams do not need to export every attachment themselves. Playwright's blob reporter contains test results and attachments, and its documented shard report names include the shard number. The supported merge flow collects those blob files and runs playwright merge-reports. That keeps result identity available to the merger instead of flattening raw attachment files first.

The following workflow uses separate CI artifact names per shard and gives the final HTML report a run-attempt key. It deliberately enables Corepack before invoking pnpm and does not ask setup-node to call a pnpm binary that may not yet be on PATH.

name: Playwright tests

on:
  pull_request:
  push:
    branches: [main]

env:
  CI_RUN_KEY: ${{ github.run_id }}-${{ github.run_attempt }}

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: lts/*
      - run: corepack enable
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec playwright install --with-deps
      - name: Run shard
        run: >-
          pnpm exec playwright test
          --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
      - name: Upload blob report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 1

  merge:
    if: ${{ !cancelled() }}
    needs: [test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: lts/*
      - run: corepack enable
      - run: pnpm install --frozen-lockfile
      - uses: actions/download-artifact@v5
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: >-
          pnpm exec playwright merge-reports
          --reporter html all-blob-reports
      - uses: actions/upload-artifact@v4
        with:
          name: playwright-report-${{ github.run_id }}-${{ github.run_attempt }}
          path: playwright-report
          retention-days: 14

Do not copy each shard's response.json into a common directory before the blob merge. That discards the very metadata the merge command uses. Also avoid forcing every shard to the same blob outputFile; separate jobs may not share a filesystem, but jobs that do share a mounted workspace can replace one another. The default blob naming already addresses shard clashes when the supported flow is used.

A second environment, such as staging and production-like, is not another shard of one identical run. The sharding documentation recommends distinguishing environments with a configuration tag so merged output retains that context. An external archive should still add the environment to its run-level namespace. Project names alone are not enough across independent sessions because the same project can run in both environments.

Roll the change through an existing suite without losing history

A naming migration can break report links even when it fixes the overwrite. Dashboards may expect error.png. Support tooling may scrape response.json. Developers may have muscle memory around one attachment label. Changing every call in one commit produces a cleaner codebase but can leave the consumers blind on the next failed build.

Begin with observation. Register a report-only variant of the audit reporter, with onEnd() printing violations but not overriding status, and collect repeated names by test ID and retry for a short, bounded rollout. At the same time, search helpers for repeated outputPath() leaf names and custom reporters for maps keyed by attachment.name. These checks find different defects. A duplicate label is an ambiguity signal; a repeated write destination is an overwrite risk.

Next, introduce one artifact helper at the fixture boundary. Give it the controlled kind, ordinal, content type, and optional metadata. Keep the counter inside the test-scoped fixture so it starts fresh for each attempt. Change one evidence family first, such as API payloads, and leave screenshots and logs alone until the report consumer can read the new manifest. This limits the number of moving parts when a result differs.

During the compatibility period, write one canonical file and one manifest entry that records the old logical label. Do not write both response.json and authorization-002.json as separate copies of supposedly identical bytes. Dual writes can diverge if one succeeds and the other fails, recreating uncertainty under two names. Let the consumer resolve the old label through the manifest, then remove that alias after its callers migrate.

An existing after-each screenshot fixture is a common worked example. It may save every failure to testInfo.outputPath('error.png'), attach it as error, and run more than once because separate hooks capture the page and a modal. Move the file creation behind a fixture-owned counter. Preserve error as a manifest tag if a dashboard filters on it, but expose labels such as page at failure [1] and dialog after dismissal [2] to reviewers. The old dashboard keeps finding the category while the files stop competing for one leaf.

Turn on create-only writes before enforcing unique labels. This order catches the destructive case first. A suite can have repeated labels that remain distinct in the reporter array, which may be untidy but not lossy. A suite that writes twice to one path has already destroyed evidence before the reporter sees it. Once the source paths are safe, make the audit reporter fail CI for newly introduced duplicate labels, then fix the backlog by project or fixture owner.

Validate the migration with counts and content, not a green test result. For a test that deliberately captures three phases, assert three distinct source paths, read the bytes back, and confirm the reporter observes three ordered entries. Run it with multiple workers and one retry to exercise the outer boundaries, but do not claim parallelism caused the original same-attempt overwrite. Finally, merge the blob reports and open the failed attempt's Attachments tab. That path checks capture, registration, serialization, merge, and presentation without assuming one layer proves all the others.

Keep the old and new formats long enough to cover the report consumers you actually operate, not an arbitrary number of days. A nightly analytics job and a pull-request dashboard have different observation cycles. Record which consumer has switched to the manifest and who owns removing its compatibility lookup. Otherwise the alias becomes permanent and future engineers cannot tell which name is authoritative.

Retention deserves its own rollout decision. Unique evidence increases the chance that secrets, personal data, or full response bodies survive. Apply redaction before attach(), not after the reporter copied the file. Set CI retention according to the evidence class. A short-lived blob report and a longer-lived sanitized summary can coexist, but they should not share a key or cleanup rule.

Accept the cost, and know when uniqueness is the wrong fix

Preventing overwrites costs storage and attention. A previously flattened result kept one file; the corrected result may keep every capture. Upload time grows with the bytes, and HTML reports become harder to scan when labels differ only by a number. A manifest and a controlled naming helper add code that someone must maintain. These are real costs, so the capture policy should say which events deserve separate evidence.

Use distinct files when later events cannot reconstruct earlier state. API responses around a transaction, screenshots on both sides of a validation step, and logs from separate services fit that rule. Attachments should answer different questions. If five files all answer “what did the page look like after submit?”, keep the most useful one or capture a trace instead of manufacturing five identities.

Do not force uniqueness onto a file that is intentionally assembled before attachment. A logger may append lines to one test-local log throughout the attempt. A HAR writer may finalize one archive when the context closes. A screenshot encoder may create a temporary file and replace it once before anyone registers it. In those cases, one owner mutates one work file, then calls await testInfo.attach() once after finalization. Create-only mode belongs at the published evidence boundary, not necessarily at every temporary write.

Worker-owned output needs a different design too. A proxy launched once per worker may produce a process log covering several tests. Putting it under the first test's outputPath() misstates ownership. Give the worker fixture its own directory keyed by the CI run and worker identity, close and flush it before export, then attach a bounded excerpt or link it through a worker manifest. Do not pretend that adding workerIndex to every test attachment solves that lifecycle.

Snapshot names are another exception. Playwright's snapshot APIs use names to locate expected baselines, and the documentation explicitly says snapshots with the same name in the same test file are expected to be the same. An attachment-event helper should not rewrite snapshot paths or add a new ordinal on every run. Baselines need stable identity across executions; diagnostic attachments need distinct identity within an execution. Mixing those policies creates endless new baselines instead of preventing evidence loss.

Repeated display labels can also be acceptable when no consumer treats them as keys and the surrounding UI makes their order clear. The reporter API preserves attachments as entries in an array. Enforcing unique labels is a team usability policy, not a documented Playwright invariant. If changing a mature label would break integrations, retain it and make the exporter use attempt plus ordinal for physical identity. The bytes stay safe even while the human label remains familiar.

Do not add timestamps, UUIDs, worker IDs, project names, and hashes to every test-local filename “to be safe.” testInfo.outputPath() already supplies the per-test-run isolation documented by Playwright. Extra scheduler data lengthens paths and leaks infrastructure details without fixing same-attempt reuse. Add only the event-level component missing inside that directory. Add run and shard identity later, at the boundary where files leave Playwright's result hierarchy.

Finally, do not keep evidence merely because it can now be named safely. Full authorization responses, cookies, access tokens, customer emails, and unbounded console logs remain liabilities under perfect filenames. Redact before capture, cap noisy collections, and expire raw artifacts as soon as the debugging window closes. Collision safety protects integrity. It does not provide privacy, relevance, or a retention policy.

The practical stop condition is observable: one capture event maps to one source file, one attachment entry, and one manifest record, while the failed attempt remains separate from its retry. Once that chain holds, another random suffix adds complexity rather than confidence.