When Playwright shard reports refuse to line up

Four shard jobs finish successfully, but the merge job sees three report families instead of one. One job ran with a different project filter, another reused output from a previous CI attempt, and the filenames are the first useful clue. Treating every changing suffix as random noise hides that distinction.

The hash in Playwright's default blob filename is run-selection metadata. It is not a digest of the ZIP contents, and it is not a universal execution ID. Once that boundary is clear, the CI design becomes much simpler: let Playwright expose accidental selection drift, add your own run identity for storage, and calculate a separate checksum if you need to verify transport.

Understand what the filename is telling you

The blob reporter writes an archive that can later be passed to playwright merge-reports. With no custom path, its filename has the documented shape report-<hash>.zip. A sharded invocation adds the shard number, so the files from one selection can coexist as report-<hash>-1.zip, report-<hash>-2.zip, and so on. That number is padded to the width of the shard total, which keeps this four-shard example single digit but turns a twelve-shard matrix into report-<hash>-01.zip through report-<hash>-12.zip, so any check that globs on the suffix must pad too.

Playwright computes the optional hash from inputs that affect which tests belong to the run. The documented inputs include --grep, --grep-invert, --project, the global testConfig.tag value, and file filters supplied on the command line. Shard position is represented separately in the filename. That is why four shards launched with the same filters can share a hash while still receiving distinct names.

This behavior catches a common CI mistake. Suppose shard 1 executes:

npx playwright test tests/checkout --project=chromium --shard=1/4

and shard 2 executes:

npx playwright test tests/checkout --project=chromium --grep @smoke --shard=2/4

Those commands are not two pieces of one test selection. The second applies an additional filter. Different hashes are useful because a folder containing both archives should make the operator stop and inspect the matrix rather than merge them as if coverage were complete.

The reverse mistake is assuming equal hashes prove equal executions. They do not. Two attempts of the same CI workflow can use identical filter inputs and therefore produce the same default names. If artifacts from attempt 1 remain in storage when attempt 2 starts, name equality cannot tell you which archive is current. The CI run ID, run attempt, commit, environment, and shard must come from your pipeline metadata.

A third mistake is calling this value an integrity hash. Changing one byte inside an archive does not turn the filename into a corruption alarm. Playwright did not name the file from its bytes. If your upload service truncates a ZIP or an operator copies the wrong file over it, calculate and verify a cryptographic digest such as SHA-256. Selection identity and byte integrity answer different questions.

Project tags also deserve care. Playwright documents testConfig.tag as a way to distinguish environments when reports are merged. If staging and production-like runs use the same tests but represent separate release evidence, setting a stable tag such as @staging or @preprod keeps that identity in the report and contributes to the default name. Do not set the tag from a clock or a random value. Playwright can evaluate configuration more than once, and unstable configuration values make comparison needlessly difficult.

Decide whether a different hash is a defect

Start with the command that actually ran, not the YAML you expected to run. CI wrappers, matrix interpolation, package scripts, and environment-specific config can all change the final selection. Record the resolved project list, grep values, file arguments, tag, shard fraction, commit, and CI attempt next to every blob archive.

Here is a small configuration that uses a stable environment tag and chooses the blob reporter only in CI. It rejects malformed tags early instead of silently creating ambiguous report identities.

import { defineConfig } from '@playwright/test';

const environment = process.env.TEST_ENVIRONMENT ?? 'local';
const tag = `@${environment}`;

if (!/^@[a-z0-9-]+$/.test(tag)) {
  throw new Error(`Invalid TEST_ENVIRONMENT: ${environment}`);
}

export default defineConfig({
  tag,
  reporter: process.env.CI
    ? 'blob'
    : [['html', { open: 'never' }]],
});

The cost of this tag is intentional report separation. A staging archive and a preproduction archive now belong to different filename families, even if every test file and project is otherwise identical. That is a benefit when the environments support different decisions. It becomes a nuisance if the environment variable varies only because two teams spell the same target differently. Normalize values at the workflow boundary.

When hashes differ, compare these inputs in this order:

1. Look for a different positional file or directory argument. tests/checkout and tests are different selections even if today they happen to discover the same files.
2. Compare every --project value. A missing browser project is a coverage gap, not a report naming issue.
3. Compare --grep and --grep-invert after shell quoting. An empty matrix value can disappear from one command while a literal pattern reaches another.
4. Read the resolved global tag. Environment tags are supposed to differ across environments.
5. Confirm that all jobs use the same dependency lockfile and Playwright version. Version drift is not documented as a hash input, but it makes merged evidence harder to trust and may change report compatibility.
6. Check the shard denominator. The shard number lives outside the hash, but 1/4 mixed with 2/5 describes an invalid overall partition.

Do not force identical filenames until those comparisons are complete. Renaming two unlike reports can make the folder look tidy while preserving the underlying coverage error. A filename override is an ownership tool, not a repair for inconsistent commands.

A changed hash is expected when a workflow deliberately splits browser projects into independent jobs. If one job runs Chromium and another runs WebKit, --project contributes to the distinction. You can still bring their blob archives into a controlled merge if that combined report is what your release process intends, but preserve project identity and verify that each expected job supplied an archive. The merge command processes the reports it receives; it cannot know that a missing matrix cell was supposed to exist.

Build report ownership into the CI command

For a conventional shard-only run, the default names already include shard numbers and Playwright's official sharding guide says they will not clash after download into one directory. Keeping the defaults gives you a free signal when selection inputs drift. Add a manifest and checksum rather than replacing the name without a reason.

This Bash entry point validates the matrix values, runs exactly one shard, lists the produced archive, writes a checksum, and records the inputs that matter. It is suitable for a Linux CI runner with sha256sum installed.

#!/usr/bin/env bash
set -euo pipefail

: "${SHARD_INDEX:?SHARD_INDEX is required}"
: "${SHARD_TOTAL:?SHARD_TOTAL is required}"
: "${CI_RUN_ID:?CI_RUN_ID is required}"
: "${CI_RUN_ATTEMPT:?CI_RUN_ATTEMPT is required}"
: "${TEST_ENVIRONMENT:?TEST_ENVIRONMENT is required}"

case "${SHARD_INDEX}:${SHARD_TOTAL}" in
  *[!0-9:]*|0:*|*:0) echo "Shard values must be positive integers" >&2; exit 2 ;;
esac

npx playwright test \
  --project=chromium \
  --shard="${SHARD_INDEX}/${SHARD_TOTAL}" \
  tests

mapfile -t reports < <(find blob-report -maxdepth 1 -type f -name '*.zip' -print)
if [[ ${#reports[@]} -ne 1 ]]; then
  echo "Expected one blob archive, found ${#reports[@]}" >&2
  printf '%s\n' "${reports[@]}" >&2
  exit 3
fi

sha256sum "${reports[0]}" > "${reports[0]}.sha256"

node -e '
  const fs = require("node:fs");
  const manifest = {
    runId: process.env.CI_RUN_ID,
    runAttempt: process.env.CI_RUN_ATTEMPT,
    commit: process.env.GIT_COMMIT ?? null,
    environment: process.env.TEST_ENVIRONMENT,
    project: "chromium",
    shard: {
      index: Number(process.env.SHARD_INDEX),
      total: Number(process.env.SHARD_TOTAL)
    }
  };
  const selectionFile =
    `blob-report/selection-shard-${manifest.shard.index}-of-${manifest.shard.total}.json`;
  fs.writeFileSync(selectionFile, JSON.stringify(manifest, null, 2));
'

That script deliberately expects one ZIP. If the reporter configuration produces several archives, change the assertion to match that design and explain why. Do not weaken it to "at least one." Extra files can be stale input, and stale input is precisely what a merge gate should reject.

Some pipelines need names that carry the CI attempt because their artifact service flattens directories, downloads multiple attempts into one location, or cannot retain the manifest beside an automatically named file. In that case, set the full path through PLAYWRIGHT_BLOB_OUTPUT_FILE. Playwright documents that this environment variable takes precedence over the blob reporter's directory and filename options.

The following TypeScript wrapper creates an explicit, per-shard path and passes it to the Playwright child process. Its sanitization prevents CI metadata from turning into path separators. The file still contains the blob report; only the storage name changes.

import { mkdirSync } from 'node:fs';
import { resolve } from 'node:path';
import { spawnSync } from 'node:child_process';

function required(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Missing environment variable ${name}`);
  return value;
}

function safePart(value: string): string {
  const normalized = value.replace(/[^a-zA-Z0-9._-]+/g, '-');
  if (!normalized || normalized === '.' || normalized === '..') {
    throw new Error(`Unsafe path component: ${value}`);
  }
  return normalized;
}

const run = safePart(required('CI_RUN_ID'));
const attempt = safePart(required('CI_RUN_ATTEMPT'));
const shardIndex = Number(required('SHARD_INDEX'));
const shardTotal = Number(required('SHARD_TOTAL'));

if (!Number.isInteger(shardIndex) || !Number.isInteger(shardTotal) ||
    shardIndex < 1 || shardIndex > shardTotal) {
  throw new Error('Invalid shard position');
}

const directory = resolve('blob-report');
mkdirSync(directory, { recursive: true });

const outputFile = resolve(
  directory,
  `report-run-${run}-attempt-${attempt}-shard-${shardIndex}-of-${shardTotal}.zip`,
);

const child = spawnSync(
  'npx',
  [
    'playwright',
    'test',
    '--reporter=blob',
    `--shard=${shardIndex}/${shardTotal}`,
  ],
  {
    stdio: 'inherit',
    env: {
      ...process.env,
      PLAYWRIGHT_BLOB_OUTPUT_FILE: outputFile,
    },
  },
);

process.exit(child.status ?? 1);

An explicit name removes the visible Playwright selection hash. Compensate by retaining the selection manifest and rejecting unexpected matrix combinations. Otherwise, two commands with different grep filters can both write names that look like members of the same run. The override buys storage clarity at the cost of losing a built-in drift clue.

Never give two concurrent processes the same PLAYWRIGHT_BLOB_OUTPUT_FILE. A run ID alone is insufficient when four shards share that run. Include shard index, and include project or environment if those dimensions execute as separate processes. If your CI platform retries a job under the same run ID, include its attempt number too.

Diagnose the merge input before running the merge

A merge failure often gets blamed on the hash because the filenames are visible. First establish whether the problem occurs before, during, or after Playwright reads the files.

Before the merge, inventory the directory. The following command fails on duplicate basenames, verifies saved SHA-256 files, prints each selection manifest, and asks unzip to test each archive structure without extracting it.

#!/usr/bin/env bash
set -euo pipefail

report_dir="all-blob-reports"
test -d "${report_dir}" || { echo "Missing ${report_dir}" >&2; exit 10; }

find "${report_dir}" -type f -name '*.zip' -printf '%f\n' |
  sort |
  uniq -d |
  tee /tmp/duplicate-blob-names.txt

if [[ -s /tmp/duplicate-blob-names.txt ]]; then
  echo "Duplicate blob basenames were downloaded" >&2
  exit 11
fi

while IFS= read -r checksum; do
  (cd "$(dirname "${checksum}")" && sha256sum --check "$(basename "${checksum}")")
done < <(find "${report_dir}" -type f -name '*.sha256' -print)

while IFS= read -r archive; do
  echo "Checking ${archive}"
  unzip -t "${archive}" >/dev/null
done < <(find "${report_dir}" -type f -name '*.zip' -print | sort)

find "${report_dir}" -type f -name 'selection-*.json' -print -exec sed -n '1,120p' {} \;

The /tmp/duplicate-blob-names.txt file belongs only to that diagnostic process and can be replaced with a runner-specific temporary path if several checks share a machine. More importantly, the duplicate check uses basenames because artifact download tools often merge separate directories into one. Two different source paths can become one destination path after flattening.

A checksum failure points to transport or storage, not filter drift. An unzip -t failure indicates an unreadable archive, again not a Playwright hash mismatch. A manifest with shard.total values of 4 and 5 proves matrix drift even when every ZIP is structurally valid. A missing expected shard proves incomplete evidence even if merge-reports successfully renders the remaining files.

A partial workflow retry creates a more subtle input set. As an illustrative case, consider a four-shard attempt in which shards 1, 2, and 4 upload successfully while shard 3 loses its runner. An operator reruns only shard 3 under attempt 2. All four archives can have the same Playwright selection hash because their filter inputs match. Every archive can also pass its checksum. Those facts still do not make attempt-1 shards and an attempt-2 shard one execution.

Sometimes that hybrid is acceptable. If tests are isolated, dependencies are pinned, the commit and environment are unchanged, and the release process explicitly permits a replacement shard, the merge job can record that decision. In stricter pipelines, one failed shard requires a fresh full attempt. Filename logic cannot choose between those policies.

Make the rule executable. Read every selection manifest, group it by run ID and attempt, and compare the group with the expected shard set. If mixed attempts are forbidden, reject more than one attempt value. If replacement shards are allowed, require an approval record that names the superseded shard and preserve both source manifests. The approval is important because a later reviewer cannot infer intent from matching hashes.

This example also explains why modification times are weak. Downloading an attempt-1 archive during attempt 2 can give the old file a newer local timestamp than the replacement. The manifest's pipeline identity is authoritative. The checksum proves which bytes arrived. The Playwright hash shows the test-selection inputs. None of the three fields substitutes for the others.

When a replacement shard is accepted, put the decision in the rendered report's provenance as well as the merge log. An incident reviewer should be able to see that shard 3 came from a later attempt without locating the original CI console. Provenance costs a little metadata; hiding the substitution costs confidence in the entire report.

Only after those gates pass should the workflow invoke:

npx playwright merge-reports --reporter=html ./all-blob-reports

The official command reads all blob reports in the supplied directory. That broad behavior is convenient and dangerous. Pointing it at a cache shared by branches or attempts can silently expand the input set. Create a fresh merge directory for one release decision, download only the named artifacts for that decision, validate the count, then merge.

Trace the source of each file when the folder contains unexpected hashes. Artifact systems can preserve nested job directories, flatten them, or restore old caches. The useful evidence is the uploader job, source artifact name, run attempt, archive checksum, and recorded selection. File modification time is weak evidence because downloads and extraction can rewrite it.

A successful HTML report is not the final diagnostic. Open its project filters and test counts. When projects from separate shards are merged, the Reporter API can expose separate project objects with the same project name. Custom post-processing that assumes project names are unique may collapse those records. If the HTML report looks correct but an internal dashboard loses shards, inspect the dashboard adapter rather than renaming blob files.

Separate a missing shard from a collapsed download

The merge job can print “expected four archives, found three” for two unrelated reasons. The test matrix may have launched shard 2 twice and never launched shard 3. Alternatively, shards 1 through 4 may all have executed and uploaded successfully, while the download or extraction step collapsed two source files into one destination. The final directory has three ZIP files in both cases, so its count alone cannot identify the broken system.

Work backward from producer evidence. Each shard job should publish its run, attempt, project, environment, shard index and total, source archive name, and archive checksum in a small manifest or job summary. The merge job should also inventory the source artifacts it requested and the destination files it received. If the producer set contains indexes 1, 2, 2, and 4, scheduling or matrix interpolation omitted shard 3. If the producer set contains 1, 2, 3, and 4 with four different source checksums, but the destination contains only three archives, the artifact transport or extraction boundary lost one. Rerunning tests is not the first fix for the second case.

A healthy diagnostic view shows four unique shard coordinates for a four-part run, one successful upload record per coordinate, four downloaded archives, matching source and destination checksums, and no unexpected archive. A matrix defect shows a duplicate coordinate before upload. A transport defect shows a complete source coordinate set and an incomplete destination set. A stale-input defect shows more than the expected set or an attempt value that does not belong to the selected release decision.

Several plausible values can mislead. “Four artifacts downloaded” may count CI artifact containers rather than ZIP entries, and one container can hold zero, one, or several reports. Four distinct filenames do not prove four distinct shard coordinates if the name contains a job label instead of the actual shard value. Four valid checksums prove four byte sequences arrived unchanged, but not that one sequence represents each expected partition. The coordinate set from the producer manifests is the coverage evidence; archive structure and checksum answer different questions.

When the source catalog is unavailable, preserve uncertainty. A clean three-file directory cannot tell you whether shard 3 never ran or vanished after upload. Mark the run incomplete and repair observability before assigning the incident to Playwright. Guessing from modification times or filename order creates a confident handoff with no causal support.

Roll out the change without mixing old and new evidence

A naming migration can create exactly the collision it was meant to solve. Roll it out in stages.

First, add the selection manifest and checksum while leaving Playwright's default filenames intact. Run the normal matrix and confirm that every shard has the same selection hash where it should, a distinct shard suffix, one manifest, and one valid checksum. This observation period tells you whether the existing command line is already drifting.

Second, make the merge job build an allow-list from CI metadata. For a four-shard run, require shard indexes 1 through 4 exactly once. If projects execute in separate jobs, require every project and shard pair. Reject extras as well as missing inputs. "At least four" accepts stale fifth files and duplicate shards.

Third, introduce PLAYWRIGHT_BLOB_OUTPUT_FILE only for the job families that need it. Keep the old default path out of the uploader include pattern so a runner workspace cannot publish both formats. During one transition window, log both the expected explicit path and the actual files found. Fail if they disagree.

Fourth, keep attempt-specific artifacts immutable. A retried workflow should write an attempt-2 name rather than replace attempt 1. The merge job should select one attempt deliberately. Preserving both helps explain why an earlier attempt failed, but combining both into a single report would double-count tests and mix two execution histories.

Fifth, document the ownership fields alongside the wrapper. Engineers should know which values define a logical run and which define a file: commit and workflow run identify the change, attempt identifies the retry of that workflow, environment or project identifies a coverage dimension, and shard identifies one partition. The Playwright selection hash remains useful only when the default name is retained.

An established pipeline needs its readers updated before the filename writer changes. Merge scripts, artifact download patterns, retention rules, report links, and cleanup jobs often contain a glob for the default report- shape. Teach the inventory layer to accept both the default and explicit formats, but require one format per producer job. A single job emitting both is an error, not compatibility. Once every consumer resolves reports through the manifest, switch one non-blocking job family to explicit names and confirm that upload, download, validation, merge, report links, and expiry all follow the new path. Expand the cutover only after that full path works.

Do not rewrite old archives into the new naming scheme. Their manifests may lack run-attempt or shard ownership, and a new filename would imply evidence the archive never carried. Keep them under the legacy reader until retention removes them. During cutover, publish counts by naming format and fail when an expected producer supplies neither. This catches an uploader include pattern that still looks only in the old directory.

The rollout adds files and checks. Checksums consume CPU and artifact manifests add small storage and maintenance costs. Matrix validation must be updated whenever the intended project or shard layout changes. Those costs are justified for release evidence that crosses several machines. A small local run with one blob file does not need the same machinery.

The checksum cost is proportional to archive size because validation must read every byte before merge. On a large suite, that adds storage I/O and delays report availability even when no tests rerun. The dual-format window also requires contract coverage for two discovery paths and keeps cleanup logic more complex until legacy artifacts expire. Those are specific operational costs, so keep the compatibility window bounded and measure it with the pipeline's own timings.

Ownership should be explicit at the handoff. The test-platform owner defines the intended selection and shard matrix. The workflow owner supplies run, attempt, project, and environment identity. The artifact-service or CI-platform owner investigates a complete producer set that becomes incomplete after upload. The report owner handles a validated blob set that merges correctly but renders or aggregates incorrectly. Send the exact executed commands, dependency lock revision, producer manifests, source artifact inventory, destination inventory, checksums, archive validation output, expected coordinate set, and the first boundary where the set changes. A screenshot of three ZIP filenames omits the evidence needed to choose an owner.

Know when not to override the hash

Keep Playwright's default naming when you run ordinary shards with identical selection inputs and download them into a clean directory. The documented shard suffix already prevents clashes. The hash provides a valuable visual warning if one job receives a different project, grep, tag, or file filter.

Do not override names to make unlike test selections appear uniform. Browser-specific jobs, smoke and regression jobs, or staging and preproduction jobs may deserve separate report families. Decide whether combining them represents one release claim before changing storage. The merge tool's ability to read several reports is not proof that the combination is meaningful.

Avoid blob reports entirely when no later merge or alternate report generation is needed. For a single CI process, the HTML, JSON, or JUnit reporter may be the direct artifact your consumers require. Blob archives add an intermediate step and are primarily intended to support merged reporting.

Skip long-lived blob retention if the rendered report and required attachments satisfy your audit policy. Blob archives can be large and may contain test output that should not be retained indefinitely. Set retention based on incident and compliance needs, not because the CI vendor offers a generous default.

Do not use the filename as a security boundary. Sanitize user-controlled metadata before putting it into a path, keep untrusted pull-request artifacts separate from trusted release artifacts, and verify checksums only against manifests obtained through a trusted channel. A digest stored beside a maliciously replaced archive can be replaced with it.

Most of all, do not debug a red merge job by repeatedly changing names. Inventory the inputs, compare the actual commands, validate archives, and check the expected matrix. Once those facts agree, the choice between Playwright's generated hash and an explicit CI name is an operational trade-off rather than guesswork.

Complete, well-owned blob reports do not prove that Playwright discovered every test the release policy intended. A conditional skip, an empty file filter, or test registration code that never ran can produce valid archives for every shard and a successful merge. Compare discovered tests and required coverage against an independent suite manifest when omission matters. Report provenance protects the evidence that exists; it cannot reveal a test that never entered the run unless another oracle names that test as expected.