Give every Selenium failure artifact an owner

Two Selenium tests fail at the same time, yet CI publishes one failure.png. The second writer replaced the first, so the screenshot attached to the checkout failure actually belongs to profile editing. Retrying the suite may turn it green, but it cannot reconstruct the evidence that was overwritten.

A reliable artifact name answers ownership without opening the file. It identifies the CI run, execution lane, unique test case, attempt, and browser session. Human-readable titles can help navigation, but none of those mutable labels should be the only barrier between parallel writers.

Choose identities that survive parallel execution

Start with the distinction between a label and an identity. Checkout rejects an expired card is a good label. It is not necessarily unique. The same title can appear under Chromium and Firefox classes, inside different nested suites, or once for every parameter set. Some runners also generate a display name separately from the internal unique ID.

Use the runner's unique test ID as the test component. With JUnit 5 extensions, ExtensionContext.getUniqueId() is the relevant value. Other runners expose their own stable case identity. Pass it into the artifact layer rather than making the artifact layer rebuild identity from class and method strings. Reconstructing it later loses parameter, dynamic-test, or engine details.

A CI run needs its own immutable key. A commit SHA is not enough because the same commit can execute more than once. A branch name is worse because many commits share it and branch text can contain path separators. Build the run key from the CI system's run or build identifier plus its retry or attempt identifier. Record the commit separately in the manifest.

Parallel nodes need a lane identifier. Shard 1 and shard 2 may intentionally execute the same test during a diagnostic rerun or browser matrix. Even if that duplication is accidental, preserving both outputs is better than letting one erase the other. Use the matrix cell, shard index, node index, or another value assigned before the process starts. Do not rely only on a Java thread ID; thread IDs repeat across JVMs and can change when runner scheduling changes.

Retries need an explicit attempt number. The first execution is usually easiest to represent as zero, with later attempts as one, two, and so forth. Keep the convention consistent with the runner. Never infer retries from a timestamp or count how many directories already exist. Two processes racing to count directories can choose the same answer.

The WebDriver session ID links browser-side evidence to the client-side test. Selenium Java's RemoteWebDriver.getSessionId() returns the current SessionId, which can be null when no session exists. Capture it after driver creation and before quit(). If your Grid or cloud provider exposes session logs, the value is often the most direct join key. Treat it as opaque. Do not parse meaning from its characters.

These fields form a useful identity tuple:

run / lane / unique test / attempt / session / artifact kind

The artifact kind is the final filename, such as failure.png, browser.log, or page.html. A test directory can contain several kinds without collision. If the framework captures multiple screenshots within one attempt, add a deterministic step or sequence key assigned by the test, not a clock-only suffix.

A checksum belongs in the metadata, not the identity tuple. SHA-256 over screenshot bytes tells you whether two files have the same content and whether transport changed those bytes. It does not tell you which test owned them. Conversely, a digest of the identity fields can shorten a path, but it says nothing about the artifact's contents.

Build a path that rejects accidental reuse

A practical layout keeps the readable parts short and hashes the long, runner-specific test ID:

artifacts/<run>/<lane>/<test-label>--<identity-digest>/attempt-<n>/session-<session-digest>/failure.png

This structure keeps files from separate CI attempts apart. It also prevents a parameterized test ID from creating an unmanageably long filename. The complete fields still belong in a manifest next to the artifact.

The following Java 17 utility canonicalizes identity fields with length prefixes before hashing them. Length prefixes avoid ambiguous concatenation. For example, fields ab and c must not hash the same canonical string as a and bc merely because a delimiter was missing.

package example.artifacts;

import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.text.Normalizer;
import java.util.HexFormat;
import java.util.List;
import java.util.Objects;

public final class ArtifactPaths {
  private ArtifactPaths() {}

  public record Identity(
      String runId,
      String laneId,
      String testId,
      String displayName,
      int attempt,
      String sessionId
  ) {
    public Identity {
      requireText(runId, "runId");
      requireText(laneId, "laneId");
      requireText(testId, "testId");
      requireText(displayName, "displayName");
      requireText(sessionId, "sessionId");
      if (attempt < 0) throw new IllegalArgumentException("attempt must be >= 0");
    }

    private static void requireText(String value, String name) {
      if (value == null || value.isBlank()) {
        throw new IllegalArgumentException(name + " must not be blank");
      }
    }
  }

  public static Path directory(Path root, Identity identity) {
    Objects.requireNonNull(root, "root");
    Objects.requireNonNull(identity, "identity");

    String canonical = canonical(List.of(
        identity.runId(),
        identity.laneId(),
        identity.testId(),
        Integer.toString(identity.attempt()),
        identity.sessionId()
    ));

    String testDigest = sha256(canonical).substring(0, 32);
    String sessionDigest = sha256(identity.sessionId()).substring(0, 16);
    String testLabel = safePart(identity.displayName(), 48);

    Path base = root.toAbsolutePath().normalize();
    Path target = base
        .resolve(safePart(identity.runId(), 48))
        .resolve(safePart(identity.laneId(), 32))
        .resolve(testLabel + "--" + testDigest)
        .resolve("attempt-" + identity.attempt())
        .resolve("session-" + sessionDigest)
        .normalize();

    if (!target.startsWith(base)) {
      throw new IllegalArgumentException("Artifact path escaped its root");
    }
    return target;
  }

  private static String canonical(List<String> fields) {
    StringBuilder value = new StringBuilder();
    for (String field : fields) {
      value.append(field.length()).append(':').append(field);
    }
    return value.toString();
  }

  private static String safePart(String value, int maximumLength) {
    String ascii = Normalizer.normalize(value, Normalizer.Form.NFKD)
        .replaceAll("\\p{M}+", "");
    String safe = ascii
        .replaceAll("[^a-zA-Z0-9._-]+", "-")
        .replaceAll("^-+|-+$", "");
    if (safe.isBlank() || safe.equals(".") || safe.equals("..")) safe = "unnamed";
    return safe.substring(0, Math.min(safe.length(), maximumLength));
  }

  public static String sha256(String value) {
    try {
      MessageDigest digest = MessageDigest.getInstance("SHA-256");
      byte[] bytes = digest.digest(value.getBytes(StandardCharsets.UTF_8));
      return HexFormat.of().formatHex(bytes);
    } catch (NoSuchAlgorithmException error) {
      throw new IllegalStateException("SHA-256 is unavailable", error);
    }
  }
}

The 32-character test digest in that example is a path-length trade-off. Truncating a cryptographic digest introduces a theoretical collision risk. The code therefore should not silently overwrite an existing file. Use CREATE_NEW for final artifacts. If the same identity attempts to write the same artifact kind twice, the resulting FileAlreadyExistsException exposes either duplicate capture or incomplete identity.

Do not call Files.write(path, bytes) with defaults for failure evidence. Its default behavior can create a file or truncate an existing file. Truncation is convenient for generated build output and dangerous for evidence. Create the directory, then create each artifact once.

This capture method uses Selenium's documented screenshot API, records the full ownership tuple, and writes the screenshot checksum. It accepts RemoteWebDriver because both getSessionId() and getScreenshotAs() are available there. Capture must happen while the driver is still alive.

package example.artifacts;

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
import java.util.Objects;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.remote.SessionId;

public final class FailureCapture {
  private FailureCapture() {}

  public static Path screenshot(
      Path root,
      RemoteWebDriver driver,
      String runId,
      String laneId,
      String testId,
      String displayName,
      int attempt
  ) throws IOException {
    Objects.requireNonNull(driver, "driver");
    SessionId session = Objects.requireNonNull(
        driver.getSessionId(),
        "WebDriver session is not active"
    );

    ArtifactPaths.Identity identity = new ArtifactPaths.Identity(
        runId,
        laneId,
        testId,
        displayName,
        attempt,
        session.toString()
    );

    Path directory = ArtifactPaths.directory(root, identity);
    Files.createDirectories(directory);

    byte[] png = driver.getScreenshotAs(OutputType.BYTES);
    Path screenshot = directory.resolve("failure.png");
    Files.write(
        screenshot,
        png,
        StandardOpenOption.CREATE_NEW,
        StandardOpenOption.WRITE
    );

    String manifest = String.join("\n",
        "runId=" + encode(runId),
        "laneId=" + encode(laneId),
        "testId=" + encode(testId),
        "displayName=" + encode(displayName),
        "attempt=" + attempt,
        "sessionId=" + encode(session.toString()),
        "artifact=failure.png",
        "sha256=" + sha256(png),
        ""
    );

    Files.writeString(
        directory.resolve("manifest.txt"),
        manifest,
        StandardCharsets.UTF_8,
        StandardOpenOption.CREATE_NEW,
        StandardOpenOption.WRITE
    );
    return screenshot;
  }

  private static String encode(String value) {
    return java.util.Base64.getUrlEncoder()
        .withoutPadding()
        .encodeToString(value.getBytes(StandardCharsets.UTF_8));
  }

  private static String sha256(byte[] value) {
    try {
      return HexFormat.of().formatHex(
          MessageDigest.getInstance("SHA-256").digest(value)
      );
    } catch (NoSuchAlgorithmException error) {
      throw new IllegalStateException("SHA-256 is unavailable", error);
    }
  }
}

The manifest uses URL-safe Base64 for text fields so a newline or equals sign inside a display name cannot corrupt the record format. That makes manual reading less pleasant, so a production framework may prefer a JSON library already present in the project. Do not hand-roll JSON escaping just to avoid one dependency.

There is one partial-write edge case. If the PNG succeeds and manifest creation fails, the directory contains an orphaned screenshot. The uploader should flag any artifact directory without a manifest. For stricter atomicity, write both files in a private staging directory and move the completed directory into place on the same filesystem. That adds filesystem complexity, particularly on Windows and network mounts, so make the move behavior part of a focused framework test.

Separate three failures that look like overwrites

The first worked failure is duplicate display names. Imagine two classes both contain can submit order. A flat destination such as artifacts/can-submit-order.png collides even without parallelism. The evidence is deterministic: runner output shows different class or unique IDs, while the artifact log shows the same resolved path. Adding a thread number only hides the duplicate until scheduling changes.

The fix is to include the unique test ID in the identity digest. Keep the display name as a prefix for humans. A reviewer can scan the directory, and the digest keeps two cases separate. The trade-off is discoverability: a digest is not searchable by meaning. The adjacent manifest restores that link.

The second failure is a retry replacing its first attempt. A listener receives testFailed for attempt zero and writes failure.png. The test runner retries the case in a fresh driver session, and the second attempt writes the same path. If it passes, some frameworks capture nothing, leaving attempt zero intact. If it fails again, the original failure disappears. Both behaviors make the evidence policy depend on outcome.

Add the attempt number before artifact kind. Preserve every failed attempt, even when a later one passes. The two screenshots may show different states, which is the evidence needed to diagnose flakiness. Storage grows with retries, so apply retention after classification rather than overwriting during capture.

The third failure occurs after local capture. Every JVM writes a unique tree, but the CI uploader uses only basenames when assembling a downloadable bundle. Separate directories each contain failure.png, and the archive keeps only one of those entries. Local logs show no FileAlreadyExistsException because local ownership was correct.

Inspect the uploaded archive's full entry paths. If local manifests and checksums are all present before upload but entries vanish afterward, the test framework is not the failing boundary. Configure the uploader to preserve directory structure, or create one archive per lane before handing files to it. The archive becomes the unit the artifact service transports.

This Bash diagnostic runs before upload. It accepts one or more artifact roots, fails when the same relative path appears under two of them, checks for orphaned screenshots, prints the count by artifact kind, and appends every root into one archive with the directory hierarchy preserved.

#!/usr/bin/env bash
set -euo pipefail

if [[ $# -gt 0 ]]; then
  artifact_roots=("$@")
else
  artifact_roots=("artifacts")
fi

for root in "${artifact_roots[@]}"; do
  test -d "${root}" || { echo "No artifact directory: ${root}" >&2; exit 20; }
done

duplicates=$(
  for root in "${artifact_roots[@]}"; do
    find "${root}" -type f -printf '%P\n'
  done |
    sort |
    uniq -d
)
if [[ -n "${duplicates}" ]]; then
  echo "Relative paths collide across the supplied roots:" >&2
  printf '%s\n' "${duplicates}" >&2
  exit 21
fi

orphaned=0
while IFS= read -r screenshot; do
  directory=$(dirname "${screenshot}")
  if [[ ! -f "${directory}/manifest.txt" ]]; then
    echo "Screenshot has no manifest: ${screenshot}" >&2
    orphaned=1
  fi
done < <(find "${artifact_roots[@]}" -type f -name 'failure.png' -print)

[[ ${orphaned} -eq 0 ]] || exit 22

find "${artifact_roots[@]}" -type f -printf '%f\n' |
  sort |
  uniq -c |
  sort -k2

rm -f selenium-artifacts.tar selenium-artifacts.tgz
for root in "${artifact_roots[@]}"; do
  tar -C "${root}" -rf selenium-artifacts.tar .
done
gzip -nc selenium-artifacts.tar > selenium-artifacts.tgz
rm -f selenium-artifacts.tar
tar -tzf selenium-artifacts.tgz >/dev/null

The duplicate check only has something to find when the script receives more than one root. GNU find's %P prints each path with its own starting point removed, so a single tree cannot collide with itself: the run and lane segments already make every relative path unique inside one lane, and passing one root leaves that branch unreachable by construction. The check earns its exit code in the aggregation job that extracts several lane tarballs into sibling directories, where a rerun lane or a reused run key produces two trees carrying identical relative paths. Because the packaging loop appends each root with its own prefix stripped, an undetected collision would become two archive entries with the same name, and extraction would keep only the last one.

The find -printf option is available on GNU find, which is normal on Linux CI but not portable to every macOS workstation. Keep this as a CI diagnostic or replace it with a small Java inventory tool if developers must run it across operating systems.

A near-miss can also produce a missing or wrong screenshot without any collision. If capture runs after driver.quit(), getSessionId() may be null and screenshot capture cannot use the ended session. If capture runs before the application reaches the failure state, the file is correctly owned but visually unhelpful. Log the capture start, active session ID, resolved path, byte length, and checksum. Ownership evidence cannot repair lifecycle ordering.

Separate an overwrite from a late capture on a reused driver

A screenshot attached to test A can depict test B even when no file was overwritten. One root cause is the familiar path collision: A writes its bytes, then B opens the same destination and replaces them. A different root cause appears when failure capture is asynchronous and the framework returns a driver to a pool before the screenshot request completes. The path and manifest still name A, but the live browser has already navigated for B. Both incidents present as “the screenshot belongs to another test,” yet only one is a storage problem.

Create-only writing separates the first case. If two writers resolve the same destination, the second create must fail and the log should show the two ownership tuples that produced that path. In the late-capture case, there is one successful create at one unique path. The separating evidence is the browser-session ownership timeline: test failure time, capture request time, session ID observed by capture, driver release time, next test assignment, screenshot completion time, and file creation result. If session ownership changes before screenshot bytes are returned, filename uniqueness cannot make those bytes belong to the earlier test.

A healthy capture record shows one unique test ID, attempt, lane, and active session; capture begins before the driver is released; the final file is created once; and the manifest checksum matches the uploaded entry. An overwrite defect shows more than one attempted owner for the resolved path or a create-only rejection. A late-capture defect shows a single owner and successful write, but the session assignment changes inside the capture interval. A valid PNG signature, nonzero byte length, and unique filename are misleading values here. They prove that an image was saved, not that the browser still displayed A's failure state.

If a framework maintains a driver pool, make its ownership lease observable. Issue an opaque lease token when a session is assigned, pass that token into capture, and record whether it remains current before and after the remote screenshot command. A token invalidated during the call turns the resulting bytes into suspect evidence even when the session ID did not change. Do not use the current URL as a lease substitute because unrelated tests can visit the same route. The lease is local framework evidence and need not become another path component.

Choose where to pay the capture cost. A remote screenshot requires a WebDriver round trip, and a synchronous listener holds the test worker until the bytes arrive and are written. Making capture asynchronous can release the worker sooner, but then the framework must retain exclusive session ownership until all capture futures finish, propagate capture failures, and bound shutdown correctly. Random delays do not establish ownership and only add latency.

Wire the ownership fields through CI

The test process should not guess pipeline identity. Pass it explicitly. This GitHub Actions fragment uses the workflow run and attempt for the run key, the matrix cell for the lane, executes Maven tests, archives the complete tree, and uploads even when tests fail.

name: Selenium tests

on:
  push:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        lane: [chrome-1, chrome-2]
    env:
      CI_RUN_KEY: ${{ github.run_id }}-attempt-${{ github.run_attempt }}
      CI_LANE_ID: ${{ matrix.lane }}
      GIT_COMMIT: ${{ github.sha }}
      ARTIFACT_ROOT: artifacts
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-java@v5
        with:
          distribution: temurin
          java-version: "17"
          cache: maven
      - name: Run lane
        run: ./mvnw -B test
      - name: Validate and package failure evidence
        if: always()
        run: ./ci/package-selenium-artifacts.sh
      - name: Upload lane archive
        if: always()
        uses: actions/upload-artifact@v5
        with:
          name: selenium-${{ env.CI_RUN_KEY }}-${{ matrix.lane }}
          path: selenium-artifacts.tgz
          if-no-files-found: error

The application code should read CI_RUN_KEY and CI_LANE_ID once at startup and fail fast in CI when either is missing. A default such as local is useful on a developer machine, but using it silently on shared runners sends every job into the same root.

The workflow deliberately uploads one tarball per lane. An artifact service now sees unique archive names and does not have to merge many same-named files. The internal paths remain available when the tarball is extracted. The cost is an extra packaging step and less convenient browser-based preview of individual files.

Keep capture and upload responsibilities separate. Selenium knows how to request a screenshot from the active session. The test runner knows the test ID and attempt. CI knows the run and lane. The artifact module combines these fields, while the uploader transports the resulting tree. Making Selenium invent a CI run ID or making YAML infer a WebDriver session reverses ownership.

Do not put secrets in display names, paths, or manifests. Test parameters sometimes include email addresses, account IDs, or tokens. Prefer runner IDs and non-sensitive case labels. If a full unique ID embeds parameter values, hash it for the path and review whether the manifest may retain it under your evidence policy.

Migrate without losing failure history

Begin by instrumenting the current writer. Before changing paths, log the unique test ID, attempt, session ID, lane, and resolved destination for every artifact. Open destinations with CREATE_NEW in a non-blocking observation branch if your release process cannot tolerate immediate failures. Collision logs show which missing dimensions matter in your suite.

Next, introduce the new hierarchy while keeping the old attachment integration pointed at both locations for a short compatibility window. Write only the new path. If a report expects the legacy failure.png, copy or link from the new owner directory during report assembly, not during parallel capture. One serial assembly process can choose a presentation name without risking writer collisions.

Add contract tests around the naming utility. Use two identities that share a display name but differ in test ID, two that share a test ID but differ in attempt, and two that differ only in lane. Assert different directories. Pass the same identity twice and assert the same directory, because deterministic naming should be reproducible. Then verify that writing the same artifact twice raises FileAlreadyExistsException.

Exercise the migration with parallel runner settings enabled. A naming unit test cannot expose a listener that accidentally reads a shared mutable "current test" field. Selenium recommends a fresh driver per test to simplify isolation. Driver ownership and artifact ownership should align: the same test-scoped component that owns the driver should pass its session ID into capture.

Run a forced-failure canary on every matrix lane. Give each canary a distinct marker visible in the page before taking the screenshot. After CI upload, map every manifest to one screenshot and checksum. Remove the canary once the artifact pipeline is proven; do not leave an intentionally failing test in normal release selection.

Change retention only after the new paths are stable. The hierarchy will preserve more evidence than the flat writer did, especially for retries. Measure actual artifact volume from your own runs before selecting a retention period. Do not publish illustrative storage figures as if they came from the suite.

Finally, remove legacy fallback paths and make orphaned files a gate. Long compatibility periods invite consumers to keep depending on the flat path. Announce the manifest format, archive layout, and decoding rules to dashboard and incident tooling owners before removal.

The consumers most likely to break are those that never wrote artifacts. Report adapters may assume a single attachment named failure.png; chat notifications may construct a shallow artifact URL; cleanup jobs may delete only one directory level; and local developer tooling may sort by display name without reading manifests. Inventory those readers before the writer cutover. Give them a manifest-based lookup that can resolve either legacy or owned paths, then switch capture to the new layout. Compatibility should be dual-read and single-write. Two capture locations reintroduce the chance that consumers display different evidence for the same failure.

Mixed framework versions need an explicit archive contract during the cutover. A large suite may launch several JVMs or modules, and one can retain the legacy listener while another writes owned directories. Record the layout version at the lane archive boundary, then dispatch the appropriate reader during aggregation. Reject two writer layouts inside one lane because that usually means duplicate listeners or a partial dependency update. Aggregating legacy lanes beside new lanes can be acceptable for a bounded window if each archive stays separate and the report index preserves which reader resolved it. Once all producers declare the new layout, remove legacy dispatch and make an unknown layout a hard infrastructure failure.

Ownership across teams follows the first broken transition. The runner integration owns unique test IDs, retry numbering, and listener ordering. The Selenium framework owner controls driver lifetime, session ownership, and screenshot capture. The artifact module owns canonical identity, safe paths, create-only writes, and manifests. The CI workflow owns lane identity, packaging, upload, and extraction behavior. The report owner resolves manifests into user-facing attachments. A handoff should include the run and lane, full runner unique ID, attempt, session ID, lifecycle timestamps, resolved path, create result, byte count, checksum, local and uploaded inventories, and the first boundary where ownership or bytes diverge. Sending the screenshot alone hides whether it was wrong at capture or changed later.

Accept the costs, and skip this design when it is unnecessary

Deterministic ownership makes paths longer. Windows path limits, archive tools, and report UIs may struggle with deep trees. Hashing the unique test ID and session keeps components bounded, but a manifest lookup is required to recover the full identity. That is a deliberate trade: filesystem safety over immediate readability.

Create-only writes turn a silent data-loss bug into a test-infrastructure failure. CI may become red when duplicate listeners capture the same artifact. That red build is useful, but teams must classify it separately from an application defect and fix the listener rather than adding a random suffix.

Preserving each retry increases storage and upload time. A flaky case with several failed attempts can carry screenshots, page source, logs, and videos for each one. Retain what helps diagnosis, compress once per lane, and expire according to policy. Do not erase attempt zero merely because the final attempt passed.

Session IDs improve traceability but tie artifacts to browser lifecycle. Setup failures before session creation need a separate pre-session phase identity. Using a literal placeholder is acceptable if run, lane, test, and attempt remain unique; do not pretend a session existed.

Skip this hierarchy for a truly serial, local script that writes one disposable screenshot and has no retries or uploader. A temporary directory supplied by the operating system may be simpler. Add structure when artifacts cross process, machine, retry, or human-review boundaries.

Do not make filenames deterministic when the content itself is a shared mutable product, such as one intentionally cumulative log protected by a single writer. Partitioning that log per test changes its semantics. Use a logging collector with correlation fields instead.

Avoid including every available field "just in case." Browser version, operating system, locale, and application environment can live in the manifest unless they define independent concurrent writers. Every path dimension becomes a contract for cleanup scripts and consumers. Identity should be sufficient, stable, and no larger than necessary.

Deterministic artifact ownership does not catch tests that corrupt one another through shared application state. Two parallel cases can have separate drivers, unique paths, correct session IDs, and perfectly attributed screenshots while both mutate the same account or server-side record. The artifacts will explain which test observed each symptom, but the naming scheme will not prevent the interference. Use isolated data, independent accounts, or explicit state coordination to test that boundary.

When an artifact goes missing, the final question is precise: did capture fail, did a create-only write reject duplicate ownership, did packaging omit a path, or did upload change the archive? With run, lane, test, attempt, session, manifest, and checksum intact, each boundary leaves different evidence. That is the point of the naming scheme.