Make custom Playwright reporters tell the truth about retries

A test fails once, passes on retry, and the dashboard displays one clean pass. The reporter keyed its record by test title, so the second onTestEnd replaced the first. CI is green, but the report has erased the exact attempt that made the test flaky.

A custom reporter is an event consumer, not a final-results callback with a few extra steps. Each test run has its own TestResult, parallel events can interleave, output can occur without an active test, and reporter failures need explicit handling. Define that contract before deciding what the dashboard should show.

Model one test case as several attempt records

Playwright documents a typical reporter sequence. onBegin is called once after discovery. onTestBegin is called for each test run with a TestResult that is still being populated. Step and standard-output events can arrive while that run is active. onTestEnd arrives after the run finishes, when status, errors, attachments, duration, and other result fields are complete. onEnd runs once after the planned tests finish or the run is interrupted.

Retries repeat the run-level part of that sequence. The TestCase remains the logical case, while TestResult.retry distinguishes attempts. Attempt zero can have status failed, and attempt one can have status passed. Playwright classifies that case as flaky. A reporter that stores only the last status changes the meaning.

Use two record levels:

• An attempt record keyed by source run, shard or job, test.id, and result.retry
• A case summary keyed by source run, shard or job, and test.id

The attempt record carries actual status, start time, duration, worker indexes, errors, attachments, and output counts for one execution. The case summary carries test.outcome() and the list of attempts after execution. Keep both. A dashboard can show one row per case while preserving a drill-down into the failed first run.

Titles do not belong in the primary key. Playwright's TestCase.id is computed from file, title, and project and is unique within a Playwright session. A title alone can repeat across files and projects. When separate CI jobs each produce their own reporter file, prefix the ID with pipeline run and shard identity because session-local uniqueness does not identify the source archive globally.

Do not use workerIndex as an attempt key. A worker process can execute many tests. Playwright also replaces a worker after a failure, so a retry commonly has a different worker index. The worker fields are diagnostic attributes, not case identity.

Event order is local, not a promise that the whole suite behaves serially. While test A is running, test B can begin in another worker and finish first. Store an explicit sequence number if consumers need the order in which the reporter observed events. Pair attempts by key instead of relying on adjacent lines.

Standard output needs a nullable owner. The Reporter API allows onStdOut and onStdErr to receive no test and no result when output occurs outside a test execution. A schema that requires testId for every output event will either crash or misattribute global setup output to whichever test happened to run most recently.

Treat output arrays as chunks, not lines. result.stdout and result.stderr can contain strings or buffers, and one application line is not guaranteed to equal one array entry. The example records chunk counts only. A reporter that needs text should decode buffers explicitly, join with no invented separators, cap retained bytes through a documented policy, and keep unowned run output separate.

Global errors are different again. onError represents a problem outside normal test execution, such as an unhandled worker exception. Do not manufacture a fake test row. Emit a run-level error record with its own error data and optional worker information supported by the installed Playwright version.

Make the reporter expose its own failures

Playwright's documentation contains an easy-to-miss warning: errors thrown by custom reporter methods are swallowed. That keeps a presentation bug from crashing the test runner, but it is dangerous when the custom file is required release evidence. A disk-full error can leave tests green and the dashboard empty.

Handle reporter I/O inside the reporter. Collect failures, print them to stderr, and use onEnd to return a failed run status when evidence could not be written. Playwright awaits an asynchronous onEnd and permits a reporter to override the final status. The reporter should not turn an already failed test run into a pass; it only needs to return failed when its own required work failed.

The implementation below writes newline-delimited JSON synchronously. Synchronous writes make event ordering simple and ensure each line reaches the file before the callback returns. They also add latency, which is a real cost discussed later. The file is opened with create-only semantics so two jobs cannot silently share it.

import type {
  FullConfig,
  FullResult,
  Reporter,
  Suite,
  TestCase,
  TestError,
  TestResult,
} from '@playwright/test/reporter';
import {
  closeSync,
  mkdirSync,
  openSync,
  writeSync,
} from 'node:fs';
import path from 'node:path';

type Options = {
  outputFile?: string;
  runKey?: string;
  shardKey?: string;
};

type JsonValue =
  | null
  | boolean
  | number
  | string
  | JsonValue[]
  | { [key: string]: JsonValue };

export default class AttemptReporter implements Reporter {
  private readonly outputFile: string;
  private readonly runKey: string;
  private readonly shardKey: string;
  private readonly problems: string[] = [];
  private readonly cases = new Map<string, TestCase>();
  private fileDescriptor: number | undefined;
  private sequence = 0;

  constructor(options: Options = {}) {
    this.outputFile = path.resolve(
      options.outputFile ?? 'reporter-output/attempts.ndjson',
    );
    this.runKey = options.runKey ?? process.env.CI_RUN_KEY ?? 'local';
    this.shardKey = options.shardKey ?? process.env.CI_SHARD_KEY ?? 'single';
  }

  printsToStdio(): boolean {
    return false;
  }

  onBegin(config: FullConfig, suite: Suite): void {
    try {
      mkdirSync(path.dirname(this.outputFile), { recursive: true });
      this.fileDescriptor = openSync(this.outputFile, 'wx');
    } catch (error) {
      this.remember('open output', error);
    }

    this.emit('run-begin', {
      workers: config.workers,
      discoveredTests: suite.allTests().length,
      projects: config.projects.map(project => project.name),
    });
  }

  onTestBegin(test: TestCase, result: TestResult): void {
    this.emit('attempt-begin', {
      attemptKey: this.attemptKey(test, result),
      testId: test.id,
      titlePath: test.titlePath(),
      project: test.parent.project()?.name ?? null,
      retry: result.retry,
      workerIndex: result.workerIndex,
      parallelIndex: result.parallelIndex,
      startTime: result.startTime.toISOString(),
    });
  }

  onTestEnd(test: TestCase, result: TestResult): void {
    this.cases.set(test.id, test);

    this.emit('attempt-end', {
      attemptKey: this.attemptKey(test, result),
      testId: test.id,
      titlePath: test.titlePath(),
      project: test.parent.project()?.name ?? null,
      retry: result.retry,
      status: result.status,
      expectedStatus: test.expectedStatus,
      durationMs: result.duration,
      workerIndex: result.workerIndex,
      parallelIndex: result.parallelIndex,
      errors: result.errors.map(error => this.errorValue(error)),
      attachments: result.attachments.map(attachment => ({
        name: attachment.name,
        contentType: attachment.contentType,
        path: attachment.path ?? null,
        bodyBytes: attachment.body?.byteLength ?? null,
      })),
      stdoutChunks: result.stdout.length,
      stderrChunks: result.stderr.length,
    });
  }

  onError(error: TestError): void {
    this.emit('run-error', {
      error: this.errorValue(error),
    });
  }

  onEnd(result: FullResult): { status?: FullResult['status'] } {
    for (const test of this.cases.values()) {
      this.emit('case-end', {
        testId: test.id,
        titlePath: test.titlePath(),
        project: test.parent.project()?.name ?? null,
        outcome: test.outcome(),
        attempts: test.results.map(attempt => ({
          retry: attempt.retry,
          status: attempt.status,
          workerIndex: attempt.workerIndex,
        })),
      });
    }

    this.emit('run-end', {
      status: result.status,
      durationMs: result.duration,
      startTime: result.startTime.toISOString(),
      reporterProblems: this.problems.length,
    });

    if (this.fileDescriptor !== undefined) {
      try {
        closeSync(this.fileDescriptor);
      } catch (error) {
        this.remember('close output', error);
      } finally {
        this.fileDescriptor = undefined;
      }
    }

    if (this.problems.length > 0) {
      console.error('Required reporter output failed:');
      for (const problem of this.problems) console.error(`  - ${problem}`);
      return { status: 'failed' };
    }
    return {};
  }

  private attemptKey(test: TestCase, result: TestResult): string {
    return [this.runKey, this.shardKey, test.id, result.retry].join('/');
  }

  private emit(kind: string, payload: JsonValue): void {
    if (this.fileDescriptor === undefined) {
      this.problems.push(`write ${kind}: output is not open`);
      return;
    }

    const record = {
      schemaVersion: 1,
      sequence: this.sequence++,
      runKey: this.runKey,
      shardKey: this.shardKey,
      kind,
      payload,
    };

    try {
      writeSync(this.fileDescriptor, JSON.stringify(record) + '\n');
    } catch (error) {
      this.remember(`write ${kind}`, error);
    }
  }

  private errorValue(error: TestError): JsonValue {
    return {
      message: error.message ?? null,
      stack: error.stack ?? null,
      value: error.value ?? null,
      location: error.location
        ? {
            file: error.location.file,
            line: error.location.line,
            column: error.location.column,
          }
        : null,
    };
  }

  private remember(operation: string, error: unknown): void {
    const message = error instanceof Error ? error.message : String(error);
    this.problems.push(`${operation}: ${message}`);
  }
}

The reporter stores attachment metadata, not attachment bodies. That is deliberate. A body can be large, and a path may refer to a file that needs separate upload. If the downstream system requires attachment bytes, build a bounded copy stage with explicit retention rather than embedding arbitrary buffers into every event line.

The implementation also keeps errors as structured fields. Error messages and stacks can contain newlines, but JSON encoding keeps one physical NDJSON line per event. Consumers should parse JSON, not split fields with regular expressions.

There is an edge case in the failure path. If opening the file fails, each attempted emit adds another problem. That can be noisy for a large suite, though memory still grows with event count. A production reporter can cap repeated messages after the first open failure. Keep at least the first cause and return a failed status.

Prove the contract with controlled test runs

A reporter deserves tests independent of the application suite. The most useful contract run produces a clean pass, a deterministic flake, a hard failure, an attachment, and output outside a test. You do not need unstable timing to create a retry. Use testInfo.retry to make attempt zero fail and attempt one pass.

This spec contains a normal pass and an intentional retry. Run it only in the reporter contract job with one retry enabled.

import { test, expect } from '@playwright/test';

test('reporter records a first-run pass', async ({}, testInfo) => {
  await testInfo.attach('contract-data', {
    body: Buffer.from(JSON.stringify({ source: 'reporter-contract' })),
    contentType: 'application/json',
  });
  expect(testInfo.retry).toBe(0);
});

test('reporter preserves failure before retry pass', async ({}, testInfo) => {
  console.log(`contract attempt ${testInfo.retry}`);

  // Attempt 0 fails. Attempt 1 passes when the job enables one retry.
  expect(testInfo.retry).toBe(1);
});

The second test is intentionally flaky by construction. Never add it to ordinary release selection. Its purpose is to verify that the reporter emits two attempt-begin and two attempt-end records for the same test ID, with retry values zero and one, followed by a case-end record whose outcome is flaky.

Validate the file as data, not by glancing at a dashboard. The following TypeScript program checks monotonically increasing sequence numbers, one end record per attempt key, and the expected failed-then-passed retry shape. It also rejects malformed or blank lines.

import { readFileSync } from 'node:fs';

type RecordLine = {
  schemaVersion: number;
  sequence: number;
  kind: string;
  payload: {
    attemptKey?: string;
    testId?: string;
    retry?: number;
    status?: string;
    outcome?: string;
  };
};

const file = process.argv[2];
if (!file) throw new Error('Usage: validate-reporter.ts <attempts.ndjson>');

const text = readFileSync(file, 'utf8');
const rawLines = text.split('\n');
if (rawLines.at(-1) !== '') {
  throw new Error('Reporter file is missing its final newline');
}
rawLines.pop();
if (rawLines.some(line => line.trim() === '')) {
  throw new Error('Reporter file contains a blank record');
}

const records = rawLines.map((line, index) => {
  try {
    return JSON.parse(line) as RecordLine;
  } catch (error) {
    throw new Error(`Line ${index + 1} is not valid JSON`, { cause: error });
  }
});

records.forEach((record, index) => {
  if (record.schemaVersion !== 1) {
    throw new Error(`Unsupported schema at line ${index + 1}`);
  }
  if (record.sequence !== index) {
    throw new Error(`Sequence gap at line ${index + 1}`);
  }
});

const attemptEnds = records.filter(record => record.kind === 'attempt-end');
const byKey = new Map<string, RecordLine>();
for (const record of attemptEnds) {
  const key = record.payload.attemptKey;
  if (!key) throw new Error('Attempt end has no key');
  if (byKey.has(key)) throw new Error(`Duplicate attempt end: ${key}`);
  byKey.set(key, record);
}

const byTest = new Map<string, RecordLine[]>();
for (const record of attemptEnds) {
  const testId = record.payload.testId ?? 'missing';
  const attempts = byTest.get(testId) ?? [];
  attempts.push(record);
  byTest.set(testId, attempts);
}
const retried = [...byTest.values()].find(attempts =>
  attempts.length === 2 &&
  attempts.some(item => item.payload.retry === 0 && item.payload.status === 'failed') &&
  attempts.some(item => item.payload.retry === 1 && item.payload.status === 'passed')
);
if (!retried) {
  throw new Error('No failed-then-passed test was preserved');
}

const flakySummary = records.find(record =>
  record.kind === 'case-end' && record.payload.outcome === 'flaky'
);
if (!flakySummary) throw new Error('No flaky case summary was emitted');

console.log(`Validated ${attemptEnds.length} attempt records`);

The validator uses only ordinary Map operations, so it does not depend on a newly introduced grouping helper. Its Array.at() call still requires a supported Node runtime. Pin the repository's runtime in CI and run the validator under the same version used for Playwright.

A hard-failure contract can be a separate command expected to exit nonzero. Keep it separate from the flaky contract so the workflow can assert each exit behavior. For interrupted runs, send the signal through the CI harness and verify that onEnd receives an interrupted status; do not make every pull request job test process signals.

Distinguish application failures from reporter failures

When the NDJSON file stops mid-run, look for the last complete record. If attempt-begin exists without attempt-end and the full run was interrupted, that may represent an unfinished test. If Playwright completed normally but the file lacks both attempt-end and run-end, inspect reporter I/O and process termination.

A test assertion failure belongs in result.errors on its attempt-end record. A global setup or worker error can arrive through onError. A reporter write failure appears in the reporter's own stderr message and forces the status returned by onEnd to failed. These categories should not share one generic "automation error" label.

If the custom reporter file is complete but the dashboard is wrong, replay the NDJSON into a local consumer. A row keyed by test ID alone will collapse retries even though the source data is correct. A row keyed by title can collapse separate projects. Fix the consumer's data model before changing the producer.

Blob-report merging creates another near-miss. Playwright invokes the same Reporter API when merge-reports generates a custom output. Official documentation notes that projects from separate shards remain separate project objects, including several objects with the same project name. Code that stores one project object per name can lose shard-specific data. Treat project name as a label and case identity as the key.

Reporter options can also collide. Four shard jobs using the default reporter-output/attempts.ndjson in one shared workspace will race at open time. The create-only flag makes the problem visible. Supply one source-specific output path per job or use the blob reporter on shards and run the custom reporter once during merge.

Do not diagnose a missing attachment by counting only attachment.path values. Attachments can carry an in-memory body instead of a path. The reporter above records bodyBytes for that case. A path attachment and a body attachment are both valid, but the transport step differs.

Separate producer loss from consumer collapse

The most deceptive second failure mode happens after the reporter has done its job. A dashboard can show one clean pass because the reporter failed to preserve attempt zero, or because a downstream upsert replaced attempt zero with retry one. The user-facing row looks the same in both incidents. Changing reporter callbacks fixes only the first.

Use the raw NDJSON file as the boundary. For a recovered failure, a healthy producer contains two attempt-end records with the same runKey, shardKey, and testId. Their attemptKey values differ because one has retry zero and status failed, while the other has retry one and status passed. A later case-end for that test has outcome flaky. Other workers can place records between those lines, so adjacency is not a health condition. The explicit identity fields are.

Read sequence as the reporter's observation order. In a complete file produced by this implementation, visible values increase one at a time from zero. The reporter allocates a sequence value before calling writeSync, so a failed write followed by a successful one leaves a visible gap. That gap proves at least one constructed record did not reach the file. The reverse is not true. If opening the file failed, or every later write failed, there may be no subsequent record to expose a gap. A gap-free prefix can still be incomplete.

The case-end value can also mislead. An outcome of flaky proves the Playwright TestCase knew about a recovered failure when that callback ran. It does not prove the failed attempt's stack, attachments, or output record reached durable storage. Require the two matching attempt-end records and inspect the first one's errors array. Conversely, a final attempt whose status is passed is not a case-level clean pass. Status belongs to one TestResult; outcome belongs to the case.

Reconcile the producer and consumer as sets of attemptKey values. A healthy ingestion has the same distinct keys on both sides after the file is processed. If a retry key is absent from the source set, the producer never supplied it. If it exists in the source set but not in storage, the loss is downstream. Compare distinct keys as well as row counts: duplicate delivery can increase rows without adding an attempt, while replacement can keep the count plausible as the set changes. Replay the immutable file twice as a contract check. A correct consumer leaves the stored set unchanged on the second pass.

When the raw file itself has only retry one, examine producer stderr and the file ending. A caught reporter failure prints Required reporter output failed: followed by the operation and error text, and onEnd returns a failed status. A CI wrapper may still hide that process result, so preserve the test step's raw status. If the reporter reached its ordinary ending, the file should also have a complete run-end record and final newline.

An interrupted process creates a nearly identical truncated file for a different reason. A forceful runner termination may prevent onEnd from running at all. In that case there is no controlled reporter failure message and no reliable run-end; the CI job's cancellation, timeout, container exit, or signal evidence becomes decisive. Do not add retry records in a repair pass to make the file look complete. They were never observed by the reporter.

One field deserves extra caution in this implementation. The status inside run-end is the FullResult value passed into onEnd. If a reporter problem is discovered and the callback then returns { status: 'failed' }, the process can fail even though that payload contains the runner's earlier status. A close failure can also be recorded after run-end was emitted. Reconcile the raw record with reporter stderr and the actual process result instead of treating one JSON value as the final authority.

This evidence gives the incident a clean split. Missing raw attempt records belong to the reporter producer or process lifecycle. Complete raw records with a wrong projection belong to ingestion or dashboard code. Complete dashboard data with missing attachment bytes belongs to artifact transport, which the event file does not validate.

Roll out the reporter as a versioned data producer

Configure the custom reporter beside a terminal reporter first. Its printsToStdio() method returns false because it does not provide human progress output. A line or dot reporter keeps CI readable while the NDJSON remains machine-oriented.

import { defineConfig } from '@playwright/test';

const runKey = process.env.CI_RUN_KEY ?? 'local';
const shardKey = process.env.CI_SHARD_KEY ?? 'single';
const outputFile =
  process.env.ATTEMPT_REPORT_FILE ??
  `reporter-output/${runKey}/${shardKey}/attempts.ndjson`;

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  reporter: [
    ['line'],
    [
      './reporters/attempt-reporter.ts',
      { outputFile, runKey, shardKey },
    ],
  ],
});

Start with a shadow dashboard. Compare counts of cases, attempts, failures, flakes, skipped tests, and global errors against Playwright's built-in report for the same run. Investigate every difference. Do not switch the release gate because a handful of happy-path tests look correct.

Land the contract in dependency order. First commit representative raw fixtures and the validator, including a first-run pass, a failed-then-passed case, a hard failure, a skipped case, a run-level error, and both path and body attachments. These fixtures freeze the meaning of existing fields without requiring the application suite to fail on demand. Keep expected attempt counts separate from expected case counts.

Next, run the producer beside built-in reporters and archive its output without feeding release decisions. Then teach ingestion to accept the schema version, store attempts by full attempt key, and tolerate replay. Only after that should the dashboard read the new case projection. Switching the producer and consumer together removes the raw comparison point and makes a missing row impossible to assign.

The first break is usually cardinality. A consumer built for one row per discovered test receives one row per attempt plus run and case records. Alerts may fire twice, duration totals may add attempts to case summaries, and a storage uniqueness rule may reject retry one. Treat those failures as schema incompatibility. Do not suppress duplicates until the team has established whether the records are duplicate delivery or separate attempts.

After the shadow comparison is clean, enable one non-release consumer and exercise replay, reporter write failure, and intentional flake cases. Move the release gate last. The gate should fail if required reporter output is unavailable, while the built-in report and raw file remain uploadable for diagnosis. The change is working when the case and attempt invariants match across ordinary, retried, merged, and interrupted contract runs, not when the dashboard merely renders without an error.

Version the schema. The example emits schemaVersion: 1 on every line. Consumers should reject unknown major versions rather than guessing. Add optional fields compatibly, and coordinate any key or status change with dashboards before deployment.

Keep the raw event file immutable. Derived dashboards can be rebuilt when grouping logic changes. If a consumer mutates the only copy into one row per case, the attempt-level history is gone.

A CI contract job can run the intentional flaky spec, validate the event file, and upload it even when validation fails:

name: Reporter contract

on:
  pull_request:
    paths:
      - "reporters/**"
      - "tests/reporter-contract.spec.ts"
      - "scripts/validate-reporter.ts"
      - "playwright.config.ts"

jobs:
  contract:
    runs-on: ubuntu-latest
    env:
      CI_RUN_KEY: contract-${{ github.run_id }}-${{ github.run_attempt }}
      CI_SHARD_KEY: single
      ATTEMPT_REPORT_FILE: reporter-output/contract/attempts.ndjson
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v6
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Produce pass and flaky attempts
        run: npx playwright test tests/reporter-contract.spec.ts --retries=1
      - name: Validate reporter schema and retry history
        run: npx tsx scripts/validate-reporter.ts "$ATTEMPT_REPORT_FILE"
      - name: Upload raw reporter output
        if: always()
        uses: actions/upload-artifact@v5
        with:
          name: reporter-contract-${{ github.run_id }}-${{ github.run_attempt }}
          path: reporter-output
          if-no-files-found: error

The tsx command assumes the repository already declares that tool. If it does not, compile the validator with the project's existing TypeScript workflow or implement it in plain JavaScript. Do not install an undeclared tool dynamically in a release job.

Give each boundary an explicit owner

The reporter maintainer owns callback handling, event identity, schema versioning, serialization, and the status override on required-output failure. The data-platform owner owns idempotent ingestion, retention, indexes, and the distinction between attempt facts and case projections. The dashboard owner owns grouping and labels. The CI platform owner owns source keys, artifact upload, process status propagation, and termination evidence. The application or test team owns the failure captured inside result.errors, not the transport that carries it.

A handoff from the reporter team should include the raw file, schema version, run and shard keys, the affected testId, every attemptKey, relevant sequence values, and the reporter's stderr. Add the validator result, process status, and whether the file ended with a newline. For a consumer defect, include the storage key or projection rule and the expected case and attempt cardinalities. For a missing attachment, include its name, content type, whether it used path or body, and the separate artifact inventory.

The richer contract costs more than disk space. Ingestion must support replay without duplication, queries must choose deliberately between attempts and cases, and schema fixtures must be maintained whenever fields change. Stacks and captured output can also carry secrets, so retention and access controls apply to the raw producer file, not only to the polished dashboard. Those are continuing data-product obligations, not one-time reporter code.

This contract does not prove that attachment bytes survived upload. It records attachment metadata available at onTestEnd; a later CI step can still omit the referenced file. It also does not determine whether a failed attempt came from the product, the test, or the runner. Preserve the event faithfully, then use trace, service, and infrastructure evidence to assign that separate failure.

Keep the reporter small when another format is enough

Synchronous writing blocks the reporter process for every event. On fast suites with many tests or slow network-mounted storage, that latency can matter. Write to a local disk and upload afterward. If measurement from your own run shows the cost is unacceptable, batch records with a bounded queue and flush in onEnd, accepting the risk of losing buffered data on abrupt termination.

Attempt-level output increases storage. Retries, attachments, stacks, and stdout counts all add records. Avoid copying large bodies by default, cap error and output payloads only through a documented policy, and retain raw files for the period your incident process actually uses.

Returning failed status for reporter I/O makes evidence availability part of CI. That is appropriate for compliance or release gating. It is excessive for an optional console decoration. Decide whether the reporter is required, and encode that choice rather than allowing swallowed errors to decide accidentally.

Do not build a custom reporter when JSON, JUnit, HTML, or blob output already satisfies the consumer. Built-in reporters carry less maintenance risk. A custom event schema is justified when your organization needs attempt-level fields, streaming integration, or a stable contract that built-in output does not provide.

Avoid using reporter callbacks to control application state or clean test data. Reporters observe execution; they are a poor home for test teardown. A reporter can run again while processing merged blob reports, where the original browser and application resources no longer exist.

Do not emit wall-clock timestamps merely to make keys unique. TestResult.startTime is useful diagnostic data, but source run, test ID, and retry already express ownership. Clock values complicate deterministic replay and can collide across processes.

Finally, do not call a case passed because its last TestResult.status is passed. Preserve each attempt, calculate the case outcome at the case level, and show the failed first run. The event contract exists to keep that claim honest.