Stop losing cleanup failures inside AggregateError

Your cleanup helper closes three resources, two closes fail, and the report shows only AggregateError: cleanup failed. The test is red, but the message does not tell you whether the database lease, mock server, or temporary account survived. On Playwright versions before 1.61, that loss is expected when you inspect testInfo.errors. Version 1.61 added separate child entries, and 1.62 retains that behavior, provided the aggregate reaches Playwright instead of being swallowed by your code.

Understand what 1.62 inherited from 1.61

JavaScript's AggregateError represents several errors under one top-level error. Its errors property holds the values supplied to the constructor, while its own message describes the group. It is useful when cleanup attempts are independent enough that one failure should not prevent the remaining attempts.

Playwright already exposed an array named testInfo.errors, but the matching names hid an important difference. Before version 1.61, throwing one AggregateError produced one serialized entry for the outer error. The JavaScript object had child errors in memory, yet the public TestInfoError available to hooks and reports did not preserve those children as its own errors property.

The official release notes place the change under Playwright 1.61, not 1.62. That distinction matters for a version gate: 1.61 is the first supported release for this behavior. The v1.62 runner keeps it unchanged. The outer aggregate is recorded, and each contained error is also listed as a separate TestInfoError. A two-child aggregate therefore produces an outer entry plus entries for the two children.

The tagged v1.62 worker source makes the traversal precise. It serializes and appends the root first. It then reads the thrown value's errors property, and when that property is an array, visits each entry recursively in array order. A nested aggregate such as outer, inner, children a and b, then sibling c currently appears as outer, inner, a, b, c. That depth-first order is useful while diagnosing a known fixture, but the public release note promises separate AggregateError sub-errors, not a parent id, depth, or stable grouping contract.

The implementation check also prevents an overstatement. Version 1.62 does not test instanceof AggregateError before following an errors array. A plain Error with a custom array property is flattened by the current source, and non-Error children are serialized through their value field. That is implementation behavior, not an invitation to manufacture AggregateError look-alikes. Application code should throw the standard JavaScript class, and reporters should support both message and value without depending on the broader traversal.

Two singular properties still behave as singular properties. testInfo.error is the first entry in testInfo.errors, and testResult.error is the first entry in testResult.errors. Existing code that logs only either singular value continues to show the outer aggregate and miss the child details. Upgrading the runner without updating the reporter can therefore leave the visible output unchanged.

Do not look for testInfo.errors[0].errors. The public TestInfoError shape includes fields such as message, stack, value, cause, and, on recent versions, additional error context. It does not expose the original AggregateError object. The flattening occurs in the surrounding array. This is a serialization boundary, so checks based on instanceof AggregateError cannot run against the serialized entries in a reporter.

Order is useful for a human reading a current report, but it is a poor data model. The public release note promises separate sub-error entries; it does not give each child an aggregate identifier or child index. A reporter should retain every entry and its original array position, but it should not invent parent-child relationships from adjacent messages. If downstream analytics require a durable resource label, put that label in each child error when you create it.

Version skew is the first thing to rule out when local and CI reports differ. A developer may run Playwright 1.62 through a workspace root while CI executes a package pinned to 1.60 or earlier. The same test then produces different error-array shapes. Record the resolved @playwright/test version in job output, and run the CLI through the same package manager context as the suite. A globally available playwright command is not evidence about the dependency that imported test.

Create aggregates that preserve real cleanup failures

An aggregate is only useful if the cleanup code actually observes every operation. Promise.all() rejects when one input rejects, so it is the wrong collection primitive when your goal is to inspect all independent cleanup outcomes. The other operations keep running, but the rejected promise returned by Promise.all() gives you only the first rejection. Promise.allSettled() waits for every input and reports each fulfilled or rejected result.

The following helper labels each failure with the resource name and keeps the original rejection as its cause. A change that makes any resource's close() reject will make the helper throw. One failure is thrown directly, while two or more become aggregate children. A completely successful cleanup returns normally.

type Closeable = {
  name: string;
  close(): Promise<void>;
};

function asError(reason: unknown): Error {
  return reason instanceof Error ? reason : new Error(String(reason));
}

export async function closeAll(resources: readonly Closeable[]): Promise<void> {
  const results = await Promise.allSettled(
    resources.map((resource) => Promise.resolve().then(() => resource.close())),
  );

  const failures: Error[] = [];
  for (const [index, result] of results.entries()) {
    if (result.status === 'rejected') {
      failures.push(
        new Error(`Failed to close ${resources[index].name}`, {
          cause: asError(result.reason),
        }),
      );
    }
  }

  if (failures.length === 1) {
    throw failures[0];
  }
  if (failures.length > 1) {
    throw new AggregateError(failures, 'One or more resources failed to close');
  }
}

This helper is appropriate for resources that may close concurrently. A browser context and a temporary HTTP server might be independent. A transaction, database connection, and SSH tunnel often are not. Closing them in parallel can replace the original failure with a race, such as the tunnel disappearing while the database client is still flushing logs.

Consider a checkout test that acquires an account lease, starts a payment stub, and opens a database connection for verification. Releasing the account and stopping the stub can usually proceed independently after the test. Rolling back a transaction and closing its database connection cannot. Put the independent operations through closeAll, but keep the transaction and connection in the ordered sequence. One large aggregate over all four operations would hide the safety relationship even if its messages were excellent.

Resource labels should survive refactors and retries. Failed to close server becomes ambiguous as soon as a second stub appears. A label such as payment-stub:checkout-482 tells an operator what to inspect without embedding a password, token, or full connection string. Use a run-scoped id that appears in the resource provider's logs. Do not generate a new id inside the catch block, because it would not identify the resource that actually failed.

There is a near-miss that Promise.allSettled() cannot solve. A cleanup promise that never settles prevents the helper from constructing any AggregateError. Eventually Playwright records a test or teardown timeout, and the rejected siblings remain hidden behind that timeout. If the resource API supports an AbortSignal or bounded close operation, apply that limit at the resource boundary. A bare Promise.race() with a timer reports a timeout but leaves the original cleanup running, which can interfere with the next test after the helper has returned.

For dependent resources, attempt teardown in the reverse of setup order and collect failures sequentially. The code below continues after a rejection, which preserves evidence from later cleanup steps. It also makes the chosen order visible in review.

type TeardownStep = {
  label: string;
  run(): Promise<void>;
};

export async function teardownInReverse(
  setupOrder: readonly TeardownStep[],
): Promise<void> {
  const failures: Error[] = [];

  for (const step of [...setupOrder].reverse()) {
    try {
      await step.run();
    } catch (reason) {
      failures.push(
        new Error(`Teardown step failed: ${step.label}`, {
          cause: reason instanceof Error ? reason : new Error(String(reason)),
        }),
      );
    }
  }

  if (failures.length === 1) {
    throw failures[0];
  }
  if (failures.length > 1) {
    throw new AggregateError(failures, 'Ordered teardown did not complete');
  }
}

Continuing teardown has a cost. A failed child cleanup may leave the parent in a state where the next operation is unsafe or destructive. If closing a transaction fails, dropping the database to force cleanup may destroy the evidence needed to diagnose it. Put a stop condition in the sequence when safety requires one, and report the skipped later steps explicitly. Complete diagnostics do not justify reckless cleanup.

Keep skipped work outside the AggregateError unless skipping itself violates the test contract. A small cleanup manifest can record attempted, succeeded, failed, and skipped-for-safety for every step, while the aggregate contains only actual failures. This keeps the test status honest and still tells the recovery job why a database or tenant was left behind. Turning every skipped step into another Error inflates the child count and makes one unsafe boundary look like several independent outages.

The child message should identify the action and owner, not repeat the outer message. Failed to close checkout mock server is useful. Two children both named cleanup failed force the reader back into stack traces and produce ambiguous alert grouping. Preserve the original error through cause rather than copying its entire stack into the new message.

Avoid fabricating children from expected states. A cleanup operation that returns already closed may be a valid idempotent result, not a failure. Convert only actual rejected operations or explicit failure results into errors. Otherwise the aggregate becomes a bag of status messages, and CI treats normal teardown as a failed test.

Inspect the complete list at the right lifecycle point

An afterEach hook can see failures that have occurred in the test body and earlier applicable hooks. It is a convenient place to attach a compact diagnostic file to the current test. Iterate over testInfo.errors, retain every entry, and keep the retry and project identity beside it.

import { test } from '@playwright/test';

test.afterEach(async ({}, testInfo) => {
  if (testInfo.errors.length === 0) return;

  const diagnostic = {
    project: testInfo.project.name,
    retry: testInfo.retry,
    testId: testInfo.testId,
    errors: testInfo.errors.map((error, index) => ({
      index,
      message: error.message,
      stack: error.stack,
      cause: error.cause,
      value: error.value,
      errorContext: error.errorContext,
    })),
  };

  await testInfo.attach('attempt-errors.json', {
    body: Buffer.from(JSON.stringify(diagnostic, null, 2)),
    contentType: 'application/json',
  });
});

test('reports independent cleanup failures', async () => {
  throw new AggregateError(
    [
      new Error('Failed to release account lease'),
      new Error('Failed to stop checkout mock server'),
    ],
    'Checkout cleanup failed',
  );
});

That test is intentionally failing. On Playwright 1.62, the attached array contains the outer aggregate and both child messages. On a runner before 1.61, it contains only the outer entry. The example uses fixed errors to make the serialization shape repeatable; production cleanup should construct the aggregate from real rejected operations as the previous helpers do.

An attachment made in afterEach is not a complete lifecycle record. Playwright tears down test-scoped fixtures after afterEach, so a fixture teardown error can occur after the attachment has been written. Worker-scoped fixture teardown happens later still, when the worker stops. If the release decision depends on all errors from the finished test attempt, capture TestResult.errors in a reporter's onTestEnd callback.

Think of the array in afterEach as a live view at one checkpoint, not a sealed result. A body failure is already available when the hook begins. An error thrown by a later afterEach hook, a test-scoped fixture teardown, or runner cleanup has not happened yet. Copying the current array into JSON freezes that partial view. The attachment remains valuable for body and hook diagnostics, but its filename should not claim to be the final attempt record.

The lifecycle boundary also explains why the hook and reporter types are not identical. Hook code reads TestInfoError, which in v1.62 can include errorContext for diagnostics such as matcher context. Reporter code reads TestError from TestResult.errors; the v1.62 reporter type includes fields such as message, stack, value, cause, location, and snippet, but not errorContext. The two mapping blocks above intentionally differ. Adding error.errorContext to the reporter by type assertion would create a field the public reporter contract does not supply.

onTestEnd runs once for each completed attempt, including each retry. That makes it the right place to bind status, retry number, attachments, and the final attempt error list. It is still not a universal process-error callback. Configuration failures, unhandled worker exceptions, and some fixture teardown failures outside active test execution arrive through reporter.onError(). That callback can also receive workerInfo, which Playwright added in 1.60 and 1.62 keeps unchanged, so a reporter can keep worker-owned cleanup separate without blaming the last test that happened to run there.

The reporter below writes one JSON line per completed attempt. It uses result.errors, not the singular result.error, and includes retry identity. The write happens in the reporter process after Playwright finishes the attempt, so it can include test-scoped fixture teardown failures that an afterEach attachment could not foresee.

import { appendFileSync } from 'node:fs';
import type {
  Reporter,
  TestCase,
  TestResult,
} from '@playwright/test/reporter';

export default class CompleteErrorReporter implements Reporter {
  private readonly outputFile =
    process.env.PW_ERROR_LOG ?? 'test-results/attempt-errors.ndjson';

  onTestEnd(test: TestCase, result: TestResult): void {
    if (result.errors.length === 0) return;

    appendFileSync(
      this.outputFile,
      JSON.stringify({
        testId: test.id,
        project: test.parent.project()?.name,
        retry: result.retry,
        status: result.status,
        errors: result.errors.map((error, index) => ({
          index,
          message: error.message,
          stack: error.stack,
          cause: error.cause,
          value: error.value,
        })),
      }) + '\n',
      'utf8',
    );
  }

  printsToStdio(): boolean {
    return false;
  }
}

Create the parent directory before the run, and give each shard a different output path. Reporters for separate CI processes do not coordinate writes. Letting five shards append to one network-mounted file risks interleaved or overwritten evidence, depending on the storage system. Publish one file per shard and merge records after upload using test ID, project, retry, and shard identity.

Errors outside a test attempt require that different callback. A worker-scoped teardown can fail when no test is active, so do not force it under the last test's ID. Keep the onError stream separate from TestResult.errors; otherwise the last test in a worker appears responsible for infrastructure it did not own. Include the worker index and project when workerInfo is present, and use an invocation id when it is not.

A reporter that writes both streams needs two record kinds. An attempt-error record carries test id, project, retry, and status. A worker-error record carries the available worker identity but no invented test id. Downstream storage can still join both records by run and shard. This small schema distinction prevents a cleanup alarm from changing ownership whenever file order or sharding changes.

Tell aggregate children from look-alike error sets

Several independent Playwright mechanisms can put multiple entries in an error array. Treating all of them as AggregateError children creates bad root-cause data.

Soft assertions are the first look-alike. Each failing expect.soft() records an assertion failure while the test continues. Those entries do not need an outer AggregateError. If a test has two soft assertion failures, testInfo.errors may contain two errors because Playwright recorded two assertions, not because JavaScript grouped them. Look at the messages and test steps rather than assuming the first item owns the rest.

Promise.any() is a less obvious source of a real AggregateError. Imagine a test helper that asks three regional endpoints for a disposable account and accepts the first successful response. When every request rejects, JavaScript rejects the Promise.any() call with an aggregate containing the individual reasons. Playwright 1.62 exposes those reasons with the same runner behavior as cleanup children. They describe alternative acquisition attempts, not three resources that leaked, so a dashboard category based only on “aggregate means teardown” is wrong.

Label those acquisition errors where the requests are made. eu-west account broker returned 503 and ap-south account broker timed out preserve the decision that failed. A wrapper that replaces all three with account allocation failed throws away the useful part before Playwright serializes it. Conversely, do not include successful but slow regions as children. The AggregateError should contain the rejected alternatives supplied by the failed operation, not a performance report.

Multiple hooks are another source. Playwright continues running applicable teardown hooks even if one fails. A failing test body followed by a failing afterEach can therefore yield separate top-level errors. They share an attempt, but one is not a child of the other. Preserve hook or step context when the reporter provides it, and avoid collapsing them into a synthetic aggregate after the run.

That body-plus-hook case often produces a misleading sequence: assertion error, outer cleanup aggregate, then cleanup children. The adjacency resembles one nested tree, but there are two roots. The public arrays do not carry aggregate relationship metadata, so a generic reporter cannot prove which following entries belong to which root. Keep the whole ordered list for human review, but base machine ownership on labels your cleanup helper created rather than on positions.

Error.cause models a chain, not a set. A high-level error might say Failed to close account fixture, with a cause of DELETE /leases/42 returned 503. Playwright exposes that relationship through the serialized error's cause property on supported versions. The cause should not be counted as another independent failed cleanup action. Aggregate children say several actions failed; a cause says why one action failed.

A timeout can also produce more than one visible message as Playwright unwinds hooks and fixtures. Do not infer that every later teardown failure caused the timeout. Inspect the first timed-out step in the trace, then read later entries as consequences or separate failures according to their stacks. Increasing the timeout because an aggregate contains a slow cleanup message can hide the original stuck action.

Retries create separate attempt records. The first attempt might contain an aggregate with three entries, while the retry passes and contains none. Combining both into one deduplicated message set changes the release claim from flaky to passing. Keep result.retry on every exported record, and retain failed-attempt traces according to the suite's trace policy.

A custom wrapper can accidentally duplicate evidence. Suppose a helper catches an AggregateError, records every child through a failing expect.soft(false).toBe(true) assertion, and then rethrows the original aggregate. Playwright now receives the soft failures and the aggregate children. The report contains two representations of each problem. Choose one failure path: normally, let the aggregate escape and add labels when it is constructed.

Non-Error children deserve a defensive branch in exporters. JavaScript permits an AggregateError to contain values such as strings or objects. The v1.62 source serializes those entries into value, so error.message is undefined. A reporter that filters with if (!error.message) return silently recreates the data loss this feature was meant to fix. Export message and value as separate optional fields, and reject non-Error cleanup reasons in your own helper if your operational schema requires stacks and causes.

Message equality is not a safe deduplication rule. Two separate resources may both reject with socket closed, and one resource may produce the same message on a retry. Keep entry position, stack, resource label, project, worker, and retry. If privacy rules require stack removal, retain an explicit resource identifier in the child message or in your own structured cleanup result before constructing the error.

Prove the behavior before changing CI reports

A package upgrade and a reporter change should not land on faith. Add a small meta-test that throws a known two-child aggregate and marks the failure as expected. A companion reporter verifies that the finished attempt contains the outer message and both child messages. Returning a failed status from onEnd is important: Playwright documents that reporter exceptions can be swallowed, while onEnd is allowed to override the run status.

// tests/meta/aggregate-error-reporting.spec.ts
import { test } from '@playwright/test';

test('AggregateError reporting probe', async () => {
  test.fail(true, 'This probe must throw');
  throw new AggregateError(
    [
      new Error('probe cache close failed'),
      new TypeError('probe socket close failed'),
    ],
    'probe cleanup failed',
  );
});

// reporters/aggregate-error-probe.ts
import { basename } from 'node:path';
import type {
  FullResult,
  Reporter,
  TestCase,
  TestResult,
} from '@playwright/test/reporter';

export default class AggregateErrorProbeReporter implements Reporter {
  private missing: string[] | undefined;

  onTestEnd(test: TestCase, result: TestResult): void {
    if (
      test.title !== 'AggregateError reporting probe' ||
      basename(test.location.file) !== 'aggregate-error-reporting.spec.ts'
    ) return;

    const actual = new Set(result.errors.map((error) => error.message));
    const expected = [
      'AggregateError: probe cleanup failed',
      'Error: probe cache close failed',
      'TypeError: probe socket close failed',
    ];
    this.missing = expected.filter((message) => !actual.has(message));
  }

  async onEnd(
    _result: FullResult,
  ): Promise<{ status?: FullResult['status'] } | void> {
    if (this.missing === undefined) {
      console.error('AggregateError reporting probe did not run');
      return { status: 'failed' };
    }
    if (this.missing.length === 0) return;
    console.error(`Missing serialized errors: ${this.missing.join(', ')}`);
    return { status: 'failed' };
  }
}

This oracle can fail for the behavior it protects. On a runner that records only the aggregate, the two child messages are missing and the reporter forces a nonzero status. On 1.61 and later, including 1.62, all three expected messages are present and the intentionally failing test has the expected outcome. If somebody changes the probe error text without changing the reporter, the check also fails, which keeps the fixture and oracle aligned.

The “did not run” branch is just as important. An empty missing array cannot represent both a successful observation and an unselected test. Keeping it undefined until onTestEnd sees the exact probe makes a bad path filter, renamed file, or project exclusion fail the job. Checking the filename as well as the title also prevents an unrelated test with a copied title from satisfying the gate.

The reporter method is asynchronous because the v1.62 reporter type permits onEnd to override status through the value of its returned Promise. Throwing from a reporter callback is not a substitute; Playwright documents that reporter errors are swallowed. Returning { status: 'failed' } is the supported decision path.

Run the probe in the same workspace and dependency context as the real suite. A lightweight CI step can check the resolved version, create the reporter output directory, and execute only the meta-test. The version check gives a quick explanation; the probe proves behavior instead of assuming every package resolved the same dependency.

version="$(node -p "require('@playwright/test/package.json').version")"
node -e '
  const [major, minor] = process.argv[1].split(".").map(Number);
  if (major < 1 || (major === 1 && minor < 61)) {
    console.error(`Playwright 1.61+ required, found ${process.argv[1]}`);
    process.exit(1);
  }
' "$version"

mkdir -p test-results
pnpm exec playwright test tests/meta/aggregate-error-reporting.spec.ts \
  --reporter=line,./reporters/aggregate-error-probe.ts

Pin the package version through the repository's normal lockfile policy, then upgrade the test package and browser binaries together where the suite launches browsers. The meta-test itself does not request a browser fixture, but passing it does not prove the application's browser projects are compatible with the new release. Run the normal upgrade validation separately.

Roll out reporter changes before using child counts in release dashboards. A count jumps after 1.61 because previously hidden children become visible, not necessarily because cleanup quality deteriorated that day. Mark the version boundary in the dashboard and compare failure categories, not raw pre-upgrade and post-upgrade totals as if their collection rules were identical.

More complete errors also increase report size and alert volume. One failing attempt can now yield an outer entry and many children. Group alerts at the test-attempt level, then show children inside the incident. Paging once per child turns better diagnostics into operational noise.

Know when aggregation is the wrong choice

Do not wrap one failure in an AggregateError. It adds an outer entry without adding information, and it forces readers to open a group to find the only cause. Throw the original error or wrap it once with cause when you need higher-level context.

Avoid aggregation when cleanup steps have a hard safety dependency. If a failed export means logs have not been copied, deleting the environment in the next step may destroy evidence. Stop at the unsafe boundary, report which later steps were skipped, and leave recovery to a controlled process.

Do not use AggregateError as a replacement for focused assertions. Ten unrelated UI assertions grouped at the end make the trace harder to follow and often test more than one behavior. Use soft assertions only when seeing several discrepancies in one state is genuinely useful, and keep the test's product decision narrow.

Worker shutdown failures should not be attached to an arbitrary completed test. They belong to worker or infrastructure reporting because several tests may have used that worker-scoped resource. Moving them into the last testInfo.errors array creates false ownership and corrupts retry analysis.

Finally, do not swallow an aggregate merely to keep CI green. If cleanup failure can contaminate later tests or leak a paid resource, the attempt should fail even when its product assertions passed. If cleanup is best-effort by design, record it through a separate operational channel and state that policy explicitly. Turning a rejected cleanup into a passing attachment is not better error handling; it is a different release rule.