Debug several Playwright signals without masking the first failure

An order test reaches the confirmation page, but the status says “Paid,” the receipt link is missing, and the fulfillment badge still says “Processing.” One hard assertion reports only the first mismatch. A careless switch to soft assertions reports more, then lets the test click into a workflow whose preconditions already failed.

The useful pattern is narrower: collect independent observations, poll only the state that may legitimately converge, preserve the values you saw, and put a hard stop before the test performs work that requires those observations to be true.

Why a single assertion hides the shape of the failure

Three Playwright features solve three different control-flow problems. Treating them as interchangeable is where diagnostic tests become misleading.

A regular assertion is a gate. If it fails, the current test path stops at that assertion. Locator assertions such as toHaveText() and toBeVisible() retry until they pass or their assertion timeout expires, but a final failure still prevents the following line from running. That is the right behavior for a login precondition, a successful seed operation, or any state without which later actions are meaningless.

A soft assertion is an observation. According to the Playwright assertion documentation, its failure does not terminate test execution, but it does mark the test as failed. Soft assertions only work with the Playwright test runner. They are useful when several sibling signals describe the same user-visible result and each signal can be inspected without relying on another one.

Polling handles time. expect.poll() repeatedly calls a function and applies a matcher to the returned value. It is suitable for an API job that moves from queued to complete, a database-backed status reflected through a read endpoint, or another state with an intentional delay. It is not a general replacement for every assertion that flakes.

The order confirmation is a good example of the boundary. Payment text, receipt availability, and fulfillment text are separate promises made on one page. If the first one fails, the other two can still tell us whether the entire confirmation projection is stale or only one component is wrong. Clicking “Track shipment” is different. That action assumes the confirmation state is usable, so it must not run after any of those observations fails.

import { expect, test } from '@playwright/test';

test('paid order exposes a usable confirmation', async ({ page }) => {
  await page.goto('/orders/ORD-42');

  await expect(
    page.getByRole('heading', { name: 'Order confirmed' }),
    'confirmation page should load',
  ).toBeVisible();

  await expect.soft(
    page.getByTestId('payment-status'),
    'payment status should be settled',
  ).toHaveText('Paid');

  await expect.soft(
    page.getByRole('link', { name: 'Download receipt' }),
    'receipt link should be available',
  ).toBeVisible();

  await expect.soft(
    page.getByTestId('fulfillment-status'),
    'fulfillment should leave processing',
  ).toHaveText(/Queued|Shipped/);

  expect(
    test.info().errors,
    'confirmation has failed signals, so tracking is unsafe',
  ).toHaveLength(0);

  await page.getByRole('link', { name: 'Track shipment' }).click();
  await expect(page).toHaveURL(/\/tracking\/ORD-42$/);
});

The first assertion is hard because no confirmation evidence is available if the page itself did not load. The next three are soft because they are peer observations. The array check is hard again. Playwright documents test.info().errors as the way to detect soft assertion failures during the test, and that guard prevents the navigation from turning one product defect into a trail of secondary errors.

Custom messages matter here. “Expected visible, received hidden” does not tell a reviewer whether the missing element was the receipt, shipment tracking, or an optional promotion. The messages appear in Playwright reporting output and let the reviewer map a failure to a product promise without reconstructing the locator.

There is a cost. Each soft locator assertion can consume its own timeout. Three sequential checks against three missing elements may take close to three assertion timeout windows, not one. Lowering every timeout to make the test fast can create a different flake. Group only observations that are worth collecting together, and keep the group small enough that its worst-case duration is acceptable.

Soft assertions also produce correlated failures. A single failed confirmation API might cause the status, receipt, and fulfillment checks to fail. That is one likely incident with three symptoms, not automatically three defects. The additional errors describe the blast radius. The trace and network evidence still decide the cause.

Avoid catching a hard assertion merely to imitate softness. A try and catch can accidentally swallow the error or replace its stack with a generic message. expect.soft records the failure through the runner, keeps it in test.info().errors, and preserves the test’s failed result. Use the runner feature when continued observation is intentional.

How to poll changing state and keep its history

An eventual-state test needs more than a longer timeout. It needs a clear definition of what may change and enough history to distinguish slow progress from the wrong response.

The polling callback should perform a read, return a compact value, and avoid side effects. Starting an export inside the callback would create another export on every attempt. Reading /api/exports/EXP-42 is appropriate if that endpoint is safe to call repeatedly. Trigger the export once before the poll, retain its identifier, then observe that identifier.

Playwright allows a message, timeout, and custom intervals in the options passed to expect.poll(). The documented default intervals are 100, 250, 500, and 1000 milliseconds. Those values are a retry schedule, not evidence that the application normally completes within a particular time. Choose a schedule from the service contract and expected load, then confirm it against real runs owned by your team.

A single status string is often too weak. Suppose the callback returns only body.state. An expired session that returns an HTML login page, a 404 caused by the wrong tenant, and a genuinely queued export can all end as “expected complete, received undefined” or a timeout. Return the signals that separate those branches: HTTP status, response shape, domain state, and artifact readiness.

The following test keeps an explicit sample history. The values are attached even when the soft poll fails, and the final hard guard stops the download step.

import { expect, test } from '@playwright/test';

interface ExportStatus {
  state: 'queued' | 'running' | 'complete' | 'failed';
  downloadUrl: string | null;
  rowsWritten: number;
}

interface ExportSample {
  observedAt: string;
  httpStatus: number;
  responseKind: 'json' | 'non-json';
  state: ExportStatus['state'] | null;
  hasDownload: boolean;
  rowsWritten: number | null;
}

test('export becomes downloadable with a diagnostic history', async ({
  request,
}, testInfo) => {
  const exportId = 'EXP-42';
  const samples: ExportSample[] = [];

  await expect.soft.poll(async () => {
    const response = await request.get(`/api/exports/${exportId}`);
    const rawBody = await response.text();

    let payload: Partial<ExportStatus> | undefined;
    try {
      payload = JSON.parse(rawBody) as Partial<ExportStatus>;
    } catch {
      payload = undefined;
    }

    const sample: ExportSample = {
      observedAt: new Date().toISOString(),
      httpStatus: response.status(),
      responseKind: payload ? 'json' : 'non-json',
      state: payload?.state ?? null,
      hasDownload: Boolean(payload?.downloadUrl),
      rowsWritten: payload?.rowsWritten ?? null,
    };

    samples.push(sample);

    return {
      httpStatus: sample.httpStatus,
      responseKind: sample.responseKind,
      state: sample.state,
      hasDownload: sample.hasDownload,
    };
  }, {
    message: 'export should reach a downloadable terminal state',
    intervals: [500, 1_000, 2_000],
    timeout: 15_000,
  }).toEqual({
    httpStatus: 200,
    responseKind: 'json',
    state: 'complete',
    hasDownload: true,
  });

  await testInfo.attach('export-poll-history', {
    body: JSON.stringify({
      exportId,
      retry: testInfo.retry,
      parallelIndex: testInfo.parallelIndex,
      samples,
    }, null, 2),
    contentType: 'application/json',
  });

  expect(
    testInfo.errors,
    'export diagnostics failed, so downloading is unsafe',
  ).toHaveLength(0);
});

The explicit history is important because a polling assertion is designed to decide whether a matcher eventually passes. Do not rely on undocumented reporter formatting to retain every intermediate value. The attached array is data your test owns, and testInfo.attach() is a documented API that copies an attachment into a location reporters can access.

Keep the history bounded. A poll that runs for ten minutes at short intervals can produce a large JSON attachment and meaningful load on the service. For a long-running workflow, use wider intervals, cap the retained sample count, or store state transitions rather than identical consecutive samples. Make that choice explicitly instead of silently discarding evidence after a failure.

An aggregate return object gives its strongest evidence when the fields come from one response. If the callback calls three endpoints in sequence, the object combines three different observation times. That may be fine for a health overview, but it does not prove the states coexisted. When atomic coherence matters, ask the product for one versioned status resource or compare a shared version identifier. Promise.all reduces elapsed skew between independent reads; it does not turn them into a transaction.

Browser UI signals have the same problem. Sequential auto-retrying assertions can observe payment at one time and fulfillment several seconds later. When the requirement is “these fields eventually agree,” poll a snapshot of the fields together. The next example samples one rendered page and records each result. The reads start together, although the browser still offers no transactional snapshot guarantee.

import { expect, type Page, test } from '@playwright/test';

interface ConfirmationSnapshot {
  payment: string | null;
  fulfillment: string | null;
  receiptLinks: number;
}

async function readConfirmation(page: Page): Promise<ConfirmationSnapshot> {
  const [payment, fulfillment, receiptLinks] = await Promise.all([
    page.getByTestId('payment-status').textContent(),
    page.getByTestId('fulfillment-status').textContent(),
    page.getByRole('link', { name: 'Download receipt' }).count(),
  ]);

  return {
    payment: payment?.trim() ?? null,
    fulfillment: fulfillment?.trim() ?? null,
    receiptLinks,
  };
}

test('confirmation signals converge on one page', async ({ page }, testInfo) => {
  await page.goto('/orders/ORD-42');

  const snapshots: ConfirmationSnapshot[] = [];

  await expect.soft.poll(async () => {
    const snapshot = await readConfirmation(page);
    snapshots.push(snapshot);
    return snapshot;
  }, {
    message: 'confirmation signals should agree',
    intervals: [250, 500, 1_000],
    timeout: 10_000,
  }).toEqual({
    payment: 'Paid',
    fulfillment: 'Queued',
    receiptLinks: 1,
  });

  await testInfo.attach('confirmation-snapshots', {
    body: JSON.stringify(snapshots, null, 2),
    contentType: 'application/json',
  });

  expect(testInfo.errors).toHaveLength(0);
});

This technique costs repeated DOM reads and can hide a brief invalid state if the final matcher eventually passes. That is correct only when the product contract allows convergence. If the UI must never show “Paid” beside “Payment failed,” an eventual matcher is the wrong oracle. Capture the transition as an event or assert the forbidden combination continuously within a purpose-built test.

Which lookalike failure are you actually seeing?

A soft poll timeout tells you that the expected terminal value did not arrive. It does not tell you why. Several causes produce nearly identical assertion headlines, so triage starts with the sample shape and then checks the trace or report.

Consider an illustrative export history. These rows show possible patterns, not measurements from a real system.

The first branch is a genuine multi-signal product failure when the contract says “complete” means downloadable. Increasing the timeout will not repair a terminal state that contradicts itself. If the contract instead permits the artifact URL to appear later, the matcher should encode that second transition and the timeout should come from that contract. The test cannot decide the product semantics on its own.

The authentication branch looks like a slow job when the callback returns only a domain field. Including HTTP status and response kind makes it obvious. Do not poll through a 401 for fifteen seconds unless token refresh is explicitly part of the behavior under test. In most suites, authentication is a precondition and should fail hard on the first unauthorized response.

A 404 needs similar restraint. Some APIs use “not found yet” as an eventual-consistency response; others guarantee the resource exists immediately after creation. Only the first contract justifies polling the 404. A generic helper that retries every non-200 response erases that distinction and can turn wrong IDs into slow failures.

Parallel data collisions often reveal themselves through identity. The attached parallelIndex identifies a parallel worker slot, and retry identifies the attempt number. Those fields do not prove a collision, but they let you compare histories from concurrent attempts. If two tests deliberately share EXP-42, the test design is already suspect. Generate a unique identifier before the action and make the service echo it in the status response when possible.

The stale UI branch requires two timelines. The API history can prove what the direct client received. A browser trace can show the page’s network traffic and DOM snapshots around Playwright actions. If the background API reached complete but the browser never requested new state, inspect the refresh trigger, subscription, or timer. If the browser requested state and received stale data, move the investigation toward caching or the projection service. If the response was current while the DOM stayed old, rendering is the stronger lead.

Be precise about what this evidence cannot prove. A 200 response containing complete proves only what that endpoint returned to the test. It does not prove that a database transaction committed correctly, an object exists in storage, or another user can download it. Add an independent read only when that behavior belongs to the requirement. More signals are useful when they close a real evidence gap, not when they make the report look thorough.

One near-miss deserves special attention: observation skew. Three sequential soft locator assertions each auto-retry. The first may wait several seconds before failing, while the second passes immediately because the page changed in the meantime. The report then appears to show an impossible combination. It actually shows values observed at different times. An attached snapshot history or trace timing separates this from simultaneous disagreement.

Another near-miss is a slow test environment. If every state transition stretches only under CI and the API history shows steady progress, the poll timeout may be too short for the agreed CI service level. That is different from repeated identical terminal contradictions. Before raising a timeout, compare the last state and transition sequence across failures. A larger limit is defensible for continuing progress; it is weak medicine for no progress.

Retries can further blur the story. Playwright classifies a test that fails first and passes on retry as flaky. A clean retry does not invalidate the first failure. The worker process is restarted after a failure, so the retry may use fresh browser and fixture state. Retain evidence per attempt and compare the retry number rather than combining both runs into one narrative.

Finally, separate assertion fanout from independent causes. One expired login can trigger a poll timeout, a missing button, and an absent receipt. test.info().errors will correctly contain multiple failures, but their count is not a root-cause count. Start with the earliest invalid precondition or the first divergence in the attached history. Then use later soft errors to see how far that cause propagated.

What the errors, report, and trace can prove

Good diagnostics survive the test function. A console line seen only in a live CI log is easy to lose, and parallel workers can interleave output. Attach structured evidence to the test result and let the reporter keep it with the correct attempt.

The TestInfo API exposes errors as the errors thrown during test execution. Its singular error property is the first element of that array. For multi-signal work, preserve the array. Do not assume the first element is always the backend root cause; it is the first recorded test error, which may be a downstream symptom if the test began observing too late.

An afterEach hook can attach a compact error index. At that point testInfo.status is available, so the record can include actual and expected status. The hook below uses documented TestInfoError fields and does not add another assertion.

import { test } from '@playwright/test';

test.afterEach(async ({}, testInfo) => {
  if (testInfo.errors.length === 0)
    return;

  const errorIndex = testInfo.errors.map((error, index) => ({
    index,
    message: error.message ?? null,
    value: error.value ?? null,
    stack: error.stack ?? null,
  }));

  await testInfo.attach('recorded-errors', {
    body: JSON.stringify({
      title: testInfo.title,
      retry: testInfo.retry,
      status: testInfo.status,
      expectedStatus: testInfo.expectedStatus,
      errors: errorIndex,
    }, null, 2),
    contentType: 'application/json',
  });
});

This hook complements the domain attachment. recorded-errors says which assertions failed. export-poll-history says what the system returned over time. Combining them by test attempt is much more useful than copying several stack traces into a ticket.

The HTML reporter produces a self-contained report folder and supports attachments. Custom assertion messages make the failure list readable. Open the failed test, inspect the error messages, then open the poll-history attachment. If retries are enabled, confirm which attempt you are viewing before comparing values.

Trace Viewer answers a different set of questions. Its Errors tab shows test error messages and marks the failure location on the timeline. The Actions view shows locators, action duration, source location, and before-and-after DOM snapshots for actions. The Network view shows browser network requests with request and response details. The Attachments tab exposes attached evidence. These features are documented in the Trace Viewer guide.

A trace is not backend telemetry. It can show that the browser received a response and that the DOM did or did not change around an action. It cannot reveal why a queue worker stalled unless that reason reached the browser or an attachment. Likewise, a direct request-fixture poll history is your explicit evidence; do not promise that every internal server event will appear in the browser Network view.

Reproduce once without retries before changing code. One worker removes parallel scheduling as a variable, and a trace from the first run prevents a successful retry from becoming the only artifact you inspect.

npx playwright test tests/export.spec.ts --workers=1 --retries=0 --trace on
npx playwright show-report

trace_file="$(find test-results -name trace.zip -print -quit)"
test -n "$trace_file"
npx playwright show-trace "$trace_file"

These commands are a diagnostic run, not proof that concurrency is irrelevant. If the failure disappears with one worker, rerun with the original worker count and unique data. The difference is evidence for a shared-state or timing hypothesis. It is not permission to ship the whole suite permanently serialized.

Check the trace in a consistent order:

1. Read the custom assertion message and locate the failing source line.
2. Inspect the attached sample history for the first divergent value.
3. Use Actions and DOM snapshots for UI observation timing.
4. Use Network for browser request status, payload, and sequence.
5. Compare retry and parallel index with another attempt.
6. Decide whether the evidence implicates product state, test identity, authentication, or environment capacity.

The order saves time because it starts with the test-owned history. Hunting through every network request first often produces an attractive but unrelated anomaly. If the attached sequence begins with a 401, authentication is already a better lead than CSS rendering.

Be careful with secrets. Poll histories can contain URLs with signed query strings, tenant identifiers, or error bodies with personal data. Attach only fields needed for diagnosis. The export example records a Boolean for download availability instead of the actual URL. That small design choice keeps the evidence useful without turning every CI artifact into a credential-handling problem.

How to keep the evidence in CI

CI needs the failed attempt, not only the retry that passed. The trace mode determines which story survives.

Playwright currently documents several trace modes. on-first-retry records the first retry, which is useful and relatively cheap, but it does not record the original failing run. retain-on-failure records every run and keeps a trace when that run fails. If an initial run fails and a retry passes, the original failed trace remains. That makes it the better choice when the exact pre-retry state matters.

The trade-off is recording overhead. Successful runs are recorded before their traces are discarded, so CPU, I/O, and test duration can increase even though artifact storage remains focused on failures. Measure that cost in your own suite. If it is too high, on-first-retry is a valid compromise as long as reviewers understand that it shows the retry environment, not the first failure.

A practical configuration keeps a readable terminal reporter, an HTML report, one retry on CI, and failed-run traces.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  retries: process.env.CI ? 1 : 0,
  reporter: [
    ['line'],
    ['html', {
      open: 'never',
      outputFolder: 'playwright-report',
    }],
  ],
  use: {
    trace: 'retain-on-failure',
  },
});

The reporter documentation confirms the HTML reporter’s open and outputFolder options. The report directory and test-results directory must be uploaded even when the test command exits nonzero. Configure that “always upload” rule in the CI system you actually use. A perfect Playwright configuration cannot recover files that the pipeline discards before artifact upload.

Artifact retention needs a policy. A poll history with sanitized states is cheap. Traces can be much larger because they include snapshots, screenshots, source, and network information. Keep longer retention for release branches or recurring failures, and shorter retention for routine pull requests. Use actual storage and triage needs to set the period; a universal number would be invented.

Do not hide flakiness behind the retry. Playwright’s retry model categorizes tests as passed, flaky, or failed. A passed retry belongs in the flaky category. Whether a flaky result blocks deployment is a team policy, but the report should retain the first attempt and the diagnostic attachments should include retry so nobody mistakes the clean run for the only run.

The HTML report is more useful when failure messages name obligations consistently. Prefer messages such as “export should reach a downloadable terminal state” over “poll failed.” Stable messages let reviewers group recurring symptoms without making the locator or endpoint part of a public test name. Keep the resource identifier in the sanitized attachment when it is safe.

CI concurrency deserves its own comparison. The local command with one worker is good for reducing variables. The normal CI run must still exercise the intended worker count. If only the parallel run fails, capture unique test IDs, parallelIndex, and attempt histories. Do not fix a collision by widening the poll timeout. Two workers mutating one record remain two workers mutating one record, only more slowly.

Add the diagnostics in stages. Turning on failed-run tracing for an entire large suite while adding verbose poll histories everywhere makes cost changes hard to attribute. Start with one high-value async workflow. Check that failures retain the HTML report, trace, and attachments. Then expand the helper to tests with the same contract.

A rollout review should inspect real failed artifacts, not only green executions. Deliberately point a test at a controlled local stub that never reaches the expected state, or use a known negative fixture in a non-production environment. Confirm that the soft poll records the failure, the attachment exists, the hard guard prevents the dependent action, and the CI job preserves the report. Remove the controlled fault after verifying the wiring.

Avoid asserting inside the evidence hook. If afterEach adds a second failure because an attachment is missing, it can distract from the original error. Let attachment failures surface naturally, but keep the hook small and use test-owned hard gates in the body for behavior that must block progress.

Where soft polling belongs, and where it does not

Soft polling is appropriate when two conditions are true. First, the system is allowed to converge over time. Second, useful independent observations remain after one signal fails. Remove either condition and a simpler assertion is safer.

Authentication is usually a hard precondition. If the session is unauthorized, continuing to poll export state mostly measures how often the endpoint rejects the same identity. Fail on the unauthorized response, refresh only if token refresh is the behavior under test, and keep the export scenario separate.

Test-data creation is another hard boundary. A missing order ID does not become valid because the status endpoint was called ten more times, unless the service contract explicitly makes creation asynchronous. Assert the create response and identifier before starting a poll. This gives a 404 later a meaningful interpretation.

Schema checks should also fail early. If a response that must be JSON is HTML, record the response kind and stop before accessing domain fields. A soft poll can be useful when a gateway temporarily returns a documented transitional response, but that exception should appear in the matcher and contract. A helper that treats parsing failures as “not ready” will turn broken deployments into timeouts.

Avoid polling actions with side effects. POSTing “start export,” clicking “Pay,” or sending a verification email from the callback repeats the business action. The callback should read. If the product offers no safe read endpoint, use a UI state or event that can be observed without replaying the command.

Do not soften cleanup simply to collect more errors. Fixture teardown should attempt owned cleanup even when the body fails, and cleanup errors deserve their own evidence. A body-level soft assertion that says “resource is gone” before teardown may be checking the wrong lifecycle point. Put cleanup verification where the fixture or API ownership model makes it true.

Do not use one giant aggregate matcher for unrelated requirements. Browser title, account balance, email delivery, and analytics ingestion may all happen in one journey, but they have different clocks and owners. A single object that waits until all four match produces one timeout and a difficult history. Split by contract, while keeping sibling signals together only where one product outcome connects them.

There is also a coverage cost to the hard guard. When payment status fails, the test will not exercise shipment tracking. That is intentional because the tracking action’s precondition is broken. Preserve tracking coverage with a separate test that establishes a valid confirmed order through a trusted setup path. Continuing one corrupted journey is not broader coverage.

Latency grows in two places. Sequential soft web assertions can each consume their assertion timeout. Poll callbacks create repeated reads until the overall timeout. Trace recording adds runtime and I/O. Those costs are justified when the retained evidence shortens diagnosis, but they are not free. Review the slowest failing path, not only passing duration.

Complexity grows as well. Sample types, sanitization, attachment hooks, and artifact retention all require maintenance. A two-second UI transition with one authoritative locator does not need a custom diagnostic framework. Use Playwright’s auto-retrying locator assertion. Add polling history when repeated failures have shown that the final value alone cannot separate likely causes.

A mature migration starts with a failure question, not an API search. Pick one flaky async test and list the hypotheses reviewers currently cannot distinguish. Perhaps they need to separate unauthorized, never-created, still-running, terminal-without-artifact, and stale-UI states. Capture only the fields that divide those branches.

Next, label existing assertions as gates or observations. Keep navigation, identity, seed success, and destructive-action preconditions hard. Convert only peer observations to expect.soft. Insert expect(test.info().errors).toHaveLength(0) before the first dependent action. Run a controlled negative case and verify that action never occurs.

Then identify the one state allowed to change. Replace manual sleeps or repeated custom loops with expect.poll or expect.soft.poll, depending on whether further read-only evidence remains useful after timeout. Give the poll a domain message, a bounded timeout, and intervals suited to the service. Record samples when their sequence changes the diagnosis.

Finally, wire attachments and trace retention into CI. Verify artifacts from the initial failed attempt, including a failed-then-passed retry. Review storage and runtime from actual pipeline runs before expanding the pattern. If the new data never changes a triage decision, remove it.

The stopping rule is straightforward: when a failed observation makes the next action unsafe or meaningless, read test.info().errors and stop with a hard assertion. When no independent evidence remains, use a hard assertion from the start. Softness is useful for seeing farther across one failure, not for pretending the failure did not happen.