Debug Playwright Visual Snapshot Diffs That Appear Only in CI

A screenshot that fails only in CI is an environment comparison until proven otherwise. Do not update the baseline or widen tolerance from the red pipeline. First reproduce the CI pixels, classify their shape, and identify which render input differs from the baseline contract.

The highest-value clue is the diff pattern. Uniform text halos suggest rasterization or fonts. A shifted page suggests viewport, scrollbar, or late layout. One dynamic region suggests data or time. A completely different state means the test captured the wrong application phase, not a pixel-comparison problem.

Preserve the three images and the render contract

Collect the expected, actual, and diff images from the same failed assertion. Keep their directory structure intact so the test and project remain identifiable. Add the trace and a small metadata attachment containing the browser name, viewport, locale, timezone, color scheme, platform, and relevant application build identity.

The official visual comparisons guide warns that rendering can vary with host operating system, settings, hardware, power source, and headless mode. It recommends comparing in the same environment where baselines were generated. That is the first boundary to verify, not an afterthought.

Reproduce inside the CI renderer

Use the same container or machine image, dependency lockfile, Playwright browser installation, command, project, and environment variables as CI. Running headless on a different local operating system is not equivalent. If baselines are platform-specific, confirm the failing job selected the expected snapshot suffix and path.

Do not regenerate snapshots during this step. Mount or check out the existing references read-only, run the failing test, and compare artifact hashes and dimensions. A clean reproduction converts an intermittent remote symptom into a controlled debugging loop.

Record render inputs in a small helper. Avoid claiming that metadata alone explains pixels; it makes hidden differences reviewable.

import os from 'node:os';
import { test as base } from '@playwright/test';

export const test = base.extend<{ renderEvidence: void }>({
  renderEvidence: [async ({ page, browserName }, use, testInfo) => {
    await use();

    const evidence = {
      browserName,
      platform: os.platform(),
      release: os.release(),
      project: testInfo.project.name,
      viewport: page.viewportSize(),
      devicePixelRatio: await page.evaluate(() => window.devicePixelRatio),
      locale: await page.evaluate(() => navigator.language),
      fontsStatus: await page.evaluate(() => document.fonts.status),
    };

    await testInfo.attach('render-evidence.json', {
      body: Buffer.from(JSON.stringify(evidence, null, 2)),
      contentType: 'application/json',
    });
  }, { auto: true }],
});

If platform release details are sensitive in your environment, omit or normalize them. The important point is to capture enough inputs to compare jobs without leaking host inventory unnecessarily.

Follow the font and text path

Text-only diff noise deserves a font investigation before tolerance changes. Confirm the intended font files exist in the image, load without network errors, and are licensed for that distribution. A font-family declaration does not prove that the desired face rendered; the browser may use a fallback with different metrics.

Assert a product readiness signal that is set after critical assets load. During diagnosis, query document.fonts.check() for the exact family and style used by the target. Keep the screenshot focused on a stable component so unrelated text does not multiply a small rasterization difference.

import { expect, test } from '@playwright/test';

test('account summary matches the approved layout', async ({ page }) => {
  await page.goto('/account/visual-fixture');
  await expect(page.getByTestId('account-summary')).toHaveAttribute(
    'data-render-state',
    'ready',
  );

  await expect.poll(() =>
    page.evaluate(() => document.fonts.check('16px "Inter"')),
  ).toBe(true);

  await expect(page.getByTestId('account-summary')).toHaveScreenshot(
    'account-summary.png',
  );
});

Replace the illustrative family with the application's actual required face. If typography is a visual requirement, substituting a test-only font removes coverage. If exact glyph rasterization cannot be aligned across platforms, maintain platform-specific baselines or run visual comparison on one canonical renderer rather than hiding large text regions.

Follow the geometry and viewport path

Large shifted blocks usually come from dimensions rather than colors. Compare viewport, device scale factor, mobile emulation, scrollbar behavior, browser project, and full-page versus element capture. A missing image or late web font can also move everything below it.

Set dimensions in the Playwright project instead of relying on a CI window default. Assert images and layout prerequisites before capture. For responsive pages, each baseline should name the intended project or viewport contract; one image should not float between desktop and mobile layouts.

Check whether a vertical scrollbar appears only in one environment because content wraps differently. Masking the edge will not fix the shifted content. Narrow the font or data cause, or make the target an element screenshot when page-level overflow is outside the claim.

Follow the motion, time, and caret path

Playwright screenshot assertions handle animations and carets according to their options, but application state can still vary through JavaScript timers, canvas drawing, rotating content, videos, and asynchronous transitions. Control the source that matters instead of sleeping until the frame looks convenient.

Use deterministic fixture data, a fixed application clock where time is part of rendering, and explicit readiness markers. A narrow mask is acceptable for a region outside the visual contract, but every mask creates an unreviewed rectangle. Pair it with another assertion for behavior hidden by that mask.

If motion itself is the feature, a static screenshot may be the wrong assertion. Test the state before and after the transition and use a focused visual reference only at a defined endpoint.

Follow the application-state path

A diff can faithfully report that CI rendered an error banner, empty state, experiment variant, or unauthenticated page. Inspect the screenshot as a product state before studying pixels. Trace network and console evidence often identifies a failed seed call, missing environment variable, expired authentication state, or feature flag mismatch.

Assert semantic prerequisites immediately before the image. The test should fail with "invoice fixture not loaded" rather than compare an empty panel against a populated reference. This also prevents a visually similar loading skeleton from becoming a candidate baseline.

Keep data values deterministic without making the component unrealistic. Names, dates, image URLs, and list lengths should be explicit fixture inputs. Random factories are useful elsewhere but make poor visual references unless seeded and stable across processes.

Review thresholds as risk boundaries

threshold, maxDiffPixels, and maxDiffPixelRatio solve different problems. Change only the dimension supported by measured evidence. A broad per-pixel threshold can hide subtle color regressions; a large allowed pixel count can hide a compact but important missing icon.

Begin from the strictest reproducible comparison. If the canonical renderer still produces a small known variation, document the observed shape and choose a narrow limit. Scope the screenshot to the component under test so the tolerance cannot be spent across unrelated page regions.

Never combine a new baseline, larger threshold, and new mask in one unexamined repair. Reviewers would be unable to tell which change made the failure disappear or how much coverage was lost.

Make baseline updates a controlled decision

Generate a candidate only in the approved render environment and only after the product change is understood. Review expected, actual, and diff images beside the source change. If the difference is a defect, fix the product or test state and keep the baseline. If it is intentional, approve the new reference as code.

Do not let a failed CI job automatically overwrite repository snapshots. That creates a path where an authentication error, missing font, or real regression becomes the accepted image. Keep candidate generation separate from comparison and require a human-readable reason for baseline changes.

Close the CI loop with retained evidence

Artifact upload should run after failed tests unless the job was cancelled. Preserve the snapshot directories, HTML report or trace, and render metadata under a name that includes project and shard identity. Confirm that the baseline files were present in the test job; an artifact from a later merge step cannot explain an earlier missing reference.

Before accepting a fix, verify:

• Expected, actual, and diff images belong to the same test and project.
• The baseline and actual use the same canonical OS and browser inputs.
• Required fonts and images load before the named readiness state.
• Viewport, device scale, locale, timezone, and feature flags are explicit.
• Dynamic data, time, animation, and masks have documented ownership.
• Any tolerance change matches measured noise and preserves defect sensitivity.

Rerun the candidate in the canonical local or container environment, then rerun the unchanged CI command. Close the incident only when both produce the same decision. A green visual test should mean the reviewed pixels are reproducible, not that the comparison has been loosened until CI stops objecting.