Playwright ariaSnapshot Boxes for AI Testing

Playwright ariaSnapshot boxes AI testing adds [box=x,y,width,height] to accessible nodes so an AI tool can relate semantic controls to viewport geometry. Use ariaSnapshot({ mode: 'ai', boxes: true }) for scoped diagnostic context, then keep product assertions separate. The first failure boundary is a missing, mislabeled, or wrongly positioned target in a controlled viewport.

That distinction matters. A boxed ARIA snapshot can tell a reviewer that a named button occupies a rectangle near another semantic element. It cannot prove that the pixels look correct, that keyboard focus reaches the button, or that an AI recommendation is trustworthy. The useful pattern is semantic assertion first, bounded evidence second, human-governed automation last.

Decide What Geometry Needs to Explain

Start with a question that a plain accessibility tree cannot answer. An unboxed snapshot already exposes roles, accessible names, text, selected states, and hierarchy. Add boxes when the consumer must correlate those semantics with position or size, such as choosing among similarly named controls, explaining an overlap seen in a screenshot, or giving an agent enough spatial context to propose a target.

The output belongs to the evidence layer, not automatically to the acceptance layer. For example, a checkout test may require that the Pay now button has an accessible name, is enabled, and submits an order. Those are deterministic requirements. Its rectangle may help explain why an agent selected a nearby Apply coupon button, but the rectangle alone does not establish correct checkout behavior.

Use the broader Playwright ARIA snapshot accessibility guide when the contract is roles, names, states, or hierarchy without geometry. Use boxed output only when location changes the diagnostic or automation decision. This keeps artifacts smaller and avoids turning every harmless responsive shift into review noise.

This guide owns the geometry boundary: box coordinates, viewport relationships, and deterministic spatial invariants. The companion guide to ARIA snapshot depth and mode options owns output selection, truncation, default-versus-AI mode, and the risk of omitting required descendants. Use both only when a workflow genuinely needs both a bounded tree and spatial evidence.

The first observable assertion should remain close to the user outcome. Confirm the scoped region and target control before collecting AI context. If that assertion fails, the defect is already localized to rendering or semantics. If it passes and an agent later chooses the wrong reference, the failure belongs to the consumer, prompt, or selection policy rather than the application.

Verify the Playwright 1.61.1 Contract

The Playwright release notes record boxes as an option added for locator.ariaSnapshot() and page.ariaSnapshot(). In the installed 1.61.1 types, the method returns Promise<string> and accepts four independent options: boxes, depth, mode, and timeout. boxes defaults to false; mode defaults to "default".

When boxes: true, each represented element can receive [box=x,y,width,height]. Coordinates are relative to the viewport and measured in CSS pixels using Element.getBoundingClientRect() semantics. Scrolling therefore changes coordinates, and an element can have a negative position when it extends above or left of the current viewport. These numbers are not physical screen pixels and should not be translated directly into a screenshot captured at another scale.

AI mode changes more than formatting. It adds references such as [ref=e4], includes snapshots from iframes under the target, and does not wait for a missing locator target in the same way as default mode. The generated references are useful within the captured context, but they are not test IDs or durable selectors. Do not store one and expect it to identify the same element after navigation, rerendering, or a later snapshot.

The official snapshot guide explains that ARIA snapshots are YAML representations of accessible structure. Boxes extend that representation; they do not replace its semantic basis. A node omitted from the accessibility representation will not become meaningful simply because a DOM element has a rectangle.

Build a Stable, Scoped Fixture

A useful fixture controls the inputs that affect both semantics and layout. Set an explicit viewport, use deterministic test data, wait for the intended UI state, and scope the capture to a landmark or component. A whole-page snapshot can include account details, navigation experiments, banners, and iframe content that the test neither owns nor needs.

Fixture: a checkout region with stable labels

<main aria-label="Checkout">
  <section aria-labelledby="payment-heading">
    <h2 id="payment-heading">Payment</h2>
    <p id="total">Total: $42.00</p>
    <button type="button">Apply coupon</button>
    <button type="submit">Pay now</button>
  </section>
</main>

This markup gives the main region a stable accessible name and each button a distinct name. A fixed total removes one source of width changes. The fixture still allows the production stylesheet to participate, which is necessary when geometry is part of the evidence. Replacing the real styles with arbitrary test CSS would make the boxes easier to stabilize while weakening their diagnostic value.

Before capture, wait for a product milestone, not an elapsed delay. A visible total, completed loading state, or enabled submit button is better than waitForTimeout. The Playwright locator guide covers the retrying locator assertions that should establish that milestone. If web fonts affect the investigation, make their readiness an explicit fixture condition or pin the CI image that provides them.

Capture Boxed ARIA Evidence in a Numbered Workflow

Use the same sequence each time so the artifact has a clear chain of custody:

1. Set the viewport, browser project, locale, reduced-motion preference, and deterministic account data before navigation.
2. Navigate through the user path that owns the state instead of injecting unrelated DOM after the fact.
3. Locate the smallest stable landmark or component that contains the decision surface.
4. Assert the target's semantic and actionable state with retrying Playwright matchers.
5. Capture ariaSnapshot with mode: "ai", boxes: true, and a depth that includes the needed descendants.
6. Sanitize or exclude sensitive content before any external model receives the string.
7. Attach the evidence with runtime metadata, then apply the release decision to deterministic assertions rather than the model response.

Playwright 1.61.1 example: scoped capture and failure evidence

import { test, expect } from '@playwright/test';

test('checkout exposes a usable payment action', async ({ page }, testInfo) => {
  await page.setViewportSize({ width: 1280, height: 720 });
  await page.goto('/checkout');

  const checkout = page.getByRole('main', { name: 'Checkout' });
  const payNow = checkout.getByRole('button', { name: 'Pay now', exact: true });

  await expect(checkout).toBeVisible();
  await expect(payNow).toBeEnabled();

  const evidence = await checkout.ariaSnapshot({
    mode: 'ai',
    boxes: true,
    depth: 4,
    timeout: 5_000,
  });

  await testInfo.attach('checkout-aria-boxes', {
    body: evidence,
    contentType: 'text/yaml',
  });

  await payNow.click();
  await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
});

The code never parses a reference and clicks it. Playwright locators execute the product flow; the boxed snapshot explains the state seen before the action. That arrangement also keeps an AI outage from converting a deterministic checkout test into an infrastructure failure.

For teams building their own matcher or attachment helper, the custom Playwright matcher guide is a useful companion. Keep that helper transparent: accept a locator, capture options, and redaction policy; return the text or attachment. It should not hide navigation, mutate the page, or silently approve a model-selected action.

Choose the Right Evidence Tool

Boxes are one answer to one kind of question. The following decision table prevents a spatial artifact from being used as a universal assertion.

The strongest test often combines two rows. A locator assertion protects the semantic target, while a screenshot or boxed snapshot explains a visual failure. The visual accessibility scenarios show why a visually present control and an accessible control are related but separate claims. Likewise, keyboard navigation testing is required when the risk is focus order or keyboard activation.

Avoid feeding a screenshot and boxed tree to a model without aligning their viewport, scroll position, device scale, and capture moment. A semantically correct box from one state cannot reliably annotate pixels from another. If the page moves between captures, recapture both or rely on a stable locator to identify the target.

Design a Boundary and Failure Matrix

The matrix below assigns each likely failure to its first observable boundary. It is concrete enough to drive test cases and triage ownership.

Add a regression case for each boundary that has caused an incident. A matrix is more valuable than dozens of coordinate permutations because it maps a failure to an owner and a response. For broader scanner coverage, compare these cases with the strengths and limits of accessibility testing tools.

Convert Geometry Into an Owned Invariant

A boxed YAML string is designed to communicate context. It is not the best data structure for coordinate arithmetic. If a product requirement genuinely depends on geometry, retrieve the target's box through locator.boundingBox() and assert the smallest relation that expresses the requirement. Keep the boxed snapshot beside that assertion so a reviewer can connect the number to the surrounding semantic tree.

Examples of legitimate invariants include a primary action remaining inside the supported viewport, a menu appearing adjacent to its trigger, or an overlay not covering a required control. Phrase each one in user terms. "Button x equals 824" is usually an implementation accident. "The payment action is fully available at the supported 1280 by 720 viewport" identifies a compatibility promise and explains why failure should block.

Geometry assertion: keep the required action inside the viewport

const viewport = page.viewportSize();
const box = await payNow.boundingBox();

expect(viewport).not.toBeNull();
expect(box).not.toBeNull();

if (!viewport || !box) throw new Error('Payment geometry was unavailable');

expect(box.x).toBeGreaterThanOrEqual(0);
expect(box.y).toBeGreaterThanOrEqual(0);
expect(box.x + box.width).toBeLessThanOrEqual(viewport.width);
expect(box.y + box.height).toBeLessThanOrEqual(viewport.height);

This assertion is intentionally stronger than toBeVisible. A partially clipped element may still satisfy visibility depending on the situation, while the requirement above demands the entire rectangle. That strength is appropriate only if full containment is the stated product contract. For a long page where scrolling is expected, assert that the user can scroll and activate the control instead of requiring it to start inside the initial viewport.

Derived relations can reduce noise. An overlap check compares two rectangles rather than storing their absolute positions. An adjacency check can allow a documented tolerance. A minimum target-size check can compare width and height with a policy value. Each derived rule still needs a supported viewport, browser scope, and product owner. Geometry varies for valid reasons, so a tolerance must come from the requirement rather than from whichever value makes the current test pass.

Do not mix capture moments. If the boxed snapshot is taken before a sticky header settles but boundingBox() is read afterward, the two artifacts describe different layouts. Freeze or disable nonessential animation, wait for the intended state, and collect related evidence together. A timestamp or named test step can make that relationship visible in the trace.

Treat absent geometry as a distinct outcome. locator.boundingBox() can return null when the element is not visible, while an ARIA node can still help explain semantic state. Do not replace null with zeroes. Report whether the target was missing, hidden, outside the chosen scope, or unavailable because the page changed. Those causes lead to different fixes and should not collapse into a generic coordinate mismatch.

Finally, keep arithmetic deterministic. A model may describe that two boxes overlap, but CI can calculate the intersection directly. Let code gate a measurable relation and let AI summarize the evidence, suggest likely causes, or prioritize inspection. That division gives reviewers a reproducible failure while retaining the explanatory value that motivated boxed snapshots in the first place.

Debug Geometry Without Hiding Semantic Defects

When boxed output is surprising, inspect the semantic node first. Confirm its role, name, state, and scope with the unboxed snapshot or a direct locator. Then inspect viewport, scroll offset, transforms, zoom, font readiness, animation state, and fixed overlays. This order catches the common case where the wrong element was captured before anyone studies its coordinates.

Do not normalize every number to make two snapshots equal. Rounding or stripping boxes may be appropriate when preparing a privacy-safe model prompt, but it defeats a test whose question is geometry. Conversely, exact box-string comparison is a poor default because harmless text wrapping can change several descendants. Write a direct geometric assertion only for a named invariant, such as a supported control being inside the viewport or two critical regions not overlapping.

When visual and semantic artifacts disagree, align their capture conditions. The guide to debugging Playwright visual snapshot CI diffs covers fonts, animation, browser projects, and environment drift. A boxed tree is especially sensitive to those inputs because text metrics affect both node size and every element below it.

An AI explanation should cite the semantic node, reference, box, and screenshot region it used. Reject an explanation that mentions a control absent from the captured snapshot or claims a visual property that geometry cannot show. The AI-generated Playwright test evidence checklist can turn those requirements into a repeatable review gate.

Govern Boxed Snapshots in CI

Pin the Playwright package and browser versions for the evidence job. Record the project name, operating system image, viewport, device scale, locale, color scheme, reduced-motion setting, and relevant feature flags beside the attachment. Without that metadata, coordinate differences are difficult to reproduce and easy to misclassify.

Keep three decisions separate. Product assertions decide whether the supported workflow works. Evidence policy decides whether the artifact is complete and safe to retain. Agent evaluation decides whether a model interpreted the artifact correctly. A product pass must not excuse leaked data, and a model mistake must not be filed as a broken button.

Define the model-input contract as carefully as the test attachment. Record which landmark may be exported, the maximum permitted depth, whether boxes and frame content are allowed, and which text classes must be removed. Validate that contract before transmission. A simple denylist is not enough for user-generated names or free-form messages, so prefer synthetic fixtures and structural selection over broad text replacement. Also record a hash of the sanitized payload rather than the private original when audit correlation is required. The hash can show that a reviewer and model evaluated the same evidence without adding another copy of sensitive page content. If sanitization removes the control name or relationship needed for the task, stop the agent workflow and route the case to local review. Sending an incomplete artifact can produce a confident but untraceable selection, which is a failed evidence pipeline rather than permission to widen collection automatically.

Use risk-based retention. A normal passing test rarely needs a boxed whole-page snapshot. Attach scoped evidence on failure or sample a small percentage of passing runs to monitor the collector. Retain incident artifacts long enough for diagnosis, with access controls matching the text and values they may contain. The Playwright ARIA snapshot guide is the right reference for reviewing semantic changes; coordinate churn needs a rendering or layout owner instead.

CI should reject bulk snapshot updates that mix semantic and geometric changes without explanation. Require a reviewer to state whether the role/name tree changed, whether only layout changed, and which supported viewport owns the new result. If model prompts or providers change, rerun a fixed evaluation set before letting agent decisions affect release status.

For custom evidence matchers, test the helper itself against missing locators, empty accessible subtrees, frames, sensitive text, and attachment failures. Custom matcher practices help keep timeout and error messages visible instead of replacing Playwright diagnostics with a generic wrapper error.

Frequently Asked Questions

What does the boxes option add to a Playwright ARIA snapshot?

The boxes option appends a viewport-relative rectangle to each represented element in the form [box=x,y,width,height]. Values use CSS pixels and follow getBoundingClientRect semantics. The option adds geometry to semantic output; it does not turn the snapshot into a screenshot, a visual assertion, or a permanent element identifier.

Must ariaSnapshot use AI mode when boxes is true?

No. boxes and mode are independent ariaSnapshot options in Playwright 1.61.1. AI mode is often the useful pairing because it also emits element references and includes iframe snapshots. Default mode can include boxes too. Choose the mode for the consumer, then decide separately whether geometry is necessary.

Should tests assert exact bounding-box coordinates?

Usually not. Exact coordinates change with viewport size, fonts, browser rendering, content, scrolling, and responsive layout. Assert the product requirement with locators or a targeted visual check. Treat boxed ARIA output as diagnostic evidence unless the test owns a narrowly defined geometric invariant with a fully controlled rendering environment.

Are boxed ARIA snapshots safe to send to an AI service?

Not automatically. The YAML can contain accessible names, visible text, values, URLs, frame content, and spatial clues. Review it as potentially sensitive test evidence. Use synthetic accounts, redact or omit private regions, restrict provider and retention settings, and send only the smallest scoped snapshot needed for the task.

How should CI retain ARIA snapshots with boxes?

Attach boxed snapshots to failed or explicitly sampled runs, together with Playwright version, browser project, viewport, locale, and test identifier. Keep deterministic assertions as the release gate. Apply short retention to routine diagnostics, longer retention to investigated failures, and never approve coordinate churn as if it were a semantic product change.

Practice the Evidence Path in QABattle

The safe implementation is small: assert one user-facing control, capture one scoped boxed tree, align it with one rendering state, and classify failures by boundary. Once that path is repeatable, add agent selection tests and privacy controls. Do not begin with a whole-page coordinate baseline or a model that can silently decide whether the application passed.

Take the same discipline into a live QA scenario at QABattle battles. Choose a UI challenge, define the semantic assertion before inspecting layout, then collect only the geometry needed to explain the failure. That practice turns boxed ARIA output into evidence with an owner instead of another artifact that CI stores but nobody can trust.