Run the tests your change can break, then run the rest

A shared helper changes on Tuesday, yet UI Mode still shows hundreds of tests and nobody knows which ones deserve the first run. One engineer filters by a guessed filename, another reruns the last failures, and CI eventually finds a checkout test neither person selected. The problem is test selection, not test execution.

Playwright can narrow the first pass to files affected by source changes. That is a useful way to shorten the feedback loop, but it is not a new definition of regression coverage. The feature is strongest when the relationship between production code and a spec is visible in the module graph, and weakest when behavior arrives through configuration, remote state, generated files, or runtime lookup.

Know what the selector can actually see

The command-line switch is --only-changed [ref]. Without a reference, Playwright considers uncommitted changes in a Git working tree. With a reference such as origin/main, it compares the current work with that reference. The command-line documentation is explicit about two boundaries: the feature only supports Git, and the optional value is a Git ref.

Selection then goes beyond asking whether a .spec.ts file changed. Playwright analyzes the suite's dependency graph. A changed test file is an obvious candidate. A test that imports a changed helper can also be selected. The release notes describe the practical case as running test files that import changed files, while the CI guide calls the overall process a heuristic that may miss tests. Both statements matter. The graph gives you more value than a filename diff, but it does not turn every real dependency into a static edge.

Think about what collection has available. The runner loads configuration, discovers projects and test files, and can observe imports involved in that process. It can relate a spec to a helper imported through normal module resolution. It cannot infer that a row changed in a shared staging database, that a feature flag now resolves differently, or that a JSON filename assembled at runtime points to new behavior. Git records changed files. It does not record the business blast radius.

UI Mode adds an option to show tests affected by source changes. That is the human-friendly path when you are investigating locally. The CLI form is better for a reproducible diagnostic because it lets you state the comparison ref in the command. Do not blur those two use cases. A developer clicking an affected-tests control against local edits and CI comparing a pull request with origin/main may be looking at different change sets even though the intent sounds identical.

Start diagnosis with collection, not execution. The following commands do not run a browser test. They verify the base reference, show the changed filenames, and ask Playwright to print the selected test list. Run them from the same checkout and with the same project filter as the failing job.

git rev-parse --verify origin/main
git diff origin/main --name-only
git ls-files --others --exclude-standard
npx playwright test --project=chromium --only-changed=origin/main --list
npx playwright test --project=chromium --list

The first command separates a Git setup failure from a Playwright selection question. Treat the outputs of the next two commands as one changed-file set. In Playwright 1.61.1, --only-changed=origin/main uses tracked changes from git diff origin/main --name-only and also includes untracked, non-ignored files from git ls-files --others --exclude-standard. A triple-dot diff is not equivalent on a diverged history, and a diff by itself omits untracked files. The two --list invocations separate collection from browser behavior. If an expected spec is absent from the ordinary list, --only-changed is not the first thing to fix. Look at testDir, testMatch, testIgnore, the chosen project, environment-dependent config, and any collection error. If the spec appears in the ordinary list but not the affected list, then inspect the change relationship.

That distinction saves time. A missing test under both lists cannot be recovered by changing the base branch. A test present under both lists does not have a selection problem at all. Once execution begins, application errors and fixtures add noise that makes a simple file-list mismatch harder to see.

Trace three different changes through the graph

The cleanest success case is a production helper imported by a spec. Suppose checkout formats a total through a module that the test also imports to calculate the expected label. Changing the helper should put the importing spec in the affected set. The assertion still checks the rendered product result, so it can fail if the application and expected behavior diverge.

import { test, expect } from '@playwright/test';
import { formatPrice } from '../src/checkout/format-price';

test('shows the payable total in the order summary', async ({ page }) => {
  await page.goto('/checkout?fixture=standard-order');

  const expectedTotal = formatPrice({ amountInMinorUnits: 2599, currency: 'USD' });
  await expect(page.getByTestId('order-total')).toHaveText(expectedTotal);
});

This example earns its selection edge through the import. It earns its test value through the DOM assertion. Replacing that assertion with expect(formatPrice(...)).toBe(formatPrice(...)) would create an oracle that cannot detect a product defect. A useful affected test must still be a useful test.

There is a trade-off in importing production logic into an expected-value calculation. If the UI and the test both call the same broken formatter, they can agree for the wrong reason. Mature suites often keep a small independent oracle for critical calculations. For a currency label, that might be an explicit expected string for a few representative fixtures. The import remains helpful for selection, but it should not be introduced solely to manipulate the dependency graph.

The second case is a changed component with an indirect import chain. A spec imports a checkout page object, the page object imports stable selectors, and the application imports the changed component. The test module may not import the application component at all. Whether Playwright can relate the full chain depends on what is part of the discovered dependency graph. Do not assume a browser navigation to /checkout creates a source dependency from the spec to every module bundled into that route. Network navigation is not a TypeScript import.

This is where teams overestimate the feature. The application build graph and the test collection graph are related in some repositories, but they are not automatically the same graph. If the web server builds separately and the spec only knows a URL, changing CheckoutSummary.tsx may not create a visible import edge from that component to checkout.spec.ts. Verify with --list; do not infer selection from the fact that a human sees the relationship.

A small manifest can make such ownership explicit, but it should be used as a suite design tool rather than a collection trick. For example, a checkout test group can import a typed list of owned application areas and assert that its expected routes exist. That import must have a real maintenance purpose. Fake imports added only to force affected selection make the graph look precise while hiding the actual coverage model.

The third case looks similar in a pull request and is fundamentally different. Imagine a file named environments/staging.json changes the payment provider from a simulator to a sandbox. A test loads the environment name from process.env, constructs the path, and reads the JSON at runtime. The source file may never appear as a static import. The affected list can omit payment tests even though the operational risk is high.

import fs from 'node:fs';
import path from 'node:path';
import { test, expect } from '@playwright/test';

type Environment = { paymentProvider: 'simulator' | 'sandbox' };

const environmentName = process.env.TEST_ENV ?? 'local';
const file = path.join(process.cwd(), 'environments', `${environmentName}.json`);
const environment = JSON.parse(fs.readFileSync(file, 'utf8')) as Environment;

test('uses the provider configured for this environment', async ({ page }) => {
  await page.goto('/checkout');
  await expect(page.getByTestId('payment-provider'))
    .toHaveAttribute('data-provider', environment.paymentProvider);
});

The test itself is valid: a configuration change can make its final assertion fail. The selection edge is uncertain because the filename is computed. Treat directories containing deployment configuration, seed data, database migrations, API schemas, localization catalogs, and generated clients as policy inputs. A change policy can add known test projects for those paths, or simply require the full suite immediately.

These three examples should produce three different decisions. A direct imported helper is a good fit for affected selection. A component reached only through a running web application needs proof from the actual list. A runtime-selected environment file deserves an explicit fallback. Calling all three "changed code" loses the detail needed to operate the feature safely.

Prove selection before diagnosing a missing test

When someone says, "UI Mode did not pick up my test," ask for four pieces of evidence: the installed Playwright version, the exact comparison point, the changed file list, and both collected test lists. A screenshot of the UI filter is useful context, but it is not enough to reproduce the selection. Branch names move, local modifications change, and project checkboxes alter collection.

Version matters because the affected-tests control arrived after --only-changed, and its UI may evolve. Record npx playwright --version in the issue. Compare that with the lockfile used by CI rather than a globally installed binary. If one engineer invokes a workspace package and another invokes a different global version, their screens and selection behavior can differ before application code enters the picture.

The base ref is the next checkpoint. In a pull request job, origin/main must exist in that checkout and point where the team expects. A shallow checkout may contain only the pull request commit. In that case, an error mentioning revision or diff setup is evidence of missing Git history, not evidence that no tests are affected. The git rev-parse check in the earlier diagnostic should fail before Playwright collection is trusted.

An empty affected list can be legitimate. Documentation-only changes, an isolated script outside the suite, or a file with no visible relationship to a collected spec may select nothing. In Playwright 1.61.1, a run using --only-changed is explicitly exempt from the normal no-tests failure, so an empty affected selection succeeds without --pass-with-no-tests. That flag is for other commands where finding no tests is an allowed result, such as an optional path or title filter. It is not required for an empty --only-changed run. The full suite must still run later, otherwise a selection blind spot becomes a green release signal.

A surprisingly small list can also come from project filtering. A spec may be collected in a mobile project but not Chromium, or a setup project may be excluded by --no-deps. UI Mode lets users select projects and, in supported versions, control dependencies. Capture those choices. Comparing an unfiltered local list with a --project=chromium CI list tells you little.

File discovery errors are another near-miss. A syntax error in a changed helper can stop collection before a complete list exists. Read standard error and the process exit code, not only the last line that mentions a total. "No tests selected" and "collection aborted" require different fixes. The former may be policy. The latter is a broken suite.

Finally, confirm the import path instead of relying on editor navigation. Barrel files, path aliases, conditional exports, and generated entry points can change which module the runner resolves. Follow the imports from the expected spec toward the changed file. If that chain ends at an HTTP URL, a database query, a filesystem lookup, or a string-based plugin loader, you found the boundary of static selection.

Preserve the selected list as plain job output when the mismatch is intermittent. File names and test titles are usually more useful than a screenshot because a reviewer can search them, compare projects, and spot duplicate titles. Keep the command beside the output so the base ref and project are not lost when someone copies the log into an issue. Redact repository paths only if they expose information your normal CI logs do not already reveal.

Check renames from both sides. A moved helper can appear as a deletion plus an addition depending on Git's rename detection and the surrounding edit. The current spec may import the new path while the comparison still contains meaningful history at the old path. Let Playwright produce the list, then test the moved boundary directly if the result is uncertain. A refactor that changes import topology is precisely when intuition about the old graph is least reliable.

Also distinguish a saved file from an editor buffer. Git and the Playwright process can only inspect content written to disk. If UI Mode appears stale while code is still unsaved, no dependency algorithm can see the intended edit. Save the files, allow collection to settle, and rerun the list before restarting browsers or clearing caches. That simple check belongs ahead of timeout changes because no amount of waiting makes an invisible change enter the graph.

Trace Viewer is not the tool for this first phase. A trace exists only after a test runs, so it cannot explain why an unselected test never started. Use UI Mode and --list for discovery evidence. Use the trace after the chosen test executes and fails. Keeping those stages separate prevents teams from searching an artifact that cannot contain the missing event.

Wire a fast pass into CI without weakening the gate

The safest CI shape is sequential. Fetch the base history, install the pinned dependencies and browser, run the likely affected tests, then run the full project. If the preliminary run finds a failure, the job stops early and produces feedback without paying for the entire suite. If it passes, the full run remains the release evidence.

name: Playwright pull request checks

on:
  pull_request:

jobs:
  affected-then-full:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: 24

      - name: Enable the repository package manager
        run: corepack enable

      - name: Install dependencies
        run: pnpm install --frozen-lockfile

      - name: Install Chromium and operating-system dependencies
        run: pnpm exec playwright install --with-deps chromium

      - name: Run likely affected tests first
        run: pnpm exec playwright test --project=chromium --only-changed="origin/${GITHUB_BASE_REF}"

      - name: Run the complete Chromium project
        run: pnpm exec playwright test --project=chromium

This workflow deliberately avoids configuring package-manager caching in actions/setup-node before that package manager exists on PATH. Caching can be added after the team chooses a supported setup order, but it is unrelated to affected selection and should not obscure the example.

The preliminary and full runs have different jobs. The first optimizes time to the first actionable failure. The second protects coverage. Do not merge their reports and then claim the combined pass rate proves selection accuracy. A test run is evidence about the tests that ran. It says nothing about an omitted test.

Keep retries consistent between the two passes during rollout. If the fast pass uses retries and reports a flaky test as ultimately passing while the full pass forbids flakes, engineers will see conflicting outcomes for the same change. --fail-on-flaky-tests is available when the team wants any recovered retry to fail the run, but that is a separate release policy. Decide it explicitly rather than smuggling it into the selection migration.

Artifacts also need a deliberate policy. Running the same selected test again in the full suite can create two reports and duplicate failure artifacts. That cost is the price of early feedback. Use failure-only screenshots or a trace retention mode that matches your debugging needs, and give the two run directories distinct names if your CI uploader would otherwise overwrite them. Do not disable evidence on the preliminary run: the first failure is the reason the stage exists.

For a large matrix, run affected selection inside the project most relevant to the pull request, then leave the complete cross-browser or cross-device matrix unchanged. A changed CSS component may justify an early Chromium pass while Firefox and WebKit still run in the release stage. Starting every project twice can erase the latency benefit.

Roll out the filter as an observed optimization

Introduce selection in report-only terms before anyone treats it as infrastructure. For each representative pull request, retain the affected file list and the complete file list. After the full run, note whether any failing spec had been absent from the preliminary list. This is not a request to publish a fabricated selection percentage. It is a way to discover the repository-specific blind spots using actual pull requests.

Choose changes that exercise different dependency shapes: a spec edit, a shared fixture edit, an application module reached through the browser, an environment file, a database migration, and a root Playwright configuration change. The expected selection should be written before the run. If the result surprises the team, update the fallback policy or the suite structure. Do not tune expectations after seeing the output.

A useful policy maps risky paths to actions. A change under tests/ can rely on the affected pass for early execution. A change to global setup, authentication state creation, reporter code, test configuration, or a common fixture may warrant all tests in the relevant project. A change to deployment manifests or backend schemas may skip the optimization entirely. The mapping belongs in code review documentation or a small CI script with tests of its own.

That script must have a real oracle. Given a changed database migration, it should emit the full API and checkout projects. Given a test-only change, it should preserve Playwright's filtered arguments. Unit tests should feed different path lists and assert different commands. A fixture that always contains the expected category and then asserts the category exists proves nothing.

Watch maintenance cost. Every explicit path rule adds ownership. Directory moves can make a fallback silently stale. Review the policy when application boundaries change, and keep the default conservative. If an unknown high-risk path appears, choosing the full suite is slower but honest.

The human workflow deserves the same discipline. In UI Mode, an affected filter helps a developer focus on likely tests while editing. Before pushing, that developer should run tests for the feature boundary they changed, even if the UI list is smaller. An engineer often knows about a runtime dependency the graph cannot see. The tool narrows attention; it does not replace that knowledge.

The benefit is still substantial when used correctly. A checkout regression can surface before unrelated account and reporting specs finish. A changed shared helper can bring its consumers to the top of the queue. The cost is duplicate execution for selected tests and extra Git history in CI. Those are visible costs, unlike the hidden cost of pretending a heuristic is complete.

Skip affected-only execution when the risk is not source-local

Do not use the filtered set as the only run for a release branch, a dependency upgrade, a browser-version update, or a Playwright upgrade. Those changes can alter behavior across every test without changing imported application modules. The same rule applies to base images, operating-system packages, certificates, proxies, and network policy.

Avoid it as the sole check when tests depend heavily on shared remote environments. A changed feature flag, seeded account, payment sandbox, email provider, or database permission can affect a distant spec with no code edge. Run the appropriate project or the full suite. If the environment is unstable, fix or isolate it rather than using selection to reduce the number of symptoms.

Generated code needs special care. If generation happens before collection and specs import the generated output, the graph may be useful. If a generator runs inside the application build or produces files ignored by Git, the comparison can miss the meaningful change. Treat schema and generator changes as broad until observed evidence supports a narrower rule.

Monorepos add another boundary. A Git ref applies to the checkout, while Playwright configuration may collect one package. A change in a sibling service can break the tested UI through an API contract even though no local spec imports that service. Repository-level task graphs can complement Playwright selection, but they are not the same mechanism. Make one system responsible for deciding which application packages changed and let Playwright decide which collected tests are related within its visible graph.

Do not force the feature into a non-Git source export. The CLI documentation says --only-changed only supports Git. An archive produced without history, a copied build directory, or a vendor workspace needs an explicit test list or a full run. Creating a dummy Git repository around it would manufacture a comparison point with no trustworthy history.

Finally, skip the optimization when the suite is already small enough that the extra stage costs more than it saves. Checkout history, a second collection, duplicate startup, and repeated selected tests all add latency. Measure those costs from real jobs if you need a business case. Until then, describe the purpose accurately: affected selection is an ordering strategy for feedback, while the complete suite remains the coverage claim.