Read custom Playwright arguments without confusing them for test filters

CI says the tests ran against staging, but the report contains records from the default tenant. The pipeline passed --qa-tenant=blue before Playwright's separator, so the value never reached the test fixture as custom data. Depending on its shape, a misplaced argument can fail CLI parsing or become a test-file filter before any test begins.

Custom arguments are useful when a run needs a non-secret business selector such as a dataset, tenant alias, or deterministic seed. They are dangerous when every consumer parses them differently or when teams expect them to rewrite configuration that Playwright has already resolved. One typed parser and one explicit ownership rule prevent most of that damage.

Know where the arguments enter the runner

Playwright 1.61 added argv to FullConfig. The official API describes it as a snapshot of process.argv captured in the runner process. It specifically points to values supplied after the -- separator and says Playwright does not parse them. Slicing, naming, validation, defaults, and type conversion belong to your code.

The separator creates a real boundary. Everything before -- remains part of the Playwright command: runner options, project selection, and optional test filters. Everything after it can remain available in FullConfig.argv without becoming a test filter. This command has one Playwright option followed by three custom values:

npx playwright test --project=chromium -- \
  --qa-environment=staging \
  --qa-tenant=blue \
  --qa-seed=17

Move --qa-tenant=blue to the left of the separator and the CLI sees an option it does not define. Remove the leading dashes and place blue before the separator, and the CLI can treat it as a regular-expression test filter because Playwright accepts non-option test-filter arguments. Neither outcome is a delayed fixture bug. Collection or argument parsing changed before the fixture could help.

Package-manager wrappers add another command boundary. If package.json defines "e2e": "playwright test", npm needs its own -- to forward following arguments to the script, and Playwright still needs the separator that marks its custom region. A direct Playwright invocation and an npm-script invocation can therefore look different even though the final child-process arguments should agree.

# Direct invocation
npx playwright test -- --qa-environment=staging --qa-tenant=blue

# npm consumes the first separator; Playwright receives the second one.
npm run e2e -- -- --qa-environment=staging --qa-tenant=blue

Do not memorize the number of punctuation tokens without inspecting the expanded script. Print a sanitized config.argv shape during migration or use a wrapper test that asserts the separator and namespaced values appear where the parser expects them. Shell functions, task runners, and container entrypoints may add their own forwarding rules. The invariant belongs at the Playwright process: one separator before custom tokens.

A script can also insert a test filter after the custom separator by mistake. Current Playwright treats post-separator values as custom rather than file filters, so that filter no longer narrows collection. Keep test paths and --project on the Playwright side. Keep only consumer-owned values on the other side. This makes the command reviewable without knowing token order inside every parser.

FullConfig is resolved configuration. Tests reach it through testInfo.config. Global setup receives it as its argument. Reporters receive it in hooks such as onBegin. Those are suitable consumers because they act after Playwright has built the run configuration.

The object passed to defineConfig() is TestConfig, not FullConfig. A value needed to choose projects, set use.baseURL, or define webServer must be available while the configuration module is evaluated. Waiting to read FullConfig.argv is too late for that job. Use an explicit configuration input, often a validated environment variable or separate config file, when the value changes runner topology.

That timing distinction keeps custom arguments narrow. A tenant alias used by tests to choose a fixture record fits. A flag that decides whether Firefox exists in the projects array does not. A seed used to reproduce generated test data fits. A token that changes the web server command does not. The first group is run context; the second group is runner configuration.

The snapshot includes the runner's process arguments, not a curated dictionary of your keys. Do not assume the last three strings always belong to you, and do not read by fixed array index. Executable paths, Playwright commands, file filters, and runner options can move as scripts change. Find the separator and parse only your namespace after it.

Use a prefix such as --qa- to avoid collisions with other consumers. The prefix also makes logs and reviews easier to scan. It does not grant security. Any value on a process command line may be visible to operating-system process tools, shell history, CI command logs, or diagnostic reporters.

Version is part of the contract. On an older Playwright type package, FullConfig does not contain argv. A local global CLI can report a newer version while the repository imports older types. Check the package version resolved by the project and the version installed in CI. The lockfile, not a developer's global executable, owns repeatability.

Parse one narrow contract and reject ambiguity

The parser below accepts only equals-form values after --. It recognizes three keys, rejects duplicates and unknown --qa- names, validates each value, and returns a frozen object. Equals form avoids ambiguity over whether the token after a key is its value or another flag.

// tests/support/run-options.ts
export type RunOptions = Readonly<{
  environment: 'preview' | 'staging';
  tenant: string;
  seed: number;
}>;

function customTokens(argv: readonly string[]): string[] {
  const separator = argv.indexOf('--');
  if (separator === -1) {
    throw new Error('Pass QA arguments after the -- separator');
  }
  return argv.slice(separator + 1);
}

export function parseRunOptions(argv: readonly string[]): RunOptions {
  const values = new Map<string, string>();

  for (const token of customTokens(argv)) {
    if (!token.startsWith('--qa-')) continue;

    const match = /^--qa-([a-z-]+)=(.+)$/u.exec(token);
    if (!match) {
      throw new Error(`Expected --qa-name=value, received ${token}`);
    }

    const [, name, value] = match;
    if (values.has(name)) {
      throw new Error(`Duplicate QA argument: --qa-${name}`);
    }
    values.set(name, value);
  }

  const known = new Set(['environment', 'tenant', 'seed']);
  for (const name of values.keys()) {
    if (!known.has(name)) {
      throw new Error(`Unknown QA argument: --qa-${name}`);
    }
  }

  const environment = values.get('environment');
  if (environment !== 'preview' && environment !== 'staging') {
    throw new Error('--qa-environment must be preview or staging');
  }

  const tenant = values.get('tenant');
  if (!tenant || !/^[a-z][a-z0-9-]{1,30}$/u.test(tenant)) {
    throw new Error('--qa-tenant must be a lowercase tenant alias');
  }

  const seedText = values.get('seed') ?? '1';
  if (!/^[1-9][0-9]*$/u.test(seedText)) {
    throw new Error('--qa-seed must be a positive integer');
  }
  const seed = Number(seedText);
  if (!Number.isSafeInteger(seed)) {
    throw new Error('--qa-seed is outside the safe integer range');
  }

  return Object.freeze({ environment, tenant, seed });
}

The allowlist deliberately excludes production. That is a product-safety decision for this example, not a Playwright limitation. If production smoke tests are authorized, add a separately reviewed mode with stronger safeguards instead of quietly widening the regular expression.

Ignoring non---qa- tokens lets another tool own custom values after the same separator. Unknown names inside the namespace still fail. A typo such as --qa-tenat=blue should not fall back to a default tenant and produce a plausible green run.

The seed defaults to one because deterministic data generation can have a safe local value. Environment and tenant remain required because guessing either could aim the run at the wrong data. Defaults should reflect risk. Convenience is not a reason to default a destructive target.

Test the parser independently of Playwright collection. Use realistic process.argv shapes and near misses that could change behavior. The negative assertions below prove that missing separators, duplicated keys, and typos cannot silently pass.

// tests/support/run-options.spec.ts
import { test, expect } from '@playwright/test';
import { parseRunOptions } from './run-options';

const runner = ['/usr/bin/node', '/repo/node_modules/playwright/cli.js', 'test'];

test('parses namespaced values after the separator', () => {
  expect(parseRunOptions([
    ...runner,
    '--project=chromium',
    '--',
    '--qa-environment=staging',
    '--qa-tenant=blue',
    '--qa-seed=17',
  ])).toEqual({ environment: 'staging', tenant: 'blue', seed: 17 });
});

test('rejects a missing separator', () => {
  expect(() => parseRunOptions([
    ...runner,
    '--qa-environment=staging',
    '--qa-tenant=blue',
  ])).toThrow('after the -- separator');
});

test('rejects a misspelled namespaced key', () => {
  expect(() => parseRunOptions([
    ...runner,
    '--',
    '--qa-environment=staging',
    '--qa-tenat=blue',
  ])).toThrow('Unknown QA argument: --qa-tenat');
});

test('rejects a duplicate value', () => {
  expect(() => parseRunOptions([
    ...runner,
    '--',
    '--qa-environment=staging',
    '--qa-tenant=blue',
    '--qa-tenant=green',
  ])).toThrow('Duplicate QA argument: --qa-tenant');
});

Each negative branch can be triggered by a real command change. The test is not comparing constants that must agree. Delete duplicate detection, weaken the namespace check, or move the separator and at least one case fails.

Avoid a general-purpose parser unless the suite needs general-purpose syntax. Supporting short flags, combined flags, optional values, arrays, negation, and positionals increases the number of ambiguous commands. A small equals-only grammar is easier to document in a CI wrapper and easier to attach safely to a report.

Equals form also handles a value containing = without splitting it into a separate argument because the parser's regular expression captures the remainder after the first delimiter. Whether such a value is allowed still belongs to key validation. The tenant grammar rejects it; a non-secret build label might allow it. Do not decode or normalize every value globally, because URL decoding, case folding, and path resolution mean different things for different keys.

Repeated values need a written policy. "Last one wins" is convenient for shell overrides but can hide a wrapper that supplied staging while a caller appended preview. Rejecting duplicates forces the conflict into the open. If arrays become necessary, use a dedicated repeatable key and define ordering rather than weakening duplicate protection for every option.

Several tenants should normally become a CI matrix, not one comma-separated argument. A matrix produces one FullConfig context, report, and failure scope per tenant. A comma grammar makes every fixture split strings, raises questions about escaping tenant names, and can mix data ownership inside one report. The matrix costs additional runs, but it preserves isolation.

Paths deserve their own resolver. A raw relative --qa-fixture-dir=./data is interpreted by your code, not Playwright. Decide whether it is relative to the current working directory, config file, or repository root and attach the resolved non-sensitive path. Without that rule, a command works from a package directory and fails from a monorepo root. The generic parser should retain the string; a path-specific validator should own resolution and existence checks.

Keep error messages actionable but sanitized. Naming --qa-tenant and the accepted grammar helps a caller fix the command. Echoing the rejected tenant may be acceptable for a public alias and inappropriate if the value can contain customer information. Validation messages should not become a loophole around the no-secrets policy.

Make fixtures and setup consume the same values

Validate once before browsers launch, then expose the same parser through a typed fixture. Global setup is appropriate for early rejection because it receives FullConfig. The fixture is appropriate for tests because testInfo.config is also FullConfig and keeps the dependency visible in the test signature.

// tests/global-setup.ts
import type { FullConfig } from '@playwright/test';
import { parseRunOptions } from './support/run-options';

export default async function globalSetup(config: FullConfig) {
  parseRunOptions(config.argv);
}

// tests/fixtures.ts
import { test as base } from '@playwright/test';
import { parseRunOptions, type RunOptions } from './support/run-options';

export const test = base.extend<{ runOptions: RunOptions }>({
  runOptions: async ({}, use, testInfo) => {
    const options = parseRunOptions(testInfo.config.argv);
    await testInfo.attach('run-options.json', {
      body: Buffer.from(JSON.stringify(options, null, 2)),
      contentType: 'application/json',
    });
    await use(options);
  },
});

export const expect = test.expect;

Parsing twice is inexpensive and protects two boundaries. Global setup stops the run before test work. The fixture gives TypeScript consumers a stable object and attaches the non-sensitive selection to each attempt. If parsing later becomes expensive, cache only by the immutable argv contents and retain both validation tests.

The attachment makes a report auditable. It should contain environment alias, tenant alias, and seed because those values explain product results. It must not contain access tokens, session cookies, database passwords, customer email addresses, or raw connection strings. Sanitizing after a secret has entered process.argv is too late because other process and shell surfaces may already expose it.

A test should consume runOptions only where the option changes its setup or oracle. Do not inject it into every page object. The following worked case uses tenant and seed to create deterministic import data, then checks the user-facing tenant identity before submitting. That identity assertion is the guard against a request accidentally reaching the default tenant.

// tests/imports/create-import.spec.ts
import { test, expect } from '../fixtures';

test('creates an import in the selected tenant', async ({
  page,
  runOptions,
}) => {
  await page.goto(`/tenants/${runOptions.tenant}/imports/new`);

  await expect(page.getByTestId('environment-name')).toHaveText(
    runOptions.environment,
  );
  await expect(page.getByTestId('tenant-name')).toHaveText(
    runOptions.tenant,
  );

  const reference = `pw-${runOptions.seed}`;
  await page.getByLabel('Import reference').fill(reference);
  await page.getByRole('button', { name: 'Create import' }).click();

  await expect(page.getByRole('status')).toHaveText('Import created');
  await expect(page.getByText(reference, { exact: true })).toBeVisible();
});

The test does not pass because the fixture returned the expected object. It compares those inputs with the loaded application's environment and tenant indicators, then observes a created record. If routing falls back to another tenant, one of the UI assertions fails before mutation. If creation fails, the status and saved reference fail.

A reporter can read the same FullConfig in onBegin and print a sanitized run banner. Use the shared parser rather than writing a third command-line loop. Reporter output is useful once per run; per-test attachments are useful after sharding and retries. If your reports leave the trusted CI system, decide whether even tenant aliases are acceptable before including them.

// tests/reporters/run-context.ts
import type { FullConfig, Reporter, Suite } from '@playwright/test/reporter';
import { parseRunOptions } from '../support/run-options';

export default class RunContextReporter implements Reporter {
  onBegin(config: FullConfig, suite: Suite) {
    const options = parseRunOptions(config.argv);
    console.log(JSON.stringify({
      event: 'qa-run-start',
      environment: options.environment,
      tenant: options.tenant,
      seed: options.seed,
      selectedTests: suite.allTests().length,
      playwrightVersion: config.version,
    }));
  }
}

The selected count comes from the discovered suite supplied to the reporter. It is a run fact, not an invented benchmark. Do not turn one historical count into a universal quality threshold. Compare identities when collection scope matters.

Parse in onBegin, not in the reporter constructor. Reporter construction receives its configured reporter options, while onBegin receives the resolved FullConfig. Moving the parser into the constructor and falling back to ambient process.argv creates two code paths and makes reporter unit tests behave differently from the runner.

Do not mutate config.argv to remove tokens after parsing. The property is a snapshot shared as configuration evidence, and other consumers may still need it. The parser accepts a readonly array and returns a frozen domain object so downstream code cannot turn Blue into Green halfway through a run. Immutability does not prevent the application from changing, but it keeps the test's declared context stable.

Global setup validation occurs before test bodies, but test collection and configuration work may already have happened. This is another reason not to use custom runtime options to rescue project selection. The main benefit is avoiding browser launches and product mutations after invalid context, not bypassing every earlier runner phase.

Parser unit tests need a launch path that is not blocked by the same required global setup they are trying to test. One option is a small Playwright config for support tests without the integration globalSetup. Another is to run the support project with valid real custom arguments while its test bodies exercise invalid synthetic arrays. Document that command in the repository. Otherwise a developer may run the missing-separator unit case and see global setup fail before the case is collected.

Diagnose separator, version, and shard failures

When no tests start, look at the command before looking at fixtures. An unknown --qa- option to the left of the separator belongs to CLI parsing. A plain word to the left can narrow file collection. The absence of the run-options.json attachment confirms the fixture never ran, but it does not distinguish those two command mistakes. Preserve the exact sanitized command structure, including the separator.

When TypeScript says argv is missing from FullConfig, check the repository's Playwright package. The API was added in 1.61. Do not silence the type error with as any and assume runtime support. Upgrade through the repository's normal process or use a supported input channel available in its current version.

When local CLI execution works but an IDE run fails, inspect how the IDE launches Playwright. Editor integrations may not include your post-separator values. A wrapper command can provide safe preview defaults, or tests can read a non-secret fallback environment variable, but do not make an implicit production target the fallback. Required run context should fail with a message that includes the accepted invocation shape.

When one shard targets a different tenant, the tests can all pass and the merged result can describe two data sets. Every shard command must receive identical business selectors unless the run intentionally partitions by tenant. Attach sanitized options to each report and validate them before merging. The shard index belongs to Playwright's runner options before --; your tenant and seed remain after it.

Do not expect a reporter invoked during a later merge command to recreate each original shard's argv from the merge process. The merge is a different command with its own process arguments. Preserve run context in the shard's report metadata or attachments at execution time, then compare those recorded values before treating the merged report as one environment. A banner printed only during execution may be lost when someone downloads artifacts without job logs.

The comparison should reject mixed contexts, not merely display them. Read the sanitized environment, tenant, and seed from every shard artifact. Four identical records establish that the merge represents one declared run. A deliberate multi-tenant matrix should produce separate merge groups or include tenant in report identity. Combining them and labeling the result "staging" erases a dimension reviewers need.

Retries receive the same FullConfig.argv snapshot for the run. A retry should not select a new random seed by reparsing the clock. The example uses an explicit integer, so every attempt addresses the same deterministic reference. If parallel workers create data, combine that seed with test identity or worker-safe allocation inside the fixture. Reusing one literal reference across all parallel tests creates collisions unrelated to argument parsing.

A reporter mismatch can expose another failure. If global setup imports one parser version while a built custom reporter resolves an older compiled copy, they can disagree over accepted keys. Keep the helper in one package path, rebuild reporter output with the test code, and include parser contract tests in the same verification. Copying the function into the reporter creates configuration drift.

Print parsed values as JSON during diagnosis. JSON makes empty strings and punctuation visible. Never print the whole config.argv array in a shared log because it contains runner paths and every argument, including any secret somebody passed despite policy. Log only allowlisted, validated, non-sensitive fields.

The trace is usually the wrong first tool for an argv defect. A malformed command can fail before a page exists. Once a test runs against the wrong tenant, the trace helps show the resulting navigation and UI, while run-options.json states what the fixture believed. Compare those two sources before blaming data seeding.

Wire CI without putting secrets on the command line

Keep custom values visible and quoted in the job, then place them after one separator. Use arrays in Bash when values come from variables so spaces do not split unexpectedly. The tenant grammar above rejects spaces, but disciplined quoting still prevents future parser changes from altering shell behavior.

jobs:
  browser-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    env:
      QA_ENVIRONMENT: "staging"
      QA_TENANT: "blue"
      QA_SEED: "17"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Run shard
        shell: bash
        run: |
          custom_args=(
            "--qa-environment=${QA_ENVIRONMENT}"
            "--qa-tenant=${QA_TENANT}"
            "--qa-seed=${QA_SEED}"
          )
          npx playwright test \
            "--shard=${{ matrix.shard }}/4" \
            -- \
            "${custom_args[@]}"

All four shards receive identical custom values. The shard option remains on Playwright's side of the separator. The custom array remains on the consumer side. This layout is readable in review and preserves each value as one shell argument.

Do not replace QA_TENANT with a secret token in the same pattern. GitHub's masking reduces accidental log display but does not change process-argument visibility or what FullConfig.argv exposes to reporters and fixtures. Configure authentication through secrets made available to the application or a dedicated credential fixture, and keep the custom argument as a non-sensitive alias.

Add a preflight command that runs the parser tests and prints the sanitized run banner before launching expensive browsers. The added job time is small compared with discovering after four shards that every test used a misspelled tenant. The parser still validates again in global setup because a unit test does not prove the actual command supplied valid values.

Make that preflight consume the same environment expansion as the browser command. A hard-coded parser-test invocation can pass while the later matrix variable is empty. A small context-only Playwright spec can request the fixture, assert the parsed aliases against the job's public expectations, and exit without opening a page. Run it with the exact custom_args array, then reuse that array for the sharded command.

If the context-only spec creates an attachment, inspect it when a job fails. An empty attachment means the fixture did not complete. No attachment means the command, configuration, global setup, or collection stopped earlier. That difference narrows diagnosis without quoting a version-specific Playwright error message.

Roll out in stages. First, add the shared parser and unit cases while existing environment variables remain authoritative. Second, make one non-blocking job pass equivalent custom arguments and compare the attached run context with existing reports. Third, move consumer fixtures to the typed object. Finally, remove duplicate process.argv loops and old defaults after every supported launch path, including local scripts and scheduled jobs, uses the new contract.

The transition costs code and developer education. A wrapper npm script can hide the long command for daily use, but keep its expansion documented. When a failure happens in CI, engineers need to see which side of -- each value occupies.

Remove legacy readers deliberately. Search for process.argv, old tenant environment variables, reporter-specific parsing, and tests with hard-coded default targets. During one compatibility window, make the old and new sources disagree in a controlled non-production job and confirm the new typed fixture wins according to the migration plan. Then delete the fallback. Leaving both indefinitely means invocation order decides run context again.

Scheduled jobs and editor launches are usually the last callers to migrate. A nightly command hidden in workflow reuse may omit the separator, while a VS Code launch may provide no custom args at all. Inventory launch surfaces before making environment and tenant required. Offer explicit preview-only scripts for developers rather than weakening production safeguards in the parser.

Know when argv is the wrong configuration channel

Do not use FullConfig.argv to choose projects, configure webServer, or set a base URL that must exist during configuration resolution. Use validated configuration input available to playwright.config.ts. Runtime context and runner topology have different lifecycles.

Do not pass passwords, API keys, cookies, private URLs containing credentials, or personal data. Arguments can appear in process listings and logs and are available to every consumer of FullConfig. Use the repository's established secret and authentication mechanisms.

Do not accept arbitrary keys and silently ignore typos inside your namespace. Unknown --qa- names should fail. Silent fallback is how a run aimed at tenant Blue ends up mutating Default while every assertion remains internally consistent.

Do not let each test parse strings independently. One test will accept 17x as a number, another will choose a different default, and a reporter will log the raw value. Parse once through a shared, tested contract and expose typed data.

Do not generate nondeterministic values during parsing. FullConfig.argv is stable, so keep derived run context stable too. Random data allocation belongs in a fixture with collision and replay rules, and its chosen identity should be attached to the test attempt.

Do not add a generic command-line framework for three fixed selectors unless requirements justify it. Every supported syntax creates another shell and validation path. Equals-only namespaced arguments are boring, which is exactly what release configuration should be.

The feature is most useful when a reviewer can read the command and answer three questions: what non-secret context changed, which parser validates it, and where the result is attached. If the value changes Playwright before FullConfig exists or would be dangerous in a process list, choose another channel.