Rebalance slow Playwright shards with measured test history

Three of four CI jobs finish in six minutes. The fourth runs for nineteen, so the pull request still waits nineteen minutes and three paid runners sit idle. Test counts look even, but checkout, visual comparison, and report export all landed on the same shard.

Playwright's built-in sharding is the right first step. When duration variance stays high, an external planner can use measured history to assign whole spec files to deterministic test lists. The important word is external. There is no documented Playwright flag that accepts historical timings and automatically balances the next run.

Confirm that allocation is the actual bottleneck

Run the normal shard matrix before building a scheduler. Playwright accepts --shard=current/total, with one-based shard numbers. When fullyParallel: true is enabled, individual tests can be distributed across shards. Without it, whole test files are the usual sharding unit. Finer granularity often improves balance without custom code.

Count balance and duration balance are different. Four shards with 100 tests each can finish at 4, 6, 7, and 19 minutes when a few tests dominate wall time. Conversely, a shard with fewer tests can be slower because it owns a long beforeAll, browser setup, data import, or serial group.

Measure these timestamps separately:

• Job queue start and runner allocation
• Dependency and browser setup
• Playwright process start
• Test execution
• Report compression and upload
• Job completion

TestResult.duration measures one test attempt in milliseconds. It does not include every cost around the runner. A shard can have equal summed test duration and still finish later because its machine downloaded browsers, uploaded more traces, or suffered CPU contention. Historical test assignment cannot repair those causes.

Use an imbalance metric that reflects the release delay. For four completed shard durations, calculate both the slowest-to-fastest ratio and idle runner time. If shards finish in 360, 372, 390, and 1,140 seconds, the ratio is 3.17. The three earlier shards contribute 2,298 runner-seconds of idle time before the gate opens. That is enough waste to justify investigation.

Separate a heavy assignment from a degraded runner

A second failure can produce almost the same headline: three jobs finish near six minutes and shard 4 takes nineteen. In the allocation failure, shard 4 received more expected test work. In the runner failure, the plan was reasonable but that machine executed ordinary work slowly. Moving files based on the second incident bakes infrastructure noise into the next plan and can create a genuinely heavy shard when the runner recovers.

Read the retained plan and timing artifact together. In plan.json, the useful shard fields are estimatedMs and files. In the timing artifact, group the attempt field durationMs by file, retaining retry and status. The estimate and actual sum do not need to match exactly, but the pattern matters. A healthy heavy assignment has a larger estimatedMs before execution, and its files take roughly their usual individual times. A degraded runner starts with an estimate similar to its peers, then most unrelated files on that one job become slower than their own histories.

For an illustrative comparison, suppose all four planned totals lie between 600,000 and 625,000 milliseconds. Shard 4 later records about 1,100,000 milliseconds of first-attempt test time, while checkout, search, and profile specs each take close to twice their recent medians. That broad multiplier points away from one underestimated file and toward the execution environment or a dependency shared by the job. If shard 4 was estimated at 1,050,000 milliseconds and finishes near that value while its individual files remain normal, allocation is the better explanation.

Job timestamps provide the second axis. Compare Playwright process start, first test start, last test end, and job completion. A long interval before the first test implicates runner allocation, dependency restoration, browser setup, or global setup, none of which changes when files are rebinned. A long execution interval accompanied by proportionally slow tests can reflect CPU pressure, memory pressure, network degradation, or a slow shared service. A long interval after the last test points at traces, report compression, or artifact upload. The scheduler owns only the portion that the selected test work explains.

Some fields look diagnostic but are not. workerIndex identifies a worker process lifetime and parallelIndex identifies a parallel slot. Neither reports CPU capacity or proves that two CI jobs ran on equivalent machines. A passed status proves the assertion outcome, not that the duration is representative. Equal files counts are also misleading because files are not equal-cost units. Treat those values as correlation handles, then use per-file history and job-boundary timestamps to classify the delay.

Repeat the comparison across complete runs before changing weights. If the same files remain slow after landing on different runners, their history is stale or their workload changed. If whichever files land on one runner class slow down together, the CI platform team needs the incident. This distinction prevents the planner from compensating for a machine problem it cannot control.

Now inspect whether the same specs repeatedly occupy the tail. A one-off nineteen-minute shard during an application outage is not a scheduling baseline. A checkout spec taking twelve minutes across five healthy main-branch runs is.

Keep retries visible during this check. Playwright reports each retry as a separate TestResult with its own duration and retry index. If a file takes two minutes normally but eight minutes whenever a flaky test retries twice, you have two problems: expected work and instability. Rebalancing can reduce tail latency, but it must not turn flaky retry spend into an accepted constant.

Before adding history, try the lower-cost controls:

1. Enable fullyParallel only if tests are isolated enough for test-level distribution.
2. Split a single oversized spec when its setup and product flow permit independent files.
3. Remove accidental fixed waits and diagnose consistently slow actions in the HTML report or trace.
4. Match worker count to runner CPU and memory instead of oversubscribing every shard.
5. Move browser installation and dependency cache work out of the shard comparison.

Duration-aware planning earns its complexity when healthy test work remains predictably uneven after those changes.

Collect history without losing attempt details

A custom reporter is the cleanest supported collection point. Playwright calls onBegin() after discovering tests and onTestEnd() after each attempt. TestCase supplies the project, source location, and ID. TestResult supplies duration, retry index, status, worker indexes, and errors.

This reporter writes both the current catalogue and every attempt. It aggregates nothing, which keeps policy out of collection. A later planner can decide whether to use first attempts, retries, medians, or another statistic.

// reporters/timing-reporter.ts
import path from 'node:path';
import { mkdir, writeFile } from 'node:fs/promises';
import type {
  FullConfig,
  Reporter,
  Suite,
  TestCase,
  TestResult,
} from '@playwright/test/reporter';

type CatalogEntry = {
  project: string;
  file: string;
};

type Attempt = CatalogEntry & {
  testId: string;
  titlePath: string[];
  durationMs: number;
  retry: number;
  status: TestResult['status'];
  workerIndex: number;
  parallelIndex: number;
};

const portablePath = (file: string) =>
  path.relative(process.cwd(), file).split(path.sep).join('/');

export default class TimingReporter implements Reporter {
  private catalog: CatalogEntry[] = [];
  private attempts: Attempt[] = [];

  onBegin(_config: FullConfig, suite: Suite) {
    const entries = suite.allTests().map(test => ({
      project: test.parent.project()?.name ?? '',
      file: portablePath(test.location.file),
    }));

    this.catalog = [...new Map(
      entries.map(entry => [`${entry.project}\0${entry.file}`, entry]),
    ).values()];
  }

  onTestEnd(test: TestCase, result: TestResult) {
    this.attempts.push({
      project: test.parent.project()?.name ?? '',
      file: portablePath(test.location.file),
      testId: test.id,
      titlePath: test.titlePath(),
      durationMs: result.duration,
      retry: result.retry,
      status: result.status,
      workerIndex: result.workerIndex,
      parallelIndex: result.parallelIndex,
    });
  }

  async onEnd() {
    const output = process.env.PW_TIMING_OUTPUT ??
      'test-results/timings.json';

    await mkdir(path.dirname(output), { recursive: true });
    await writeFile(output, JSON.stringify({
      generatedAt: new Date().toISOString(),
      catalog: this.catalog,
      attempts: this.attempts,
    }, null, 2));
  }

  printsToStdio() {
    return false;
  }
}

The reporter runs in Playwright's reporting process, so ordinary parallel workers can call onTestEnd() without writing the same file independently. In a sharded CI matrix, each job still needs a unique PW_TIMING_OUTPUT. Two machines cannot safely upload to one local path.

Configure the timing reporter beside the blob reporter used for merged shard reports:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: process.env.CI
    ? [
        ['blob'],
        ['./reporters/timing-reporter.ts'],
      ]
    : [['list']],
});

The blob report remains the source for the combined human report. The small timing JSON is planner input. Keeping them separate avoids parsing a reporter's internal archive format and lets the timing schema evolve under repository review.

Generate a current catalogue without executing tests:

PW_TIMING_OUTPUT=test-results/current-catalog.json \
  npx playwright test --list --project=chromium \
  --reporter=./reporters/timing-reporter.ts

During real shard jobs, choose attempt-specific names:

PW_TIMING_OUTPUT="test-results/timings-${SHARD_INDEX}.json" \
  npx playwright test \
  --project=chromium \
  --test-list="test-lists/shard-${SHARD_INDEX}.txt"

Upload every timing file even if a shard fails. Use only complete, trusted main-branch pipelines as future baseline inputs. A partial run stopped by maxFailures lacks observations for later tests and will bias a naïve average.

The raw attempt rows answer questions an aggregate cannot. A file might show 90 seconds on retry zero, 88 seconds on retry one, and status flaky at the test-case level. The planner below uses first-attempt cost. A separate quality report should still flag the retry.

Build deterministic whole-file bins

Whole spec files are a practical scheduling unit. They preserve file-scoped setup, serial groups, and test titles without generating a fragile list of every individual case. The cost is coarser balance: one twelve-minute file cannot be divided across two machines.

A longest-processing-time planner is simple and auditable. Estimate each current spec, sort from heaviest to lightest, then place each file into the currently lightest shard. This greedy method is not mathematically perfect for every input, but it usually removes obvious tails and produces a stable plan when weights and file paths are stable.

The script below reads a fresh catalogue plus any number of timing artifacts. It sums first-attempt durations per project and file within each artifact, uses the median observed total, assigns the median known weight to new files, and writes Playwright test-list files.

// scripts/plan-shards.mjs
import path from 'node:path';
import { mkdir, readFile, writeFile } from 'node:fs/promises';

const [project, countText, catalogPath, ...historyPaths] =
  process.argv.slice(2);
const shardCount = Number(countText);

if (!project || !catalogPath || historyPaths.length === 0) {
  throw new Error(
    'Usage: node scripts/plan-shards.mjs <project> <count> ' +
    '<catalog.json> <history.json...>',
  );
}
if (!Number.isInteger(shardCount) || shardCount < 1) {
  throw new Error('Shard count must be a positive integer');
}

const load = async file => JSON.parse(await readFile(file, 'utf8'));
const median = values => {
  const sorted = [...values].sort((a, b) => a - b);
  if (sorted.length === 0) return 0;
  const middle = Math.floor(sorted.length / 2);
  return sorted.length % 2
    ? sorted[middle]
    : (sorted[middle - 1] + sorted[middle]) / 2;
};

const catalogReport = await load(catalogPath);
const currentFiles = [...new Set(
  catalogReport.catalog
    .filter(entry => entry.project === project)
    .map(entry => entry.file),
)].sort();

if (currentFiles.length === 0) {
  throw new Error(`No tests found for project ${project}`);
}

const observations = new Map();

for (const historyPath of historyPaths) {
  const report = await load(historyPath);
  const totals = new Map();

  for (const attempt of report.attempts) {
    if (attempt.project !== project) continue;
    if (attempt.retry !== 0 || attempt.status === 'skipped') continue;

    totals.set(
      attempt.file,
      (totals.get(attempt.file) ?? 0) + attempt.durationMs,
    );
  }

  for (const [file, totalMs] of totals) {
    const values = observations.get(file) ?? [];
    values.push(totalMs);
    observations.set(file, values);
  }
}

const knownWeights = currentFiles
  .map(file => median(observations.get(file) ?? []))
  .filter(value => value > 0);
const fallbackMs = median(knownWeights) || 1_000;

const weightedFiles = currentFiles
  .map(file => ({
    file,
    estimatedMs: median(observations.get(file) ?? []) || fallbackMs,
    historicalRuns: observations.get(file)?.length ?? 0,
  }))
  .sort((a, b) =>
    b.estimatedMs - a.estimatedMs || a.file.localeCompare(b.file),
  );

const shards = Array.from({ length: shardCount }, (_, index) => ({
  index: index + 1,
  estimatedMs: 0,
  files: [],
}));

for (const item of weightedFiles) {
  const target = shards.reduce((lightest, shard) =>
    shard.estimatedMs < lightest.estimatedMs ? shard : lightest,
  );
  target.files.push(item);
  target.estimatedMs += item.estimatedMs;
}

const outputDir = process.env.PW_SHARD_OUTPUT_DIR ?? 'test-lists';
await mkdir(outputDir, { recursive: true });

for (const shard of shards) {
  const body = shard.files
    .map(item => item.file)
    .sort()
    .join('\n') + '\n';
  await writeFile(
    path.join(outputDir, `shard-${shard.index}.txt`),
    body,
    'utf8',
  );
}

const plan = {
  project,
  fallbackMs,
  shards: shards.map(shard => ({
    index: shard.index,
    estimatedMs: Math.round(shard.estimatedMs),
    files: shard.files,
  })),
};

await writeFile(
  path.join(outputDir, 'plan.json'),
  JSON.stringify(plan, null, 2),
  'utf8',
);
console.log(JSON.stringify(plan, null, 2));

Create four Chromium lists from three recent pipelines:

node scripts/plan-shards.mjs \
  chromium \
  4 \
  test-results/current-catalog.json \
  history/main-4101.json \
  history/main-4102.json \
  history/main-4103.json

Validate coverage before publishing the lists. Balance is irrelevant if one spec disappears. Add this check after assignment and before any writeFile() calls:

const assignedFiles = shards.flatMap(shard =>
  shard.files.map(item => item.file),
);
const assignedCounts = new Map();

for (const file of assignedFiles) {
  assignedCounts.set(file, (assignedCounts.get(file) ?? 0) + 1);
}

const missing = currentFiles.filter(file => !assignedCounts.has(file));
const duplicates = [...assignedCounts]
  .filter(([, count]) => count !== 1)
  .map(([file, count]) => ({ file, count }));
const unknown = assignedFiles.filter(file => !currentFiles.includes(file));

if (missing.length || duplicates.length || unknown.length) {
  throw new Error(JSON.stringify({ missing, duplicates, unknown }, null, 2));
}

This check compares the plan with discovery from the current commit, not with historical filenames. Renamed and deleted specs may remain in history for weeks, but they must not appear in today's lists. Newly discovered specs must appear once even when their weight uses the fallback.

Test-list paths are selection inputs, so keep them repository-relative and use forward slashes as the reporter does. Do not generate shell commands by concatenating filenames. A test list is safer for paths containing spaces or characters the shell would interpret, and Playwright documents the file format directly.

The sample chooses files rather than individual test titles for another practical reason. Titles can be parameterized, duplicated in different suites, or moved between line numbers. A file path remains a stable enough scheduling identity across ordinary edits. The downside is that adding one very slow case changes the entire file's weight only after new history arrives. The nonzero fallback protects brand-new files, while a canary comparison catches a suddenly enlarged existing file.

If file-level setup is the expensive part, summing individual TestResult.duration may underestimate the true file cost because hooks and worker fixtures are not always attributed the way your job wall clock is. Compare the planned file weight with timestamps around the complete spec or with shard totals. When setup dominates, split that setup, model an additional fixed file cost, or keep the affected file isolated. Do not pretend millisecond precision from test results captures work it never measured.

Every CI job receives the generated lists and runs one index with --test-list. Do not also pass --shard to those jobs. The external plan has already selected each job's work, and a second sharding pass would run only a fraction of that list.

Use the same project while collecting, planning, and running. Chromium, Firefox, WebKit, mobile emulation, and authenticated projects can have different duration distributions. One combined weight can balance none of them.

Prove the plan under real CI conditions

The planner should print estimated totals before the matrix starts. A credible plan might look like this:

shard 1: 618000 ms, 17 files
shard 2: 612400 ms, 21 files
shard 3: 620900 ms, 19 files
shard 4: 615300 ms, 24 files
fallback: 28400 ms
new files using fallback: 2

Do not stop at estimated balance. Compare actual Playwright execution and full job duration for each shard. If test sums are balanced but one job remains slow, inspect machine type, worker count, cache misses, report upload, and fixture setup outside test bodies.

Preserve the generated plan.json as a small CI artifact. A developer should be able to answer which files were assigned to shard 4 and which timing observations produced their weights. Dynamic assignments with no retained plan make a failure difficult to reproduce.

Reproduce a shard locally with the exact list:

npx playwright test \
  --project=chromium \
  --test-list=test-lists/shard-4.txt \
  --workers=1

One worker helps diagnose order or shared-state interference. If the list passes with one worker and fails with normal concurrency, the conflict is between parallel workers or external resources. If it fails only when another spec shares the list, minimize the list until the contaminating pair is clear.

Changing assignment can expose hidden dependencies. A test that relied on another file creating an account or warming a cache may fail when the planner moves it. That is not a reason to pin the accidental order. Fix setup ownership, use a project dependency where appropriate, and make each scheduled unit independent.

Compare estimates with actuals by file. A spec estimated at 40 seconds and observed at 400 seconds after a fixture migration needs fresh history. A file that alternates between 30 and 300 seconds needs investigation, not a cleverer median. Capture the input data size, retry count, browser project, and runner class before blaming the planner.

Retry behavior is a common near-miss. The sample plan counts retry zero and stores later attempts separately. Actual shard wall time includes retries. Report both values:

shard 2 first-attempt test time: 10m 14s
shard 2 retry time:              4m 52s
shard 2 setup and artifacts:     1m 03s
shard 2 job wall time:          16m 31s

The allocation is acceptable in this example; retry cost is the tail. Weighting the flaky file at fifteen minutes might equalize tomorrow's bars, but the better action is to fix or quarantine the instability according to team policy.

Watch for stale and poisoned history. Browser upgrades, backend migrations, larger seed data, and worker-count changes can invalidate old timings. An incident run can inflate several specs together. Use a rolling set of trusted main runs, prefer a robust statistic such as median, and record the environment alongside the artifact.

Roll out the scheduler with a cheap rollback

Start with one browser project and the same shard count used today. Generate plans in observation mode for several runs without using them. Compare predicted totals with actual file durations from the existing shards. This catches path mismatches, missing projects, and fallback mistakes before test selection changes.

For an established suite, land the collector before any job consumes generated lists. The first change should produce uniquely named timing artifacts while the existing native shards remain authoritative. Confirm that failed jobs still publish their partial evidence, that complete main runs can be distinguished from partial runs, and that project names and repository-relative paths survive artifact transfer unchanged. Nothing about test selection should depend on this stage.

Land catalogue generation and planner validation next. Keep the resulting lists observational, and compare their union with the catalogue from the same commit. Working-directory differences and artifact fan-in are usually the first integration failures: a path that was valid on the collection machine may not select anything on the execution machine, or one shard artifact may overwrite another before planning. Detect both before asking the blocking matrix to trust the plan.

Only then add the matrix consumer. The consumer must receive the catalogue, plan, and lists produced from the same revision. A plan created from an earlier checkout can pass its own internal validation yet omit a newly added spec. During the first blocking runs, retain the native assignment as a fallback path and compare discovered test inventory in the merged report with the prior baseline. A lower slowest-shard time is not a successful rollout if the report contains fewer intended tests.

The change is working when selection completeness stays constant, fallback-weight use is visible for new files, actual file totals track the direction of their estimates, and the critical-path shard falls across several comparable runs. One fast pipeline can come from a warm cache or a quiet backend. Promotion should depend on repeated complete runs, not the most flattering sample.

Next, run the historical plan on a non-blocking main-branch job. Check four conditions:

1. Every current spec appears in exactly one test list.
2. No deleted spec appears in a list.
3. The merged report contains the expected tests for the selected project.
4. Actual slowest-shard duration improves without a rise in failures or missing artifacts.

Add a planner validation that fails if a current file is absent or duplicated. The sample algorithm constructs from a unique catalogue, so duplicates should indicate a later transformation or CI packaging error. Test selection completeness is more important than perfect balance.

Promote the plan to the blocking matrix only after the non-blocking comparison is stable. Keep a pipeline switch that returns to native --shard=x/y without reverting code. A scheduler depends on history downloads, catalogue generation, planner execution, and test-list artifacts. Any one can fail before tests start.

Update history from trusted main runs, not every pull request. Pull requests may run only changed tests, use temporary feature flags, or execute on fork infrastructure. Feeding those partial measurements into the baseline gives popular files more observations and leaves others stale.

Store enough history to smooth noise without preserving an obsolete architecture forever. Five to ten comparable runs is often a practical starting range. The correct number depends on suite frequency and variance. Trigger an immediate reset or separate baseline when the browser project, runner hardware, worker count, or major fixture design changes.

Name the ongoing costs:

• The reporter and planner need maintenance when Playwright or the repository layout changes.
• History artifacts need storage, validation, and access control.
• Whole-file bins cannot split a single long spec.
• Assignments may change when timings change, exposing isolation defects.
• Debugging requires the exact plan from the failed run.
• A planning step adds latency before shard jobs begin.

The benefit is lower critical-path time and less idle runner spend. Measure both. A two-minute improvement on a suite that runs twice a day may not repay the maintenance; the same improvement across hundreds of pull requests can.

Assign ownership at the artifact boundary

The test-infrastructure owner should own the reporter schema, catalogue rules, weighting policy, and coverage validation. The CI platform owner should own artifact retention, fan-in, matrix delivery, runner classification, and job-boundary timestamps. Product teams still own a spec whose workload or fixture becomes intrinsically slow, and the quality owner for a flaky test owns retry cost. Without that division, an infrastructure incident becomes a planner tweak and a test regression becomes a runner ticket.

A handoff must contain the current catalogue, retained plan, exact test list for the slow shard, raw timing artifacts, retry and status fields, the four job-boundary timestamps, and the environment identity used to choose comparable history. It should also state whether the delay appears before tests, across many ordinary tests, inside one file, or after tests. The receiving team can then reproduce the selected workload and reject the nearest alternative without requesting another run merely to recover missing evidence.

There is a specific maintenance price. Collection adds a reporter artifact to every shard, planning adds catalogue discovery and artifact transfer before execution, and schema changes require the collector and planner to remain compatible while old history ages out. Whole-file scheduling also leaves a hard lower bound equal to the longest indivisible file. If one file takes longer than the target shard duration, more shards increase runner spend without reducing that bound.

This technique does not catch tests that never enter the current catalogue. A project-selection mistake, discovery configuration error, or unintended filter can omit a test before coverage validation begins. The planner may then assign every discovered file exactly once and report perfect completeness while part of the intended suite never ran. Guard the intended inventory separately, using a reviewed expectation or an independent comparison that does not derive from the same faulty discovery pass.

Know when count-based sharding is better

Keep native Playwright sharding when shard durations are already close. It has fewer moving parts, follows the documented runner behavior, and needs no historical artifact.

Prefer fullyParallel when tests are isolated and the imbalance comes from uneven counts inside large files. It gives the runner finer units without an external scheduler. Do not enable it merely for speed if file-scoped order or shared state is still part of the suite.

Do not build history around a single giant spec. Split or redesign that spec first. No bin-packing algorithm can divide an indivisible twelve-minute file across four machines.

Avoid historical planning for a suite whose duration is dominated by random backend incidents or frequent retries. Stabilize the tests and environment before optimizing assignment.

Skip it when CI provides a dynamic work queue that already assigns tests to available workers. Static bins cannot react when one runner slows down mid-run; dynamic scheduling can, though it brings service cost and different reproducibility trade-offs.

Do not use the planner to preserve order dependencies. A stable list can make dependent tests look reliable until the next file moves. Each test or intentional serial unit must own its state.

Finally, do not claim that balanced test sums equal balanced job cost. Browser installation, global setup, workers, traces, uploads, and runner contention still decide the wall clock. Historical file durations solve one well-defined allocation problem. Use them only after the evidence shows that problem is the longest pole.