Plan Grid capacity for the burst, not the daily average

A nightly suite averages forty sessions an hour, so eight Grid slots look generous. At 02:00, the scheduler releases twenty-four Chrome tests together and the last group waits longer than the tests themselves. The average described a quiet hour; it did not describe the queue the release team experiences.

Convert the workload into capability-specific demand

Grid capacity is not a count of machines. It is the rate at which compatible slots can accept and complete sessions while meeting a wait target. The Session Queue holds new-session requests until the Distributor can match them to free slot stereotypes. That means both demand and supply must be grouped by the capabilities that affect matching.

Start with browser name. Add browser version, platform, or a custom capability only when it changes the candidate Nodes. A request for Windows Chrome cannot consume an idle Linux Chrome slot. A request for Firefox cannot consume either. Reporting one global utilization percentage turns those scheduling boundaries into invisible waste.

Three timestamps define the user's wait:

• when the client begins the new-session request;
• when a session ID returns;
• when the request fails or the client stops waiting.

Two more define slot service time:

• when the session is allocated;
• when quit or forced termination releases it.

Session-start time and session duration answer different questions. Long startup with free compatible slots points to browser creation, Node health, or the allocation path. Long duration after a quick start means slots turn over slowly. Both can create a queue, but they have different fixes.

For a steady first estimate, Little's Law is useful: average concurrent sessions are arrival rate multiplied by average slot-hold time. If Chrome receives six sessions per minute and each holds a slot for two minutes, the average offered concurrency is twelve. Twelve is not yet a production capacity recommendation. Bursts, duration variance, failures, retries, rolling maintenance, and startup delay all require headroom.

Do not multiply every worst-case number together. A 95th-percentile arrival rate times a 95th-percentile duration assumes the two extremes always coincide. It can buy a large idle fleet without proving the wait objective. Replay observed arrivals with observed durations, preserve correlations where possible, and test candidate slot counts.

Define the objective before the model. “No queue” is expensive and often unnecessary. “95 percent of supported pull-request sessions start within 30 seconds, and 99 percent within 90 seconds” is testable. A nightly regression suite may accept a two-minute wait if its overall finish time stays inside the release window. Different workloads can share Nodes, but their service targets should remain visible.

Retries are new demand. If a runner abandons a request at sixty seconds and retries while Grid still retains the first request under its queue policy, one logical test can contribute two queued requests. Record attempt IDs and align timeout policies before sizing from the queue. Otherwise the capacity plan buys browsers for an avoidable retry storm.

Collect measurements that can reproduce the queue

Instrument the client first because it knows how long the caller waited. Use a monotonic clock for durations and UTC timestamps for correlation. Emit one JSON line when a new-session call succeeds or fails, then another when the session closes. Include the requested capability class, CI run, shard, and attempt.

This Python context manager records creation and slot-hold timing around a real Remote WebDriver session. It always attempts to quit and keeps product assertions outside the measurement helper.

import json
import os
import time
from contextlib import contextmanager
from datetime import datetime, timezone

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def emit(event: dict) -> None:
    print(json.dumps(event, sort_keys=True), flush=True)

@contextmanager
def measured_chrome_session(test_id: str):
    options = Options()
    options.add_argument("--headless=new")
    request_started = time.monotonic()
    requested_at = datetime.now(timezone.utc).isoformat()

    try:
        driver = webdriver.Remote(
            command_executor=os.environ["SELENIUM_GRID_URL"],
            options=options,
        )
    except Exception as error:
        emit({
            "event": "session_create_failed",
            "test_id": test_id,
            "requested_at": requested_at,
            "browser": "chrome",
            "startup_seconds": round(time.monotonic() - request_started, 3),
            "error_type": type(error).__name__,
            "error": str(error),
        })
        raise

    created = time.monotonic()
    session_id = driver.session_id
    emit({
        "event": "session_created",
        "test_id": test_id,
        "requested_at": requested_at,
        "browser": "chrome",
        "startup_seconds": round(created - request_started, 3),
        "session_id": session_id,
        "ci_run_id": os.environ.get("CI_RUN_ID"),
        "attempt": int(os.environ.get("TEST_ATTEMPT", "1")),
    })

    try:
        yield driver
    finally:
        held_seconds = time.monotonic() - created
        try:
            driver.quit()
        finally:
            emit({
                "event": "session_closed",
                "test_id": test_id,
                "session_id": session_id,
                "held_seconds": round(held_seconds, 3),
            })

The helper keeps startup and slot hold separate. Its startup_seconds value includes queue and browser creation, while held_seconds begins after the session exists and stops at the moment the helper is about to issue driver.quit(), before that call goes out. The delete-session round trip and the Node's own teardown are therefore excluded, so the recorded hold is a small underestimate of true slot occupancy. That is usually what you want when the question is how long a test used a browser. Move the measurement to after driver.quit() returns when the question is how long the slot was unavailable to anyone else, and keep the two definitions apart in the plan rather than mixing them across suites. Combine those client events with Grid session data when you need the Node's exact allocation and release timestamps. Do not subtract timestamps from hosts whose clocks are not synchronized. Monotonic durations are safe inside one process; cross-system timelines need disciplined time synchronization.

Sample the Grid during known bursts. GraphQL exposes queue size, queued request payloads, session details, Node status, and stereotypes. The status endpoint provides another view of registered Nodes and slots. Store raw responses, not only dashboard aggregates, so a future capability parser can be corrected.

The collection interval must be shorter than the event you want to see. A five-minute scrape will miss a queue that grows and clears in ninety seconds. Five or ten seconds is reasonable for a short controlled test, while a production interval should account for monitoring cost and Grid load.

Read each capture as a timeline, not as a collection of independent totals. In a healthy bounded burst, sessionQueueSize may move above zero while sessionCount approaches maxSession, then fall as completed sessions release slots. The important shape is turnover: queued requests leave at roughly the same moments that compatible sessions end, and the queue returns to its baseline without a wave of creation failures. A capacity shortage has the same early shape, but the compatible pool stays occupied, its Nodes remain available, and matching requests continue to arrive faster than slots turn over.

A Node registration loss can produce nearly identical client logs. Callers report long session creation followed by a timeout, and the global queue rises, just as it does during an undersized burst. The separating evidence appears immediately before the rise. With genuine demand pressure, the advertised capacity for the requested stereotype remains stable and occupied. With registration loss, a Node disappears or becomes unavailable, the corresponding stereotypes and slots vanish from the raw status view, and several queued requests become stranded at the same timestamp even though arrival rate did not increase. Buying permanent capacity treats the wrong cause. The platform team must repair Node health, registration, or the rollout that removed supply.

Global values can be actively misleading. A sessionCount below maxSession looks like headroom, but the unused slots may advertise Firefox while every queued payload requests Chrome on another platform. A zero sessionQueueSize is also not proof of a healthy window if clients already timed out or stopped waiting. Pair it with creation results and attempt IDs. The most useful diagnostic record therefore keeps the queued request capabilities beside Node stereotypes and client outcome. Healthy means a compatible slot becomes free and the request receives a session ID. Broken capacity means compatible occupancy remains at its limit until turnover. Broken registration means that limit itself falls before the wait begins.

Collect these fields for at least two representative delivery cycles:

• new-session request and result timestamps;
• complete requested capabilities;
• session ID and owning Node when created;
• session end time and end reason where available;
• queued request samples and queue size;
• Node availability, stereotypes, and slot occupancy;
• CI matrix fan-out, retries, cancellations, and scheduler limits;
• Node scheduling, startup, and Grid registration times.

Separate product test failures from session-creation failures. A failed checkout assertion still held a browser slot. A SessionNotCreatedException may consume creation effort without ever holding one for a test. A network error before the Router may not enter the queue at all.

Simulate the burst instead of averaging it away

A discrete-event replay can answer a practical question: with N compatible slots, how long would each observed request wait if sessions held slots for their measured durations? It does not need to model every Grid internal. It needs arrival time, duration, and the correct service class.

Save a CSV with arrival_seconds relative to the start of a window, duration_seconds for slot occupancy, and browser. This script filters one browser, assigns each request to the next available slot, and reports wait percentiles. It assumes every request can use every slot in that selected pool, so split further when version or platform matters.

#!/usr/bin/env python3
import argparse
import csv
import heapq
import math

def percentile(values: list[float], fraction: float) -> float:
    ordered = sorted(values)
    index = max(0, math.ceil(len(ordered) * fraction) - 1)
    return ordered[index]

parser = argparse.ArgumentParser()
parser.add_argument("events_csv")
parser.add_argument("--browser", required=True)
parser.add_argument("--slots", type=int, required=True)
args = parser.parse_args()

if args.slots < 1:
    raise SystemExit("--slots must be at least 1")

with open(args.events_csv, newline="", encoding="utf-8") as source:
    rows = [
        row for row in csv.DictReader(source)
        if row["browser"] == args.browser
    ]

events = sorted(
    (
        float(row["arrival_seconds"]),
        float(row["duration_seconds"]),
    )
    for row in rows
)
if not events:
    raise SystemExit(f"no events found for browser {args.browser!r}")

available_at = [0.0] * args.slots
heapq.heapify(available_at)
waits = []

for arrival, duration in events:
    next_free = heapq.heappop(available_at)
    started = max(arrival, next_free)
    waits.append(started - arrival)
    heapq.heappush(available_at, started + duration)

print(f"browser={args.browser}")
print(f"requests={len(events)}")
print(f"slots={args.slots}")
print(f"wait_p50_seconds={percentile(waits, 0.50):.1f}")
print(f"wait_p95_seconds={percentile(waits, 0.95):.1f}")
print(f"wait_p99_seconds={percentile(waits, 0.99):.1f}")
print(f"wait_max_seconds={max(waits):.1f}")

Run it across a range of slot counts and plot cost against wait. The first count that meets the service objective is a candidate, not the final answer. Remove one Node's worth of slots and rerun to model maintenance or failure. Add measured Node startup to any scenario that depends on reactive capacity.

Preserve correlations when possible. If long video tests always launch in the same release stage, randomly shuffling durations among arrivals understates the worst period. If retries arrive only after a particular Grid slowdown, sampling them independently can exaggerate normal demand and understate incident demand at the same time.

Model separate workload classes when their priorities differ. Pull-request smoke tests may need fast admission, while a nightly suite can wait. A single FIFO queue and shared pool can allow a large nightly burst to delay release checks. Capacity alone may not provide priority isolation. Scheduling the suites at different times or operating separate pools can be clearer.

Work three plans with different constraints

Take the twenty-four-test nightly burst. All requests arrive within twenty seconds, each Chrome session holds a slot for about three minutes, and the pool has eight slots. The first eight start. The next eight wait roughly one service time. The last eight wait roughly two. The daily average stays low because nothing else runs for most of the hour.

Twenty-four warm slots nearly eliminate Grid wait, but they idle after the burst. Eight slots with CI max-parallel=8 also eliminate the Grid queue, yet the remaining tests wait in CI and the job finishes in three waves. A scheduled increase to sixteen slots creates two waves and can be a useful compromise. The choice is cost versus release elapsed time, not correct versus incorrect engineering.

Now consider a Grid with twelve Chrome slots and eight Firefox slots. Eight Firefox slots are idle while sixteen Chrome requests arrive. Aggregate free capacity is 40 percent, but Chrome has a four-request deficit. Buying another mixed Node helps only if it advertises Chrome slots. The plan should say “four more Chrome-current slots for this window,” not “increase Grid by 20 percent.”

The evidence that distinguishes this from a broken matcher is the free compatible count. If four Chrome-current slots are free while requests with the same capability class wait, capacity is not the first suspect. Inspect the complete capabilities, Node status, Distributor path, and browser creation. A simulator that assumes those free slots are usable will produce a false plan.

The third workload has a long tail. Most sessions finish in ninety seconds, but ten percent run video exports for twelve minutes. An average-based estimate may look reasonable until two long tests occupy half a small pool and a release burst arrives. Adding permanent slots covers the tail. Moving export tests to a separate schedule or replacing browser-idle portions with API calls can reduce demand instead.

Isolation has a cost. A separate long-test pool may sit idle and adds another browser image and dashboard. Refactoring the tests changes coverage boundaries. Permanent headroom costs compute. Use the replay to quantify each option rather than arguing from the longest test in the suite.

A fourth plan covers maintenance reserve. Twelve Chrome slots meet the normal p95 wait, but those slots live on three four-slot Nodes. During a rolling browser upgrade, draining one Node removes a third of the pool before its replacement registers. Replaying the burst with eight slots shows whether the service target survives that ordinary event.

You can keep four extra warm slots, replace before draining, or accept a temporary lower objective during a published maintenance window. Warm reserve costs continuously. Replace-first rollouts briefly run extra capacity and need platform headroom. A relaxed objective costs developer time at the moment changes are already risky. The plan should include one of these choices rather than assuming every Node is always available.

Model correlated loss as well. Two Nodes on one host or availability zone can disappear together. Subtracting one arbitrary slot understates that failure. Group slots by their real failure domain and replay the loss of the largest group the service claims to tolerate.

A useful planning record shows assumptions and results together:

window: "nightly-regression-02:00Z"
service_class: "chrome-linux-current"
observations:
  requests: 24
  arrival_span_seconds: 18
  session_duration_seconds:
    p50: 174
    p95: 228
  node_ready_seconds:
    p95: 86
objective:
  session_start_p95_seconds: 60
candidate:
  warm_slots: 16
  ci_max_parallel: 16
simulation:
  wait_p50_seconds: 0
  wait_p95_seconds: 181
result: "does_not_meet_objective"
next_candidate: "24 warm slots or an earlier staggered release"

The numbers are illustrative, but the rejection is important. Capacity documents should keep failed candidates. Otherwise a later reviewer cannot tell why sixteen slots were chosen or why the same unsuccessful proposal returned.

A near-miss appears when application slowness increases session duration. The queue grows because browsers stay occupied, not because Grid changed. More slots can increase load on the slow application and make the duration worse. Compare session-start time, in-session duration, and application latency before approving supply. Sometimes the capacity plan should throttle browsers until the product incident is fixed.

Another near-miss is unsupported demand. One request asks for a browser version no Node advertises. It can wait while ordinary slots remain free. No reasonable slot count fixes the exact request. Reject or correct it, or restore the missing stereotype pool.

Read the actual value before filing it as a mismatch, because only some of them constrain matching. Selenium's default slot matcher ignores a browser version that is absent, empty, or the literal string stable, so a request carrying stable matches every Chrome stereotype and can never be the request that is stranded. A concrete number is what strands work: browserVersion: "119" against a fleet advertising 120 and 121 waits until the queue times out while free Chrome slots sit beside it.

Roll out capacity and admission changes safely

Begin with observation. Instrument session timing and capture Grid state without changing concurrency. Publish p50, p95, and p99 startup wait by capability class alongside session duration and failure count. Review individual slow attempts so percentiles do not hide a parser or correlation error.

Replay at least one busy week. Test candidate counts with normal Nodes, one Node unavailable, and the slowest measured startup. Include retries as separate arrivals, then run a second scenario with retry duplication removed. The difference tells you how much capacity is serving policy rather than original work.

Put confidence around sparse data. A pool used only during monthly release testing may have too few bursts for a stable p95. Keep the individual observations, run the known worst release shape, and label the recommendation provisional. Inventing a precise percentile from twenty sessions creates false confidence.

Demand forecasts also change when teams see more capacity. A new Grid limit often leads suites to raise their own parallelism. Record the CI concurrency assumed by the plan and add an alert when observed fan-out exceeds it. Otherwise the new supply can be consumed immediately and appear undersized even though the workload contract changed.

Apply admission control before or alongside infrastructure changes. It is the fastest reversible way to stop an unbounded matrix from overwhelming a fixed pool. This GitHub Actions job caps a sixteen-shard Chrome matrix at eight concurrent jobs. The other eight wait in CI, where ownership and cancellation are usually clearer than in the Grid queue.

name: chrome-regression

on:
  workflow_dispatch:

jobs:
  chrome-shard:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      max-parallel: 8
      matrix:
        shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
    env:
      SELENIUM_GRID_URL: ${{ secrets.SELENIUM_GRID_URL }}
      SHARD_TOTAL: "16"
    steps:
      - uses: actions/checkout@v4
      - run: ./gradlew test
        env:
          SHARD_INDEX: ${{ matrix.shard }}

The cost appears in job wall time. Report it. If two waves add six minutes but avoid twenty minutes of random timeouts, the throttle is valuable. If the release target cannot accept six minutes, provision more compatible slots or shorten sessions.

Change one browser pool at a time. Increase or reduce per-Node concurrency only after a controlled host test. Browser processes compete for memory, shared memory, CPU, descriptors, and disk. Selenium documents that overriding its recommended maximum sessions can harm reliability. More advertised slots are not useful when browser creation starts failing.

Run the original burst against the candidate pool. This capture loop records queue size, current sessions, and maximum sessions every five seconds while the CI wave runs. It creates a simple timeline instead of relying on the final empty queue.

#!/usr/bin/env bash
set -euo pipefail

GRID_URL="$SELENIUM_GRID_URL"
OUT="artifacts/queue-timeline.jsonl"
mkdir -p "$(dirname "$OUT")"
: > "$OUT"

for sample in $(seq 1 120); do
  observed_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  payload="$(
    curl --fail --silent --show-error \
      -H 'Content-Type: application/json' \
      --data '{"query":"{ grid { sessionQueueSize sessionCount maxSession } }"}' \
      "$GRID_URL/graphql"
  )"
  python3 - "$observed_at" "$payload" >> "$OUT" <<'PY'
import json
import sys

observed_at = sys.argv[1]
document = json.loads(sys.argv[2])
grid = document["data"]["grid"]
print(json.dumps({"observed_at": observed_at, **grid}, sort_keys=True))
PY
  sleep 5
done

Success requires more than a lower maximum queue. Compare client wait percentiles, end-to-end pipeline time, session-creation failures, Node resource pressure, application error rate, and cost. A change that empties the queue by failing creation attempts is not capacity improvement.

Roll back when browser creation failures rise, Node health becomes unstable, application rate limits trigger, or session wait misses the objective. Keep the old CI limit and Node configuration ready. Capacity changes are operational releases and deserve the same reversible discipline as application changes.

For an existing suite, land correlation before changing either demand or supply. First make CI run, shard, attempt, capability class, and session result visible in one record. Next remove duplicate retries from the planning data and set the initial admission limit to the concurrency the current pool has already survived. Only then canary a capacity change for one stereotype and one delivery window. The first break is often not a longer queue. It is browser creation failing under higher per-Node pressure, the tested application rejecting the extra traffic, or a legacy job exceeding its wall-clock budget after admission moves waiting back into CI. Keep those three signals beside queue wait so a green queue chart cannot approve a harmful rollout.

This order has a measurable operating cost. Short-interval raw captures create monitoring traffic and retain much more data than one dashboard percentile. A CI limit leaves paid runners waiting and can extend the critical path. A canary pool temporarily duplicates image maintenance and may require spare host capacity. Those costs buy attribution and rollback, not more test coverage, so remove temporary capture frequency and duplicate pools after the decision while retaining the event fields needed for incident correlation.

Ownership should follow the evidence boundary. The test-platform owner supplies the arrival trace, capability grouping, acceptable wait, retry semantics, and a reproducible burst. The CI owner controls matrix fan-out, cancellation, and admission. The Grid or infrastructure owner controls Node availability, stereotypes, host pressure, and replacement timing. When the symptom crosses those teams, the handoff needs the UTC window, CI run and attempt IDs, full requested capabilities, session IDs that were created, raw queue and Node samples, the affected failure domain, and the before-and-after service objective. A screenshot of a queue graph is insufficient because it omits the request class and the supply that could actually serve it.

Review after browser image upgrades. A larger image can double cold-start time. A browser release can use more memory and make the old per-Node concurrency unsafe. Test-suite refactors can shorten or lengthen slot occupancy. CI scheduler changes can shift a smooth arrival stream into a burst without changing test count.

Know when queue capacity is not the problem

Do not buy slots when queued requests match no stereotype. Restore or correct the capability class. Aggregate capacity can be abundant while usable capacity for that request is zero.

Do not increase --session-request-timeout as a capacity plan. The option governs how long queued new-session requests may wait. A larger value can accommodate measured turnover or node startup, but it also holds callers longer and may retain work they have abandoned.

Do not raise the Distributor's session-creation thread pool blindly. That setting controls concurrent creation work, not browser slot supply. Excess threads can add context switching, and a constrained Node or container platform remains constrained. Investigate traces and creation timing before tuning internal concurrency.

Do not size Grid in isolation from the tested application. Browser supply can become a load generator. Respect application test-environment limits, account quotas, data locks, and third-party rate limits. The fastest Grid is useless if it makes every checkout test fail with 429 responses.

Do not operate a separate always-on pool for a tiny, rare capability without comparing on-demand startup and cloud alternatives. Warm capacity gives predictable latency but may be mostly idle. A longer accepted wait can be the economically correct choice for low-priority tests.

Do not use a simulation as proof of production behavior. It simplifies matching, creation failures, retry timing, and correlated infrastructure events. Validate with a controlled burst and keep measured error bars around the plan.

Capacity replay does not catch a browser or application correctness regression. A session can start inside the target wait and then render the wrong page, lose network responses, or fail an assertion. Keep functional results and command-level health outside the admission decision. This technique answers whether compatible sessions start in time, not whether those sessions behave correctly after allocation.

Every lever moves waiting or cost. More slots move waiting out of the queue and into the infrastructure bill. CI throttles move it into the scheduler. Scheduled waves move it into orchestration. Test refactoring moves effort into engineering. Longer timeouts move it into runner occupancy. A credible plan states where the wait went and proves that the new location is acceptable.