Why a Grid with free slots can still fail under load

The dashboard says twelve slots are available, yet six Chrome jobs are still waiting and two have timed out. Adding another mixed-browser node looks sensible until the new Firefox slots sit empty beside the same queue. The number at the top of the Grid UI was true, but it answered the wrong question. Operations has to measure capacity in the capability class that a request can actually use.

Why total slot count gives the wrong answer

A Grid does not allocate a generic worker. A new-session request enters the Session Queue with a set of capabilities. The Distributor looks for a free slot whose stereotype matches those capabilities, then asks the Node to create the browser session. A Firefox slot cannot satisfy a Chrome request. A Linux Chrome slot may also be unusable when the request requires a custom capability that only exists on another pool.

That matching step is the first reason aggregate utilization lies. Imagine a Grid with eight Chrome slots and eight Firefox slots. All eight Chrome slots are occupied, all Firefox slots are free, and four Chrome requests are queued. Aggregate occupancy is 50 percent. Effective Chrome occupancy is 100 percent, and the next Chrome request has no place to go.

The second reason is that a registered slot is not the same as a productive slot. The Grid status response reports Node availability, sessions, and slots. That is valuable control-plane state, but it does not promise that a browser will start quickly or survive a test. A Node can be reachable while its host is swapping, its browser processes are crashing, or its container runtime is taking forty seconds to create each session. Capacity exists on paper while session-start latency grows.

The third reason is burst shape. An hourly average smooths away the moment your CI scheduler releases a matrix. Forty sessions per hour sounds small. Twenty of them arriving within ten seconds can still exhaust a ten-slot Chrome pool. The relevant question is not “How many tests ran today?” It is “How much compatible work arrived during the time occupied slots could not turn over or new nodes could not become ready?”

Treat each meaningful stereotype as a service class. Browser name is the minimum split. Add browser version, platform, or a custom capability when it changes which slots can accept the work. Do not create dozens of labels merely because capabilities exist. Split on a field only when it creates a real scheduling boundary or a materially different service time.

Four measurements describe that service class better than a node count:

• compatible requests arriving per minute, including retries;
• occupied and free compatible slots;
• time from the new-session call to a returned session ID;
• failed session creations, separated from test failures after creation.

Queue age belongs beside them. A queue length of ten for two seconds may be a normal burst. A queue length of one for four minutes is an incident. The oldest wait tells the operator which case is happening.

Build a capacity picture from the Grid itself

Start with the Grid interfaces before adding host metrics. The status endpoint shows the registered Nodes and their slots. GraphQL exposes the current session count, maximum session count, queue size, session details, Node status, and stereotypes. Those views let you correlate demand with what the Distributor believes it can allocate.

This query is a useful incident snapshot. It asks for queue size, current sessions, and the stereotypes advertised by each Node. Save the raw response with a UTC timestamp so later screenshots do not replace the evidence.

#!/usr/bin/env bash
set -euo pipefail

GRID_URL="${GRID_URL:-http://localhost:4444}"
OUT_DIR="${OUT_DIR:-artifacts/grid-capacity}"
mkdir -p "$OUT_DIR"

stamp="$(date -u +%Y%m%dT%H%M%SZ)"
curl --fail --silent --show-error \
  -H 'Content-Type: application/json' \
  --data '{"query":"{ grid { maxSession sessionCount sessionQueueSize } nodesInfo { nodes { id uri status slotCount sessionCount stereotypes } } }"}' \
  "$GRID_URL/graphql" |
  tee "$OUT_DIR/snapshot-$stamp.json"

curl --fail --silent --show-error "$GRID_URL/status" \
  > "$OUT_DIR/status-$stamp.json"

Do not derive a browser-specific queue from grid.sessionQueueSize. That field is intentionally an aggregate. Query sessionsInfo.sessionQueueRequests when you need the queued payloads, then inspect the requested capabilities. Preserve the unparsed value too. It is your defense against a parser that silently puts an unfamiliar capability into an “other” bucket.

A small status summarizer makes slot imbalance visible during an incident. This example accepts the JSON returned by /status on standard input. It counts occupied and free slots by the browser name in each slot stereotype. It also reports Node availability so a down or draining Node is not mistaken for spare supply.

#!/usr/bin/env python3
import json
import sys
from collections import defaultdict

document = json.load(sys.stdin)
nodes = document["value"]["nodes"]
summary = defaultdict(lambda: {"occupied": 0, "free": 0, "nodes": set()})

for node in nodes:
    availability = str(node.get("availability", "UNKNOWN")).upper()
    node_id = node["id"]
    for slot in node.get("slots", []):
        stereotype = slot.get("stereotype") or {}
        browser = stereotype.get("browserName", "unknown")
        key = f"{browser}:{availability.lower()}"
        summary[key]["nodes"].add(node_id)
        if slot.get("session") is None:
            summary[key]["free"] += 1
        else:
            summary[key]["occupied"] += 1

for key in sorted(summary):
    row = summary[key]
    print(
        f"{key:24} occupied={row['occupied']:2d} "
        f"free={row['free']:2d} nodes={len(row['nodes']):2d}"
    )

A healthy-looking sample might print chrome:up occupied=6 free=2 and firefox:up occupied=1 free=7. If five Chrome requests are waiting, the two free Chrome slots deserve investigation. They may require a different version, platform, or custom capability. They may also be in the short interval between assignment and a session appearing. Take several samples instead of declaring a cause from one frame.

Session-start latency must come from the client side as well. Measure from immediately before constructing RemoteWebDriver until the constructor returns or raises. That interval includes the Router wait, queue time, Distributor work, and Node session creation. It is the user-visible result. A Grid trace can divide that total among components, but a server-side “session created” timestamp alone omits the wait experienced by the caller.

Here is a load probe that records that result without running an application test. It starts a bounded number of Chrome sessions, holds each for a configurable period, and always attempts to quit. Run it only against a non-production pool or during an approved capacity window because it consumes real slots.

#!/usr/bin/env python3
import concurrent.futures
import json
import os
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

GRID_URL = os.environ.get("GRID_URL", "http://localhost:4444")
SESSIONS = int(os.environ.get("SESSIONS", "6"))
HOLD_SECONDS = float(os.environ.get("HOLD_SECONDS", "5"))

def start_one(sequence: int) -> dict:
    driver = None
    started = time.monotonic()
    try:
        options = Options()
        options.add_argument("--headless=new")
        driver = webdriver.Remote(command_executor=GRID_URL, options=options)
        allocated = time.monotonic()
        return {
            "sequence": sequence,
            "outcome": "created",
            "startup_seconds": round(allocated - started, 3),
            "session_id": driver.session_id,
        }
    except Exception as exc:
        return {
            "sequence": sequence,
            "outcome": "failed",
            "startup_seconds": round(time.monotonic() - started, 3),
            "error_type": type(exc).__name__,
            "error": str(exc),
        }
    finally:
        if driver is not None:
            time.sleep(HOLD_SECONDS)
            driver.quit()

with concurrent.futures.ThreadPoolExecutor(max_workers=SESSIONS) as pool:
    results = list(pool.map(start_one, range(SESSIONS)))

for result in results:
    print(json.dumps(result, sort_keys=True))

Record a cold run and a warm run. A cold run includes image pulls, pod scheduling, driver startup, and browser initialization if your platform creates nodes on demand. A warm run measures Grid allocation when supply is already registered. The difference is the headroom an autoscaler or scheduled pre-warming policy has to cover.

Work the incident from symptom to cause

Consider a morning pipeline with twelve registered slots: six Chrome, four Firefox, and two Edge. Eight Chrome jobs arrive together. Six start, two wait, and the UI still advertises six free slots. The queue is not proof of general exhaustion. It is proof that the immediately available capacity cannot satisfy at least one queued request.

Capture sessionQueueRequests and compare the Chrome request with Node stereotypes. Suppose the queued request pins browserVersion: "119" because a suite froze that version months ago, while every registered Chrome Node now advertises 120 or 121. No stereotype can satisfy the pinned value, so adding identical Nodes only creates more nonmatching slots. The diagnostic signature is stable: Chrome-class queue age rises, matching Chrome free slots remain zero, other browsers remain free, and host CPU stays moderate.

Be precise about which version strings actually constrain matching, because not all of them do. Selenium's default slot matcher treats a browser version that is absent, empty, or the literal string stable as match-anything, and only then falls through to a real version comparison. A request carrying browserVersion: "stable" therefore matches every Chrome stereotype regardless of the number that Node advertises, and it can never produce the signature above. Only a concrete value such as 119 narrows the candidate slots enough to strand a request while free slots sit beside it. When a team convention puts a word rather than a number in that field, the resulting incident is usually the opposite one: requests match pools nobody intended them to reach.

The fix is to align the request with an advertised stereotype or provision a pool that advertises the required capability. Reject unsupported capabilities immediately when the Grid is not designed to create nodes on demand, using the documented Distributor option for that behavior. The cost is loss of flexibility. A misspelled or temporarily unavailable capability fails early rather than waiting for future capacity.

A second incident presents differently. Four Chrome slots are free, queued requests ask for ordinary Chrome on Linux, and the stereotypes match. New-session calls take seventy seconds, then return SessionNotCreatedException. Node status continues to report the Node as up. This is not a slot-count problem. Inspect Node logs and a distributed trace for the session-creation span. On the host, correlate browser process exits, memory pressure, container startup duration, and filesystem exhaustion with the same period.

If memory is exhausted, reducing max-sessions can improve throughput even though it lowers the capacity number shown in the UI. Four stable browsers that start in five seconds outperform eight slots that fight for memory and fail half their launches. The cost is visible concurrency. You trade an impressive slot total for fewer retries and a predictable completion rate.

A third incident is a burst problem. Ten Chrome slots finish about one test every three minutes. At the top of the hour, twenty-four jobs arrive within fifteen seconds. No nodes are unhealthy and every request matches. The first ten sessions start, the next fourteen wait, and queue age falls as sessions complete. Daily utilization remains below 40 percent because the Grid is quiet for the rest of the hour.

Adding permanent capacity fixes the peak but pays for idle browsers most of the day. Staggering the CI matrix, limiting job fan-out, or warming temporary capacity before the scheduled burst may be cheaper. Each option moves cost somewhere: a concurrency limit lengthens pipeline wall time, staggered schedules complicate orchestration, and temporary nodes add startup and platform complexity.

Real diagnostic output should form a timeline, not a collage. For the burst case, a reduced record might look like this:

observed_at: "2026-08-04T03:00:20Z"
service_class: "chrome-linux"
new_session_requests_last_30s: 24
compatible_slots:
  occupied: 10
  free: 0
oldest_queued_request_seconds: 18
session_start_seconds:
  p50: 4.8
  p95: 19.4
failed_session_creations: 0
nodes:
  up: 5
  draining: 0
  down: 0

Thirty seconds later, occupied slots may still be ten while queue age reaches forty-eight seconds. That progression supports a capacity diagnosis. If free compatible slots stay above zero while queue age rises, investigate matching, the Distributor, or session creation instead. If Node availability changes to down, investigate health and registration.

Prove which boundary is failing

Begin with the caller's new-session timestamp and outcome. A timeout in the application test after the session exists does not belong in a Grid capacity incident. Neither does a DNS failure that prevented the request from reaching the Router. Keep those failures out of the queue SLO or the number stops describing a single boundary.

Next, ask whether the request appeared in the Session Queue. If it never appears, examine the client URL, authentication, proxy, Router logs, and HTTP response. A request visible in the queue has crossed the Router boundary. Its presence also tells you that repeatedly restarting the test runner will add more work rather than repair the Grid.

Then decide whether a compatible slot existed during the wait. Compare the complete request capabilities with stereotypes, not only browserName. Platform names, versions, and extension capabilities can narrow the candidate pool. Take snapshots over the whole wait because a slot seen free after the timeout may have been occupied when the request was eligible.

When a compatible slot is free, inspect the assignment and creation path. Grid tracing is useful here because a single new-session request crosses the Router, Session Queue, Distributor, and Node. Preserve trace IDs from Grid event logs where available. Look for time spent waiting versus time spent creating the browser. The two delays demand different fixes.

Node state comes next. “Up” means the control plane can see the Node and its status, not that every browser startup is fast. Check the Node's session-creation failures and resource condition. A repeating browser exit, a full temporary filesystem, or a bad driver-browser pairing consumes attempts without producing useful sessions.

Finally, measure the outcome the team cares about. Useful service objectives include “99 percent of supported Chrome requests receive a session within 45 seconds” and “fewer than 1 percent of session-creation attempts fail for Grid-owned reasons.” Define supported requests from the stereotypes you intentionally operate. Otherwise an impossible capability can violate the objective forever.

Avoid an alert on instantaneous queue length. Alert when the oldest supported request exceeds a duration, when session-start latency breaches its percentile target, or when a pool has no healthy compatible slots. Add a short persistence window so a routine burst does not page someone before the first browser starts.

The near-miss is application slowness. A session starts in four seconds, then every test blocks on login for two minutes. Slot occupancy rises, later requests queue, and the Grid looks saturated. More Grid slots may amplify load against the already slow application. Separate “waiting for a session” from “holding a session.” If session duration rises while session-start time remains normal, inspect the tested service and the test flow before scaling browsers.

Another near-miss is a retry storm. A client timeout of thirty seconds can expire while Grid's queue request remains eligible under its own timeout. The runner retries, creating another request for the same logical test. Queue length grows faster than original demand, and dashboards suggest a larger workload than CI intended. Correlate queue entries with test attempt IDs and align client and Grid timeout policies so abandoned work does not dominate the measurement.

Separate real queue pressure from a lost new-session response

There is a second failure that looks almost identical from the runner. The constructor waits until its client-side deadline, the test records a new-session failure, and retry logic submits another attempt. Queue size and occupied slots can both rise. This can be ordinary capacity pressure, but it can also mean Grid created the first session and the successful response never reached the caller. A proxy reset, a network interruption, or a shorter caller deadline can strand a valid session that no test process knows how to use. The retry then asks for another slot and turns a response-path problem into apparent demand.

The separation depends on server outcome, not the exception text at the runner. Give each logical test and each construction attempt distinct correlation values in client logs before the new-session call begins. When a call succeeds, record the returned session ID against that attempt. At the same timestamps, preserve the Node session records and the queue requests. In a genuine shortage, the attempt remains queued while compatible occupancy is full, no new session ID is created for it, and the Node sessionCount changes only when a known waiting job finally receives a session. In the lost-response case, a new session appears on a Node around the failed attempt's deadline, the queue entry disappears, but no runner records receiving that session ID. A retry can then create a second session for the same logical test.

Read sessionCount next to attribution, not by itself. A healthy value is the number of sessions that active jobs can name, allowing for the brief interval while a success response is in flight. A broken value is a count that increases while the set of session IDs acknowledged by runners does not. The startTime on the unmatched session should align with the timed-out attempt. The misleading value is a falling sessionQueueSize: it may mean waiters received sessions, or it may mean a request left the queue by creating an orphan whose response was lost. A client duration equal to its deadline is also ambiguous. It proves when the caller gave up, not whether the remote end completed creation.

Distributed traces can make the boundary explicit when they cover both sides of the request. Compare the client span outcome with the Grid and Node work for the same attempt window. A server-side creation completion followed by an interrupted or undelivered response supports the response-path diagnosis. Repeated Node creation errors without a lasting session support a browser-start diagnosis. A request that stays in the queue without reaching Node creation supports a compatible-supply diagnosis. If traces cannot be joined, use the raw session start time, Node ID, proxy access record, CI attempt time, and the absence of a returned session ID. Do not delete a supposedly unmatched session merely because log ingestion is late. Require a bounded attribution delay and retain the evidence used for cleanup.

Existing suites need observability before retry behavior changes. First, land attempt identifiers and success-side session-ID logging in the common driver factory. Then inventory wrappers that catch construction failures and retry without exposing the original attempt. Shared fixtures, generic retry annotations, and setup helpers that log only the final outcome usually break attribution first. Add a shadow report that lists sessions no runner has claimed, but take no cleanup action. Once delayed logs and legitimate long setups have been measured, canary any retry or orphan-cleanup policy on one service class. The rollout is working when unclaimed sessions fall, attempts per logical test fall, and compatible queue age improves without a rise in false cleanup of live sessions.

Ownership crosses three boundaries. The test-platform team owns attempt identity, client deadlines, and retry semantics. The Grid team owns Node and queue evidence plus any policy for retiring a confirmed orphan. The network or proxy team owns proof that a completed response was reset, timed out, or delivered. A handoff must contain the logical test ID, every attempt start and end time, returned session IDs or their absence, the candidate orphan's ID, start time and Node ID, queue snapshots, trace references, proxy outcome, and all relevant deadlines. “Grid timed out” is not enough because it does not say which side stopped waiting first.

The cost is measurable. Attempt-level traces and session attribution create high-cardinality telemetry, and a conservative attribution delay leaves orphaned browsers consuming slots longer. Aggressive cleanup recovers capacity faster but can terminate a legitimate session whose success log arrived late. Reducing automatic retries lowers amplified demand but exposes more first-attempt failures to the suite and may lengthen recovery while the underlying response path is repaired. Choose the delay and retention policy from observed logging lag, then state that lost capacity in the pool budget.

This method does not catch a session that is created and returned successfully but is unusable for the test. A browser can receive a session ID and still fail to reach the application, load the required certificate, or execute the intended workflow. Queue, attribution, and startup-latency signals all look healthy in that case. A separate functional canary must exercise the browser behavior the release depends on.

Roll out limits and alerts without shocking the suite

Inventory the last two to four weeks of session starts before changing capacity. Group by the capability fields that affect matching. Capture arrival time, allocation time, end time, outcome, and attempt number. Remove synthetic health checks from product-suite demand or label them separately. They often run at a fixed cadence and distort low-volume percentiles.

Establish a baseline per service class. Report median and 95th-percentile session-start time, peak arrivals during a node-startup window, maximum concurrent sessions, session duration, and creation failure rate. Daily averages can remain on an overview dashboard, but they must not drive the limit.

Add a canary probe with bounded concurrency. It should fail clearly when no session can be created, preserve its JSON results, and quit every successful session. A probe that leaks sessions turns monitoring into the incident it is meant to detect.

This CI job runs the earlier probe only when a Grid URL is supplied, stores the Grid status whether the probe passes or fails, and uploads the evidence. The commands are ordinary shell and can be adapted to another CI system without changing the measurement.

name: grid-capacity-canary

on:
  workflow_dispatch:

jobs:
  chrome-session-start:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    env:
      GRID_URL: ${{ secrets.SELENIUM_GRID_URL }}
      SESSIONS: "4"
      HOLD_SECONDS: "2"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python3 -m pip install --requirement requirements.txt
      - name: Capture pre-run status
        run: curl --fail --silent --show-error "$GRID_URL/status" > grid-before.json
      - name: Run bounded session-start probe
        run: python3 tools/grid_capacity_probe.py > session-starts.jsonl
      - name: Capture post-run status
        if: always()
        run: curl --silent --show-error "$GRID_URL/status" > grid-after.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: grid-capacity-evidence
          path: |
            grid-before.json
            grid-after.json
            session-starts.jsonl

Canary one browser pool before changing every Node. If a host currently advertises four slots, lower or raise the limit on one Node class and compare session-start success, host memory, and suite wall time. A one-week observation window is often more informative than a single load test because it includes real test-duration variance.

Add admission control before buying capacity. CI max-parallel limits, suite sharding, and scheduled offsets can keep arrivals inside the service envelope. Document the consequence in minutes of pipeline duration so the team can choose knowingly. A hidden throttle feels like random slowness.

Keep a warm reserve when node startup is slower than the wait objective. If a Chrome Node takes ninety seconds to register and callers expect a session within forty-five seconds, scaling only after a request enters the queue cannot meet that objective from zero. Either retain ready Chrome slots, pre-scale before known bursts, or relax the objective. No autoscaling formula removes startup time.

Review the model after browser upgrades, base-image changes, suite parallelism changes, or major shifts in test duration. Those events alter supply or demand even when the Node count is unchanged. Capacity planning is an operational feedback loop, not an annual spreadsheet.

Know when capacity work is the wrong fix

Do not add nodes when queued capabilities match no advertised stereotype. Fix the request, restore the missing pool, or reject unsupported capabilities. More of the wrong stereotype increases cost and makes the aggregate dashboard look healthier while users continue to wait.

Do not raise max-sessions because host CPU has spare headroom. Browsers consume memory, shared memory, processes, file descriptors, network bandwidth, and sometimes GPU resources. Increase concurrency only after a controlled test shows stable creation and execution at the new level. Selenium's own guidance warns that overriding the processor-based recommendation can hurt stability.

Do not lengthen the Session Queue timeout to make timeout errors disappear. A longer timeout gives a request more opportunity to start; it does not create supply. It also keeps callers and CI workers blocked longer. Change it only when the business is willing to wait and measured recovery or scale-out can complete within that window.

Do not scale the browser tier while the tested application is rate-limiting or failing under the current load. More sessions can worsen the application incident and turn clean 429 responses into harder failures. Compare session-start latency with in-session duration and application response data first.

Do not count draining Nodes as future supply. Draining exists so ongoing sessions can finish without receiving new work. Capacity dashboards should show those slots separately, especially during rolling upgrades. A pool with four occupied draining slots and zero active replacements has no admission capacity even though four browsers are still running.

Do not use a synthetic probe that asks for easier capabilities than production. A generic Chrome check can stay green while the only Windows Chrome pool is down. Probe each service class whose failure would block a release, but keep the number of probes small enough that monitoring does not consume meaningful capacity.

Reliable operation has three different levers: demand shaping, compatible supply, and Node health. Each has a different cost. More supply costs infrastructure. Throttling costs elapsed CI time. Conservative per-Node concurrency costs density. Health isolation and disposable Nodes cost startup time and platform work. The right choice is the one supported by queue, matching, and creation evidence, not the largest slot number on the screen.