Your Grid is healthy, but every browser request is waiting

The Grid status page is green, yet a new Chrome session takes four minutes to start. Ten more requests arrive while the team watches, and the ordinary health check still returns 200. Nothing is contradictory here. Health, queue pressure, and capability compatibility are separate facts, and a useful alert has to measure the latter two directly.

Ask GraphQL questions that match the incident

A new WebDriver session does not go straight from the Router to a browser. The Router places the request in the New Session Queue. The Distributor consumes queued work, compares requested capabilities with available slot stereotypes, and assigns a matching Node. Existing-session commands follow another path and do not wait in this queue.

That sequence explains why “Grid is up” is a weak capacity check. A Router can answer readiness while every matching Chrome slot is occupied. A Node can be UP while all its slots are busy. Ten Linux Chrome slots can be free while one macOS Safari request waits forever because none of them match it.

Selenium exposes the relevant state through the Router's /graphql endpoint. The documented schema includes:

• grid.sessionQueueSize for the count waiting now;
• grid.sessionCount and grid.maxSession for broad utilization;
• sessionsInfo.sessionQueueRequests for the queued request payloads;
• nodesInfo.nodes with status, sessionCount, maxSession, URI, and stereotypes.

Queue size alone is a symptom. Duration tells you whether the symptom matters. Request payloads help explain compatibility. Node fields show whether the apparent shortage is global, limited to one browser class, or caused by Nodes leaving service.

Begin with one small query. Avoid asking for every session field on every poll just because GraphQL permits it. The monitoring loop should be cheap enough to run every 15 or 30 seconds without becoming noticeable Grid traffic.

set -euo pipefail

GRID_URL="$1"

curl --fail-with-body --silent --show-error \
  -H 'Content-Type: application/json' \
  --data '{"query":"{ grid { sessionQueueSize sessionCount maxSession } nodesInfo { nodes { id uri status sessionCount maxSession stereotypes } } sessionsInfo { sessionQueueRequests } }"}' \
  "$GRID_URL/graphql" | python3 -m json.tool

The query uses documented field names. A successful HTTP response is not enough, because GraphQL can return HTTP 200 with an errors array. The consumer must reject that response as unknown rather than reading missing data as zero.

A quiet snapshot might look like this. GraphQL returns every selection the query asked for, so nodesInfo appears even when nothing is queued. A response that is missing a requested selection is a sign of a different query or a proxy rewriting the body, not of a quiet Grid.

{
  "data": {
    "grid": {
      "sessionQueueSize": 0,
      "sessionCount": 6,
      "maxSession": 12
    },
    "nodesInfo": {
      "nodes": [
        {
          "id": "7a2d1dc2-7f91-4e80-b491-6c9cf9eb8d15",
          "uri": "http://selenium-node-chrome-1:5555",
          "status": "UP",
          "sessionCount": 6,
          "maxSession": 12,
          "stereotypes": "[{\"slots\":12,\"stereotype\":{\"browserName\":\"chrome\",\"platformName\":\"linux\"}}]"
        }
      ]
    },
    "sessionsInfo": {
      "sessionQueueRequests": []
    }
  }
}

Note that stereotypes arrives as a JSON-encoded string rather than a nested object. Parse it as a second step, and keep the raw text in your snapshot so a later parser change can be applied to old evidence.

During pressure, the useful evidence is not a screenshot of a dashboard taken later. Store timestamped snapshots. At minimum retain the queue count, queued payloads, Node identity and status, Node session counts, and the Grid's aggregate session values. Those records let an incident reviewer reconstruct whether demand rose, capacity fell, or requests stopped matching.

Do not treat maxSession minus sessionCount as a count of compatible slots. It is only broad headroom. Node stereotypes describe what each slot can accept. A Grid with aggregate headroom can still have zero capacity for a particular request. Exact capability matching is Grid logic; monitoring should present the queued request and candidate stereotypes rather than implementing a casual, incomplete copy of the matcher.

The first alert should answer one operational question: “Have session requests been waiting long enough that a human or autoscaler should act?” Diagnosis can then classify the cause. Trying to encode every cause in a single threshold usually produces either noise or silence.

Build a sampler that fails honestly

The sampler below uses only Python's standard library. It posts the documented query, rejects non-JSON responses, rejects GraphQL errors, checks the expected types, and writes one compact JSON object per run. It does not require a Selenium client library because this is an HTTP monitoring call, not a browser session.

#!/usr/bin/env python3
import argparse
import datetime as dt
import json
import sys
import urllib.error
import urllib.request

QUERY = """
{
  grid {
    sessionQueueSize
    sessionCount
    maxSession
  }
  sessionsInfo {
    sessionQueueRequests
  }
  nodesInfo {
    nodes {
      id
      uri
      status
      sessionCount
      maxSession
      stereotypes
    }
  }
}
"""

def fetch(grid_url: str, timeout: float) -> dict:
    body = json.dumps({"query": QUERY}).encode("utf-8")
    request = urllib.request.Request(
        grid_url.rstrip("/") + "/graphql",
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        if response.status != 200:
            raise RuntimeError(f"GraphQL HTTP status {response.status}")
        payload = json.load(response)

    if payload.get("errors"):
        raise RuntimeError("GraphQL errors: " + json.dumps(payload["errors"]))

    data = payload.get("data")
    if not isinstance(data, dict):
        raise RuntimeError("GraphQL response has no data object")

    grid = data.get("grid")
    nodes_info = data.get("nodesInfo")
    sessions_info = data.get("sessionsInfo")
    if not isinstance(grid, dict):
        raise RuntimeError("GraphQL response has no grid object")
    if not isinstance(nodes_info, dict):
        raise RuntimeError("GraphQL response has no nodesInfo object")
    if not isinstance(sessions_info, dict):
        raise RuntimeError("GraphQL response has no sessionsInfo object")

    queue_size = grid.get("sessionQueueSize")
    if not isinstance(queue_size, int):
        raise RuntimeError("grid.sessionQueueSize is not an integer")

    return {
        "observedAt": dt.datetime.now(dt.timezone.utc).isoformat(),
        "queueSize": queue_size,
        "sessionCount": grid.get("sessionCount"),
        "maxSession": grid.get("maxSession"),
        "queueRequests": sessions_info.get("sessionQueueRequests", []),
        "nodes": nodes_info.get("nodes", []),
    }

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--grid-url", required=True)
    parser.add_argument("--timeout", type=float, default=5.0)
    args = parser.parse_args()

    try:
        print(json.dumps(fetch(args.grid_url, args.timeout), separators=(",", ":")))
        return 0
    except (urllib.error.URLError, json.JSONDecodeError, RuntimeError) as error:
        print(f"grid_graphql_sample_failed: {error}", file=sys.stderr)
        return 2

if __name__ == "__main__":
    raise SystemExit(main())

Run it once from the same network location as the monitoring service:

python3 grid_snapshot.py --grid-url http://grid-router:4444 --timeout 5 \
  >> grid-queue-snapshots.ndjson

If it exits 2, alert or log a monitoring-path failure. Do not append a fabricated sample with queueSize zero. A blank graph during a Router outage should remain visibly unknown. Otherwise the capacity alert clears at the exact moment observability disappears.

Keep authentication and network exposure in mind. Selenium's documentation cautions against exposing the Router widely. Put the sampler inside the trusted network, and use the same authentication controls configured for the Grid. Do not publish /graphql to the internet merely to satisfy an external monitoring service.

Timestamp at the sampler, not only at log ingestion. Ingestion can be delayed, batched, or reordered. UTC timestamps make it possible to compare queue state with CI job submission, Node lifecycle events, and Grid logs without arguing about time zones.

Turn samples into a duration-based alert

A queue of one for fifteen seconds is normal in many Grids. A test runner can request several sessions at once, and the Distributor needs time to start browsers. Paging on any nonzero count trains the team to ignore the alert.

A better rule has three inputs:

1. a queue threshold, such as at least three waiting requests;
2. a duration, expressed as consecutive valid samples;
3. context attached to the notification, including aggregate sessions, Node status, and queued request payloads.

The evaluator below reads newline-delimited snapshots from standard input. It emits an alert only after the threshold is met in four samples that are consecutive both in the stream and in time. Dropping failed observations is necessary but not sufficient: omission keeps a bad value out of the stream, yet it does nothing to bound the streak in time. If the sampler dies for an hour, the four surviving lines are still adjacent, and a counter that never looks at observedAt will report a sustained minute of pressure that actually spanned an hour. The evaluator therefore compares each sample's timestamp with the previous one and restarts the streak when the gap falls outside the expected sampling interval. Without that check, the duration the alert claims to measure is fiction. In a production service, persist state across process restarts or use your monitoring system's duration operator.

#!/usr/bin/env python3
import argparse
import datetime as dt
import json
import sys

def parse_observed_at(value: str) -> dt.datetime:
    parsed = dt.datetime.fromisoformat(value)
    if parsed.tzinfo is None:
        raise ValueError("observedAt must carry a timezone offset")
    return parsed

def unavailable_nodes(nodes: list[dict]) -> list[dict]:
    return [
        {"id": node.get("id"), "uri": node.get("uri"), "status": node.get("status")}
        for node in nodes
        if node.get("status") != "UP"
    ]

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--queue-threshold", type=int, default=3)
    parser.add_argument("--samples", type=int, default=4)
    parser.add_argument("--max-gap-seconds", type=float, default=45.0)
    args = parser.parse_args()

    streak = 0
    streak_started_at = None
    previous_observed_at = None

    for line_number, line in enumerate(sys.stdin, start=1):
        if not line.strip():
            continue
        try:
            sample = json.loads(line)
            queue_size = int(sample["queueSize"])
            observed_at = parse_observed_at(sample["observedAt"])
        except (json.JSONDecodeError, KeyError, TypeError, ValueError) as error:
            print(f"invalid sample at line {line_number}: {error}", file=sys.stderr)
            return 2

        gap = None
        if previous_observed_at is not None:
            gap = (observed_at - previous_observed_at).total_seconds()
        previous_observed_at = observed_at

        if gap is not None and (gap <= 0 or gap > args.max_gap_seconds):
            print(
                f"sample spacing at line {line_number} is {gap:.1f}s; streak reset",
                file=sys.stderr,
            )
            streak = 0
            streak_started_at = None

        if queue_size >= args.queue_threshold:
            if streak == 0:
                streak_started_at = observed_at
            streak += 1
        else:
            streak = 0
            streak_started_at = None

        if streak >= args.samples:
            alert = {
                "kind": "selenium_grid_queue_sustained",
                "observedAt": sample.get("observedAt"),
                "windowSeconds": round(
                    (observed_at - streak_started_at).total_seconds(), 1),
                "queueSize": queue_size,
                "sessionCount": sample.get("sessionCount"),
                "maxSession": sample.get("maxSession"),
                "unavailableNodes": unavailable_nodes(sample.get("nodes", [])),
                "queueRequests": sample.get("queueRequests", []),
                "consecutiveSamples": streak,
            }
            print(json.dumps(alert, indent=2))
            return 1

    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The windowSeconds field in the alert is what makes the claim auditable. A responder can see that four samples covered forty-five seconds rather than assuming it, and a window that looks wrong points at the sampler rather than at the Grid. Set --max-gap-seconds slightly above the sampler's interval so ordinary jitter does not reset a legitimate streak, and treat a reset line on standard error as a monitoring-path event worth its own signal.

Exit 1 means sustained queue pressure; exit 2 means bad monitoring input. Preserve that distinction in CI or a service manager. The next runner polls every 15 seconds for five minutes, stores every snapshot, and evaluates the completed window. It is suitable as a scheduled infrastructure check. A continuously running monitor would evaluate as samples arrive and notify sooner.

#!/usr/bin/env bash
set -euo pipefail

GRID_URL="$1"
SNAPSHOTS="$(mktemp)"
trap 'rm -f "$SNAPSHOTS"' EXIT

for sample_number in $(seq 1 20); do
  if ! python3 grid_snapshot.py --grid-url "$GRID_URL" >> "$SNAPSHOTS"; then
    printf 'GraphQL sampling failed on sample %s\n' "$sample_number" >&2
    exit 2
  fi
  sleep 15
done

if python3 queue_window.py \
  --queue-threshold 3 --samples 4 --max-gap-seconds 45 < "$SNAPSHOTS"; then
  printf 'No sustained queue pressure detected\n'
else
  status="$?"
  if test "$status" -eq 1; then
    printf 'Sustained Grid queue pressure detected\n' >&2
  fi
  exit "$status"
fi

Twenty samples add five minutes to this job. That is an explicit cost, not a reason to reduce the rule to one sample. For real-time operations, keep a small resident process or use a metrics system capable of evaluating “for” duration. The same principle remains: the alert state comes from multiple successful observations.

Queue age would be even stronger than queue count if your telemetry stack derives it from request arrival timestamps. The documented GraphQL field provides current request payloads, and the exact content can vary with Selenium versions. Do not invent an age field in the query. If age is not directly available, consecutive observation of a stable or growing request set is a defensible proxy, while consecutive nonzero counts measure pressure more generally.

Reproduce three causes that look identical on a dashboard

A controlled load probe makes alert tuning less theoretical. Configure a test Grid with one Chrome slot, then start three sessions concurrently. The first browser holds its slot while the other calls wait in the New Session Queue. Use a generous session request timeout so the exercise observes queuing rather than immediately timing out.

import java.net.URI;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

public final class QueuePressureProbe {
  public static void main(String[] args) throws Exception {
    URI grid = URI.create(System.getenv().getOrDefault(
        "GRID_URL", "http://localhost:4444"));
    int clients = 3;

    ExecutorService pool = Executors.newFixedThreadPool(clients);
    CountDownLatch start = new CountDownLatch(1);
    List<Callable<String>> tasks = new ArrayList<>();

    for (int index = 0; index < clients; index++) {
      final int client = index;
      tasks.add(() -> {
        start.await();
        long requestedAt = System.nanoTime();
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless=new");

        RemoteWebDriver driver = new RemoteWebDriver(grid.toURL(), options);
        long waitMillis = Duration.ofNanos(
            System.nanoTime() - requestedAt).toMillis();
        try {
          System.out.printf(
              "client=%d session=%s waitMillis=%d%n",
              client, driver.getSessionId(), waitMillis);
          Thread.sleep(Duration.ofSeconds(40).toMillis());
          return driver.getSessionId().toString();
        } finally {
          driver.quit();
        }
      });
    }

    List<Future<String>> results = new ArrayList<>();
    for (Callable<String> task : tasks) {
      results.add(pool.submit(task));
    }
    start.countDown();

    for (Future<String> result : results) {
      result.get();
    }
    pool.shutdown();
  }
}

In the first failure mode, demand exceeds valid capacity. All requests ask for Chrome, the only Chrome slot is occupied, the Node stays UP, and the queue falls as sessions quit. Evidence shows sessionCount equal to maxSession while the queue is nonzero. The fix may be more Chrome capacity, lower test-runner concurrency, shorter sessions, or better scheduling. Extra capacity costs compute and may sit idle outside peak CI windows.

The second failure mode is a capability mismatch. Request Safari on a Grid that only advertises Linux Chrome. Aggregate sessionCount can be zero, maxSession can be positive, and the queue still grows. The queued payload names Safari while stereotypes do not. Adding more Linux Chrome Nodes changes no matching outcome. Either route that test to a provider with Safari, correct an accidental capability, or configure an appropriate Node. The cost is specialized capacity and, for macOS browsers, a different operating environment.

The third failure mode is capacity loss. Begin with three matching Nodes, then stop two while the load probe runs. The queue rises because supply disappeared, not because planned demand changed. GraphQL shows Nodes missing or no longer UP, and infrastructure events line up with the first sustained samples. The right response is Node recovery or safe job throttling. Permanently raising capacity can hide unstable Nodes while increasing spend.

These causes deserve different notification text. “Queue above three for one minute” is enough to open an incident. The attached context should then guide classification:

• full matching pool: demand or slow session turnover;
• broad free capacity plus unmatched request: capability mismatch;
• Node count or UP status dropped: capacity loss;
• queue payload unchanged while all matching slots appear free: investigate registration, Distributor health, or stale observability.

Do not claim an exact matcher in the alert unless you use Selenium's own matching logic. Vendor-prefixed capabilities, platform normalization, browser versions, and configured slot stereotypes make naive dictionary equality unreliable. Showing both sides is safer than confidently labeling a request unsupported on an incomplete comparison.

Separate real demand from sessions whose owners disappeared

One failure looks almost exactly like ordinary demand pressure in the sampled fields. The queue rises, every Node remains UP, aggregate sessionCount equals maxSession, and waiting capabilities match the advertised stereotypes. Adding workers appears reasonable. The different root cause is that some occupied sessions no longer have a live test worker. A cancelled CI job, killed runner process, lost runner machine, or failed cleanup path can leave the Grid counting a session until server-side cleanup releases it. During that interval, a dead owner consumes the same slot as a productive test.

The separator is ownership evidence, not another capacity total. Start with a timestamp inside the alert window and identify the sessions that occupied the matching pool at that moment from the Grid and Node logs available in your deployment. Join each session identifier to the CI run, shard, and attempt that created it. A genuinely busy pool has a live job for each occupied session, and those jobs continue to issue commands or reach normal teardown. An abandoned-session incident has at least one session assigned to a job that was already cancelled, timed out, or lost before the queue began to climb. Its client log has no successful cleanup after the last command, while Grid-side evidence continues to count the session.

The shape of recovery is useful corroboration. With excess legitimate demand, capacity returns as active tests finish, usually across several completion times. With abandoned owners, the queue can drain in a step when server-side cleanup finally removes one or more sessions, even though no Node was added and no CI fan-out ended at that instant. That timing is not proof by itself, but it tells the responder which session identifiers to trace. A Node status of UP cannot separate these cases because the Node can serve commands correctly while holding a session whose client has vanished.

Fix the lifecycle before purchasing capacity. Put cleanup in the suite's shared session fixture so ordinary test failures cannot bypass it, then test cancellation and runner termination paths separately from assertion failures. A graceful finally path does not run when a machine disappears or a process is forcibly terminated, so the Grid's server-side cleanup policy remains part of the safety net. The Grid platform owner should not shorten that policy blindly, because a value that reclaims abandoned sessions faster can also terminate a legitimate session during a long quiet phase. The concrete cost is a narrower allowance for tests that do useful work outside the browser between commands.

This distinction also prevents a misleading handoff. “All twelve slots were full” only proves allocation, not useful work. The incident record must say how many occupied sessions mapped to live CI attempts, how many mapped to completed or missing attempts, and which logs established that mapping. Without that join, the capacity team cannot tell whether another Node treats the cause or merely gives abandoned sessions more room to accumulate.

Read the alert object as a time sequence

The evaluator's output contains several numbers that answer different questions. queueSize is the count in the final accepted sample. consecutiveSamples is the number of adjacent valid samples at or above the configured threshold. windowSeconds is elapsed time from the first qualifying observation to the current one. In an illustrative broken sequence sampled every fifteen seconds, queueSize values of four, five, five, and seven produce consecutiveSamples of four and windowSeconds near forty-five. The increasing final value says arrivals are outpacing allocations at the end of the window. The time fields say the pressure persisted long enough to satisfy the policy.

A healthy recovery looks different even if its first snapshot is worse. An illustrative sequence of eight, four, two, and zero never reaches four qualifying samples when the threshold is three. The first number is large, but the streak resets as allocation catches up. That is why the responder should read the stored lines before the alert and not only the final notification.

Two values are easy to overinterpret. windowSeconds is not the age of the oldest queued request. It measures how long the aggregate condition was observed. consecutiveSamples also does not prove that one request remained queued throughout the streak. Four requests can be allocated while four new ones replace them, leaving the count unchanged. The queued capability payloads can show whether the mix changed, but identical payloads may belong to different tests. Unless the retained evidence contains a trustworthy request identity from an existing logging path, describe the result as sustained aggregate pressure, not as a specific request waiting forty-five seconds.

The most misleading snapshot is a stable queue. A sequence of five, five, five, and five may look less urgent than rapid growth, yet it can mean that one request arrives for every request allocated. Compare CI submission and successful session-creation events during the same UTC interval. Equal queue counts with continuing churn indicate a pool running at its service limit. Equal counts with no successful creations point instead to stalled allocation or sessions that are not being released.

Tell a real queue incident from a broken monitor

A failed /graphql call is not proof that queue size is high. It can result from Router unreachability, authentication changes, a proxy returning HTML, DNS failure, TLS failure, or a query that no longer matches the deployed schema. Alert on consecutive scrape failures separately and include the transport or GraphQL error. Capacity responders need a different runbook from platform responders.

A sudden queue reset to zero may also be bad news. Clearing the New Session Queue rejects waiting client requests. A Router or queue component restart can remove or fail work depending on architecture and storage. Compare the zero with client SessionNotCreatedException reports, component restart events, and queue logs before celebrating recovery.

Client timeouts are a common near miss. The New Session Queue has its own session request timeout and retry settings. A CI framework or HTTP proxy may give up earlier. In that case requests can fail from the client's point of view even though the server-side queue never reaches your page threshold. Measure the time from RemoteWebDriver construction to success or exception, and align the alert with the strictest meaningful user-facing timeout.

Long browser startup can resemble queue delay. Once a request leaves the queue and a Node begins creating the session, queue size may fall while the client still waits. Look at session creation logs and Node resource use. A slow driver binary, container image pull, profile setup, or browser crash belongs to provisioning diagnosis, not queue-capacity alerting.

Existing-session slowness is another different path. If clicks and reads are slow after a session has started, the New Session Queue can remain empty. Examine Router-to-Node latency, browser CPU, application response, and tracing. A queue alert should never be presented as a complete Grid performance monitor.

Clock and sampling errors can invent duration in both directions. If two sampler replicas write to one stream, four “consecutive” samples may represent the same fifteen-second instant. If one sampler stalls, four adjacent lines may span an hour. The --max-gap-seconds check catches the second case; the first needs an upper bound too. Identify the sampler, sort by observedAt, and either elect one evaluator or aggregate into fixed time buckets. Reject timestamps too far in the past. Duration is only meaningful when observations cover duration.

Roll out the alert without teaching people to ignore it

For an existing suite, land correlation before policy. Record the start and outcome of every session-construction attempt in the runner logs, using the CI system's existing run, shard, and retry identifiers. After creation succeeds, record the returned session identifier in the same context. Keep capability secrets and internal URLs out of that record. This instrumentation must reach all shared fixtures before the queue alert pages anyone, or the first incident will produce Grid evidence that cannot be joined to a test owner.

Deploy the collector next with no notification destination. Validate three paths against the deployed environment: a normal sample, a deliberately invalid query that must become unknown, and a controlled queue created by the load probe. Then canary the evaluator against one representative pipeline while the rest of the suite continues to supply baseline traffic. The canary should include a normal high-fan-out run and a deliberately cancelled run. The first catches a threshold that fires on expected bursts. The second proves whether abandoned sessions can be attributed to the cancelled attempt.

Only after those joins work should a team change capacity or runner concurrency. Change one control at a time and retain the same sampling cadence through the comparison period. If evidence shows excess legitimate demand, land a runner-side concurrency limit before lowering an alert threshold, so responders have an action that can reduce pressure. If evidence shows abandoned sessions, land cleanup and cancellation handling before adding Nodes. If evidence shows unsupported capabilities, correct routing or the requested capability before expanding the common browser pool.

The first suite-level break from a concurrency reduction is usually elapsed pipeline time, especially in a shard already close to its CI deadline. For example, purely as an illustration, halving six equal workers to three doubles the idealized execution portion before setup and teardown are counted. The queue graph improves immediately, but a release gate can still regress because work now waits in the runner instead of the Grid. Track end-to-end job duration, session-construction wait, completion rate, and queue duration together. The change is working only when Grid waiting falls without moving failures to the CI deadline or violating the suite's delivery objective.

Collect a week of snapshots before paging anyone. Include normal weekday peaks, nightly regressions, and at least one maintenance window. Plot queue count alongside session count, max session, Node count, and CI job submissions. Find how high and how long ordinary bursts run.

Start the rule in record-only mode. For every would-be alert, save the attached GraphQL context and ask what action a responder could have taken. If most events resolve before a person could open the dashboard, lengthen duration. If user-facing session creation already times out before the rule fires, shorten it or lower the threshold.

Separate warning from paging. A warning can mark queue growth that a team reviews during working hours. Paging should represent sustained impact or imminent timeout. One practical arrangement is a lower threshold for five minutes and a higher threshold for two minutes, but derive both from your own arrival pattern and session timeout.

Route by ownership. A mismatch involving a rare browser may belong to the team that requested it. A broad Node loss belongs to the Grid platform owner. CI fan-out beyond an agreed concurrency limit belongs to the submitting pipeline. One generic channel produces debates instead of recovery.

Keep one owner for detection and another for remediation when needed. The observability owner retains the incident until the sampler is proven valid. The Grid platform owner takes a capacity or session-lifecycle case once matching pool state and session ownership are attached. The CI platform owner takes retry amplification, cancellation, or fan-out cases. The test repository owner takes an accidental capability request or cleanup path specific to its fixture. This prevents a failed scrape from being handed to the team that happens to run the largest suite.

A useful handoff is a compact evidence packet, not a dashboard link. Include the UTC start and end of the valid observation window, sampler identity, raw snapshot location, threshold and sample spacing, queue minimum and maximum, matching Node identities, redacted waiting capabilities, CI run and shard identifiers, and any occupied session identifiers already correlated to those runs. State the disputed fact and the requested action, such as “two sessions belonged to cancelled shards; inspect cancellation cleanup.” Also name who will verify recovery and the signal they will use. That final field stops a ticket from being closed after a configuration change while the same queue pattern continues.

Put the evidence in the alert itself: observed interval, queue range, sessionCount and maxSession, count of non-UP Nodes, and a redacted view of queued capabilities. Capability payloads can contain vendor options or test metadata. Review them before sending to a third-party incident system, and remove secrets or internal URLs.

Exercise the rule deliberately each quarter. Use the one-slot load probe to trigger demand pressure, send an unsupported capability to create a mismatch, and take a disposable Node out of service. Confirm alert timing, classification clues, notification delivery, and clearing behavior. A monitoring rule that has never fired under controlled conditions is an untested production feature.

During a Selenium upgrade, run the sampler against the new Grid before moving test traffic. GraphQL is typed, so an invalid field returns errors clearly if the consumer checks them. Store one fixture response from each supported version for parser tests, but let live validation catch network and authentication differences.

Accept the cost, and skip this pattern when it cannot help

Polling every fifteen seconds adds four GraphQL requests per minute per Grid. A focused query is cheap, but dozens of independent dashboards and alerts can duplicate that traffic. Prefer one collector that publishes normalized metrics or snapshots to multiple consumers. That adds a small service to operate, yet it gives consistent timestamps and error handling.

Retention costs grow when queueRequests contains full capability payloads. Keep high-resolution samples for the incident window and aggregate older data. Redact sensitive fields before storage. Counts can remain longer for capacity planning, while raw requests often need shorter retention.

Duration delays notification by design. Four samples at fifteen seconds cannot alert before roughly forty-five to sixty seconds, depending on evaluation timing. That delay is the price of suppressing momentary bursts. If every new session must start within ten seconds, a one-minute alert is structurally too slow. Monitor client wait time or enforce admission limits closer to the request.

This technique does not catch a new-session request that never reaches the Grid queue. A DNS error, TLS failure, authentication rejection, or reverse proxy rejection can fail every client while sessionQueueSize remains zero. Measure session-construction outcomes at the runner and monitor the network edge separately. A green queue panel in that incident is accurate, but it says nothing about reachability before admission.

Do not use this alert for a small local Standalone Grid that no one operates as a service. The developer can see the one occupied browser, and a polling process adds more machinery than insight.

Skip paging when queued work has no user impact and automatically drains within the CI service objective. Keep the data for capacity planning instead. An alert should represent an action, not curiosity.

Do not autoscale solely from total queue count when Nodes have different stereotypes. The system can add ten Chrome workers while Safari remains unmatched. Scale from a capability-aware signal that has been validated against Grid behavior, or require a human to classify rare pools.

Finally, do not let a queue dashboard replace test-runner controls. Unbounded parallelism can overwhelm the application under test, its test data, or downstream services even if Grid adds Nodes fast enough. Set concurrency where end-to-end systems remain reliable. Queue alerting tells you that allocation pressure exists. It does not decide how much parallel execution the product can safely absorb.