Why a healthy Selenium Grid still starves one browser pool, and the SLO that catches it

The release dashboard showed 96 percent pass rate and a mean time-to-session comfortably inside target, so the Grid was declared healthy and the release went ahead. Meanwhile the payments team's Safari suite had not completed a full run in four days. Their sessions were not failing in any interesting way: they were sitting in the queue, timing out after five minutes, retrying, and eventually being marked flaky by a framework retry policy that nobody had looked at since it was added.

The Grid was healthy. It was also starving one pool. Both statements were true at once, and the reason is that the metric on the dashboard averaged across every stereotype in the fleet. Chrome held the overwhelming majority of the slots, chrome sessions started fast, and chrome therefore decided what the average said. That is not a monitoring bug. It is what happens when a single number is asked to describe a heterogeneous resource pool.

The mechanism, in the order it actually happens

Four documented behaviours combine to produce this. Each is individually reasonable.

The queue is FIFO. The Grid components page describes the New Session Queue as a component that "Holds all the new session requests in a FIFO order." It is a single ordered queue for the whole Grid, not one queue per browser.

Matching is per-request, against slots. The Distributor is described as "Responsible for maintaining a model of the available locations in the Grid where a session may run (known as 'slots') and taking any incoming new session requests and assigning them to a slot." Two pluggable pieces do the work, and the CLI reference names both: --slot-matcher (default org.openqa.selenium.grid.data.DefaultSlotMatcher), "used to determine whether a Node can support a particular session", and --slot-selector (default org.openqa.selenium.grid.distributor.selector.DefaultSlotSelector), "used to select a slot in a Node once the Node has been matched."

Consumption is batched against availability. --sessionqueue-batch-size, default 20, is documented as the "Maximum number of session requests that can be consumed from the queue at a time, based on the available slots."

The deadline is uniform. --session-request-timeout, default 300 seconds, is described as: "A new incoming session request is added to the queue. Requests sitting in the queue for longer than the configured time will timeout." It is checked on an interval set by --session-request-timeout-period, default 10 seconds.

Put those together and the starvation is structural. A request for a scarce stereotype waits until a slot that matches it becomes free. Every chrome request that arrives behind it can still be satisfied as soon as a chrome slot frees, so throughput looks excellent and the aggregate latency distribution stays tight. The scarce request contributes one sample. If you have a hundred times more chrome traffic than Safari traffic, your ninety-ninth percentile is still a chrome number.

There is a fifth behaviour worth knowing, because it silently corrupts the metric you are about to build. --reject-unsupported-caps is a Distributor flag defaulting to false, documented as: "Allow the Distributor to reject a request immediately if the Grid does not support the requested capability. Rejecting requests immediately is suitable for a Grid setup that does not spin up Nodes on demand." With the default, a request naming a browser that no stereotype in the fleet can serve does not fail fast. It queues, exactly like a legitimate request waiting for capacity, until the timeout fires. A typo in browserName and a genuine capacity shortage look identical in any metric that only knows about waiting.

What Grid will and will not tell you

Be precise about this, because a lot of monitoring effort gets wasted assuming the Grid exposes more than it does.

The documented GraphQL schema gives you a grid object with uri, totalSlots, nodeCount, maxSession, sessionCount, sessionQueueSize and version, plus a node view with id, uri, status, maxSession, slotCount, sessionCount, stereotypes, osInfo and the sessions currently running on each node. The endpoints documentation adds GET /se/grid/newsessionqueue/queue, which returns the total request count and the payloads of the queued requests.

What that gives you is a depth gauge and an inventory. sessionQueueSize is a single scalar for the whole Grid. It tells you that things are waiting. It does not tell you what is waiting, or for how long, or whether the wait is spread evenly.

What it does not give you is a per-request or per-stereotype wait histogram. So the SLI has to be assembled from two sources that answer different questions:

1. The client. Time the W3C New Session command in your driver factory, tagged with the stereotype you asked for. This is authoritative for the user experience, because it is the user experience.
2. The queue. Poll GET /se/grid/newsessionqueue/queue, bucket the returned payloads by requested capability, and you have queue composition over time. This is what turns "the queue is deep" into "the queue is deep with Safari".

The first is your SLI. The second is your diagnostic. Do not try to make one do both jobs.

Instrumenting the admission event

This factory emits one structured record per new session attempt. It is the single most valuable piece of instrumentation in this article, and it is about thirty lines.

package dev.example.grid.slo;

import com.fasterxml.jackson.databind.ObjectMapper;
import java.time.Duration;
import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.UUID;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.SessionNotCreatedException;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/** Creates a RemoteWebDriver and emits exactly one admission event per attempt. */
public final class InstrumentedDriverFactory {

  private static final Logger ADMISSION = LoggerFactory.getLogger("grid.admission");
  private static final ObjectMapper JSON = new ObjectMapper();

  private final java.net.URL router;

  public InstrumentedDriverFactory(java.net.URL router) {
    this.router = router;
  }

  public RemoteWebDriver create(Capabilities requested, String suite) {
    // The pool label is the SLO dimension. Keep it low-cardinality on purpose:
    // "chrome-linux", "safari-macos", "firefox-linux". Not the full capability map.
    String pool = requested.getBrowserName() + "-" + requested.getPlatformName();
    String attemptId = UUID.randomUUID().toString();
    Instant start = Instant.now();

    try {
      // Clock starts immediately before New Session and stops on the response.
      RemoteWebDriver driver = new RemoteWebDriver(router, requested);
      emit(attemptId, pool, suite, "ADMITTED", Duration.between(start, Instant.now()),
          driver.getSessionId().toString(), driver.getCapabilities(), null);
      return driver;
    } catch (SessionNotCreatedException e) {
      // Distinguish "waited and gave up" from "rejected outright". They are
      // different failures with different owners, and merging them hides both.
      String outcome = String.valueOf(e.getMessage()).toLowerCase().contains("timed out")
          ? "QUEUE_TIMEOUT"
          : "REJECTED";
      emit(attemptId, pool, suite, outcome, Duration.between(start, Instant.now()),
          null, null, e.getMessage());
      throw e;
    }
  }

  private void emit(String attemptId, String pool, String suite, String outcome,
                    Duration elapsed, String sessionId, Capabilities returned, String error) {
    Map<String, Object> event = new LinkedHashMap<>();
    event.put("ts", Instant.now().toString());
    event.put("attempt_id", attemptId);
    event.put("pool", pool);
    event.put("suite", suite);
    event.put("outcome", outcome);
    event.put("admission_ms", elapsed.toMillis());
    event.put("session_id", sessionId);
    event.put("returned_browser_version", returned == null ? null : returned.getCapability("browserVersion"));
    event.put("error", error);
    try {
      ADMISSION.info(JSON.writeValueAsString(event));
    } catch (Exception ignored) {
      ADMISSION.warn("admission event serialisation failed for {}", attemptId);
    }
  }
}

Three decisions in there are the whole design.

The clock brackets exactly the New Session command. The W3C WebDriver specification defines New Session as the command that establishes the session and returns its id and capabilities, so this window is the queue wait plus slot matching plus browser launch, which is precisely what the test author waits through. Anything wider measures your own framework.

outcome is a first-class field with three values, not a boolean. A request that waited five minutes and gave up is not a slow success, and it must never be folded into a latency percentile as a large number. That single modelling mistake is how these dashboards start reporting a healthy ninety-ninth percentile during an outage: everything that failed left the latency series entirely.

pool is deliberately coarse. Browser plus platform, nothing else. If you tag by full capability map you get thousands of series, your percentile buckets go empty, and the alert becomes noise. If you tag by nothing you are back to the average that started this article.

An illustrative record, so you can see the shape. The numbers here are invented to show the fields, not measured from any real fleet:

{
  "ts": "2026-08-04T09:41:22.117Z",
  "attempt_id": "3f2c9a1e-77b5-4a2c-9e14-0c8f1d6a4b20",
  "pool": "safari-mac",
  "suite": "payments-regression",
  "outcome": "QUEUE_TIMEOUT",
  "admission_ms": 300412,
  "session_id": null,
  "returned_browser_version": null,
  "error": "Could not start a new session. ...",
  "_note": "Illustrative values. Percentiles must come from your own admission stream."
}

Sampling queue composition

The client stream tells you a pool is suffering. The queue sampler tells you why. Run it on a short interval alongside the Grid and it turns a scalar depth into a breakdown.

#!/usr/bin/env bash
# grid-queue-composition.sh
# Samples the New Session Queue and buckets pending requests by browserName,
# then prints the fleet-side counterpart from the GraphQL node view.
set -euo pipefail
ROUTER="${ROUTER:-http://localhost:4444}"
INTERVAL="${INTERVAL:-15}"

while true; do
  now="$(date -u +%FT%TZ)"

  # Demand: what is actually sitting in the queue right now.
  queued="$(curl -sS --fail "$ROUTER/se/grid/newsessionqueue/queue" \
    | jq -c --arg ts "$now" '
        [ .. | objects | select(has("browserName")) | .browserName ]
        | group_by(.) | map({browser: .[0], queued: length})
        | {ts: $ts, kind: "demand", pools: .}')"
  echo "$queued"

  # Supply: what the Distributor believes it can serve.
  curl -sS --fail -X POST -H 'Content-Type: application/json' \
    --data '{"query":"{ grid { totalSlots maxSession sessionCount sessionQueueSize } nodesInfo { nodes { id status slotCount maxSession sessionCount stereotypes } } }"}' \
    "$ROUTER/graphql" \
    | jq -c --arg ts "$now" '{ts: $ts, kind: "supply", grid: .data.grid,
         nodes: [.data.nodesInfo.nodes[] | {id, status, slotCount, sessionCount}]}'

  sleep "$INTERVAL"
done

The jq walk over the queue payload is intentionally defensive. The queue returns request payloads, and capability maps are nested differently depending on whether the client sent alwaysMatch, firstMatch, or both, so the recursive descent for any object carrying a browserName survives all three shapes without you having to hardcode one.

Two lines of output per interval, one demand and one supply, is enough to answer the question that matters during an incident: is the queue deep because there is more work than slots, or deep because the work needs a stereotype that barely exists? Those have completely different fixes and they look identical on a depth gauge.

Writing the objectives so they survive a review

An SLO that a platform team can be held to has four parts: an indicator, a target, a window, and a consequence. Most Grid dashboards have zero of the four.

Indicator one, admission latency per pool. The proportion of New Session attempts for pool P that reach ADMITTED within a threshold specific to P. Different thresholds per pool are not a compromise, they are the correct model: a pool backed by disposable containers and a pool backed by four physical Macs have genuinely different achievable latencies, and pretending otherwise produces either a useless target or a permanently red one.

Indicator two, admission success per pool. The proportion of attempts that reach ADMITTED at all, counting QUEUE_TIMEOUT and REJECTED as failures. This is your availability indicator and it is the one the release manager should actually be reading.

Indicator three, capacity visibility. The proportion of supply samples in which totalSlots for pool P is at or above an agreed floor. This is a leading indicator and it catches the case where somebody scaled a deployment down at 17:00 on Friday. It is worth having because it goes red before either of the other two do.

Set targets from your own baseline. Collect at least two full weeks of admission events before proposing a number, then set the initial target at roughly the reliability you already deliver rather than at the reliability you would like. An objective that is already breached on the day it ships trains everyone to ignore it, and that is a harder problem to undo than a target that turns out to be too lenient.

The consequence has to be written down too, and it has to be something other than a notification. The usual pairing: while a pool's error budget is intact, feature work on that pool proceeds; when the budget is exhausted, the next change to that pool is a capacity or reliability change. If nobody will agree to that sentence, you do not have an SLO, you have a graph.

Wiring the alert so it pages for the right reasons

Multi-window burn rate alerting exists because single-threshold alerts on a percentile either page constantly or page too late. The rules below assume you have shipped the admission events into Prometheus as a histogram and a counter. The thresholds are placeholders, and they are marked as such in the file, because filling them in is your job and not mine.

# prometheus/grid-capacity-slo.rules.yaml
groups:
  - name: grid-capacity-sli
    interval: 30s
    rules:
      # Fraction of admissions that met the per-pool latency threshold.
      # The "le" bucket boundary per pool comes from YOUR baseline, not from a blog post.
      - record: grid:admission_latency_good_ratio:rate5m
        expr: |
          sum by (pool) (rate(grid_admission_seconds_bucket{outcome="ADMITTED", le="30"}[5m]))
            /
          sum by (pool) (rate(grid_admission_total[5m]))

      - record: grid:admission_success_ratio:rate5m
        expr: |
          sum by (pool) (rate(grid_admission_total{outcome="ADMITTED"}[5m]))
            /
          sum by (pool) (rate(grid_admission_total[5m]))

      - record: grid:admission_success_ratio:rate1h
        expr: |
          sum by (pool) (rate(grid_admission_total{outcome="ADMITTED"}[1h]))
            /
          sum by (pool) (rate(grid_admission_total[1h]))

  - name: grid-capacity-slo-burn
    rules:
      # Fast burn: short and long window must BOTH agree before paging.
      # 14.4x burn against a 99% objective exhausts a 30d budget in ~2 days.
      - alert: GridAdmissionBudgetBurningFast
        expr: |
          (1 - grid:admission_success_ratio:rate5m) > (14.4 * 0.01)
            and
          (1 - grid:admission_success_ratio:rate1h) > (14.4 * 0.01)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Pool {{ $labels.pool }} is burning its admission budget fast"
          runbook: "https://runbooks.internal/grid/admission-budget"

      # Starvation detector: this is the alert the average would never fire.
      # One pool failing while the Grid overall looks fine.
      - alert: GridSinglePoolStarving
        expr: |
          grid:admission_success_ratio:rate1h < 0.95
            and on()
          (sum(rate(grid_admission_total{outcome="ADMITTED"}[1h])) / sum(rate(grid_admission_total[1h]))) > 0.99
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Pool {{ $labels.pool }} is starving while the fleet average is healthy"

      # Leading indicator: capacity disappeared before anyone complained.
      - alert: GridPoolCapacityFloorBreached
        expr: grid_pool_total_slots < on (pool) grid_pool_slot_floor
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "Pool {{ $labels.pool }} has fewer slots than its agreed floor"

GridSinglePoolStarving is the rule that would have caught the opening scenario. It fires precisely when a per-pool ratio is bad and the fleet-wide ratio is good, which is the condition an averaged dashboard is structurally incapable of showing. If you implement only one thing from this article, implement that comparison.

How to tell it is queue starvation and not a look-alike

Three failures produce "sessions take forever to start" and they have three different owners.

Slow browser launch on the Node. Here the request is admitted quickly but the browser takes a long time to become usable. The distinguishing evidence is that sessionQueueSize stays near zero while admission_ms is high, and sessionCount sits well below maxSession. Nothing is waiting for a slot; the slot is just slow. Look at Node host resources, container shm_size, and image pull time.

Genuine capacity exhaustion. Queue depth is high, sessionCount is pinned at maxSession, and the queue composition sampler shows demand spread across the same pools as your slots. This is the honest case and the fix is capacity or scheduling.

Starvation. Queue depth is high, but the composition sampler shows the queue dominated by one browser, and sessionCount for that pool's nodes is at its ceiling while other pools sit idle. totalSlots is healthy in aggregate and irrelevant in detail. This is the case the average hides.

There is a fourth that deserves its own mention because it is so cheap to rule out. If your REJECTED and QUEUE_TIMEOUT counts are dominated by a single suite, check that suite's requested capabilities against the stereotypes field in the GraphQL node view. With --reject-unsupported-caps at its default of false, a capability that matches nothing waits out the full --session-request-timeout before failing, so a single mis-configured job can generate a continuous stream of long waits that look exactly like a capacity problem. Enabling that flag on a Grid that does not autoscale converts those into immediate failures, which is both faster feedback for the author and a cleaner signal for you.

Second failure mode: the SLI that lies because of retries

This one is worth a section because it invalidates everything above if you get it wrong.

Many teams have a driver factory that retries session creation two or three times before giving up. It was added years ago to paper over a flaky node and nobody has revisited it. With that retry in place, a pool that takes three attempts to admit a session reports ADMITTED and looks fine. The starvation is completely invisible, and the only trace is a slightly elevated latency that gets attributed to "the Grid being slow today".

The fix is not to remove the retry. Retries are frequently the right operational choice. The fix is to make each attempt its own admission event, which is why the factory above generates a fresh attempt_id per call and emits before it rethrows. Then a wrapper that retries produces three events, two of them QUEUE_TIMEOUT, and your success ratio tells the truth while your suite still passes.

The same discipline applies to client-side HTTP timeouts. If your HTTP client gives up before the Grid's --session-request-timeout fires, you will record a client error rather than a queue timeout, and you will spend an afternoon looking for a network fault that does not exist. Make sure the client-side read timeout is comfortably longer than the Grid's queue deadline, or make sure you can distinguish the two in the error field. Either is fine. Not knowing which one fired is not.

Rolling this out

Week one: instrument, do not alert. Ship the driver factory change and the queue sampler. Emit events. Build no dashboards, set no targets, page nobody. You are collecting the baseline that everything else depends on, and any target you set before this data exists is a guess wearing a suit.

Week two: describe, do not judge. Build the per-pool latency distribution and success ratio views. Show them to the teams who own each suite. Expect at least one conversation that starts with "we thought that pool was fine" and at least one pool whose numbers embarrass someone. That is the whole point of the exercise and it is why doing it without targets first is kinder.

Week three: propose targets. One per pool, set at roughly current performance, written down with the window and the consequence. Get the consequence agreed by whoever can actually stop feature work, not just by the platform team.

Week four: alert on burn rate. Start with ticket severity only. Watch for a full cycle. Promote to page only the rules that fired when something was genuinely wrong.

Ongoing: revisit quarterly. Pools change. A team migrates from Safari to a cloud provider, someone adds an ARM node fleet, a browser version bump changes launch time. An SLO that is never revised becomes folklore.

The trade-offs, stated plainly

Headroom costs money. The only reliable way to improve admission latency for a scarce pool is to hold capacity that is idle most of the time. That is the trade, and it should be made explicitly with a number attached rather than absorbed quietly by whoever pays the cloud bill. For physical device pools the number is capital expenditure and lead time, which makes the conversation slower and more important.

Separate Grids buy isolation and cost operations. Because --session-request-timeout is a queue-level setting, one Grid cannot offer two admission deadlines. Splitting a scarce pool onto its own Grid gives it its own queue and its own deadline, and it genuinely stops chrome demand from sharing a FIFO with Safari demand. It also doubles the number of Routers, Distributors and upgrade windows you maintain. Do it when the pools have genuinely different service expectations, not merely because the graph looks nicer.

Tighter alerts trade sleep for detection. A short burn window catches incidents sooner and pages on transient blips. The multi-window rules above exist specifically to buy back some of that, but there is no setting that gives you both. Decide which mistake you would rather make, and write the decision in the runbook so the next person does not silently reverse it.

Per-pool tagging trades cardinality for insight. Every dimension you add multiplies your series count. Browser plus platform is usually the right stopping point. Adding browser version gives you sharper attribution during upgrades and a metrics bill that grows every time a browser ships, which is roughly monthly.

When not to do this

If your total Grid demand is small and bursty, an SLO is the wrong instrument. Reliability objectives are statistical and they need volume to mean anything. A team running a few dozen sessions a day should set a simple threshold alert on queue depth and spend the saved effort elsewhere; percentiles computed over a handful of samples per hour will swing wildly and teach nobody anything.

If your Grid is created and destroyed per pipeline run, skip it. There is no steady state to have an objective about. Measure job duration and node registration time instead, which are the things that actually vary.

If you have exactly one homogeneous pool, the averaged metric you already have is not lying to you. The entire premise of this article is heterogeneity. A Grid with nothing but identical chrome nodes has no pool to starve, and the added machinery buys you nothing until the day someone adds Firefox.

If nobody will agree to a consequence, do not call it an SLO. Build the dashboard, keep the instrumentation, and be honest that it is monitoring. The word carries an implied commitment, and using it without one devalues it for the next team who wants to make a real commitment.

And do not start here if you do not yet know whether your Grid is losing nodes. Capacity objectives assume the fleet you think you have is the fleet the Distributor can see. Confirm that first, because a Grid quietly shedding nodes will breach every objective you write for reasons that have nothing to do with capacity planning.

Related reading

For the query layer that feeds all of this, see building a Selenium Grid capacity dashboard with GraphQL. If your fleet is containerised, the scaling behaviour interacts directly with running Selenium Grid on Kubernetes with disposable nodes, and geographic splits change the arrival pattern in ways covered in Selenium Grid multi-region architecture. Before you trust any capacity number, rule out the coordination failure described in debugging Selenium Grid event bus connectivity. And if you are being interviewed on this, Selenium Grid Kubernetes interview questions covers the surrounding ground.

FAQ

Does Selenium Grid expose queue wait time per browser directly?

Not as a per-request metric. The documented GraphQL schema gives you sessionQueueSize as a single grid-wide number, and the New Session Queue endpoint returns the pending request count along with their payloads. To get wait time attributed to a specific stereotype you have to measure it where the request originates, in your driver factory, or reconstruct composition by sampling the queue payloads and bucketing them yourself.

Can I set a different session request timeout for Chrome and for Safari on one Grid?

No, and this constraint drives most capacity architecture decisions. --session-request-timeout is a SessionQueue setting that applies to the queue as a whole, so a single Grid gives every browser class the same admission deadline. Teams that genuinely need different guarantees per class end up running separate Grids, or separate Routers with their own queues, rather than trying to tune one queue two ways.

Where should the SLI clock start and stop?

Start it immediately before the client issues the W3C New Session command and stop it when the response returns a session id. That window contains the queue wait, slot matching and browser launch, which is exactly the experience the test author has. Starting the clock earlier drags in your own framework setup, and starting it later measures nothing useful.

Why do requests for a browser nobody has still burn the error budget?

Because the Distributor's default behaviour is to keep them queued. --reject-unsupported-caps defaults to false, so a typo in browserName or a version that no stereotype matches sits in the queue until --session-request-timeout expires rather than failing fast. Those requests are indistinguishable from genuine capacity waits in a naive latency metric, which is why the flag matters to your SLI and not only to your users.

How many nodes do I need to hit a given admission target?

Nobody can answer that from outside your system, and any article that gives you a number is guessing. Derive it from your own arrival pattern: sample concurrent demand per stereotype during your peak window, compare it against totalSlots for that stereotype from the GraphQL node view, and size so the busiest ten-minute window still has headroom. The shape of the arrival burst matters far more than the daily total.

Should a session that failed to start count against the availability SLO or the latency SLO?

Both, but only once each, and they must be separated at the point of measurement. Record the outcome as a distinct field on every admission event: admitted, timed out in queue, or rejected. Latency percentiles then get computed only over admitted requests, while the availability objective counts the other two. Folding a timeout into the latency series as a very large number is the most common way these dashboards start lying.

Practice this

Open your current Grid dashboard and ask one question: if a single browser pool were completely unavailable right now, which panel would change? If the honest answer is "none of them until someone files a ticket", you have found the gap this article is about. Then take it into the QABattle battle arena, pick a Selenium infrastructure scenario, and argue the case for a per-pool objective against the obvious objection that it is more work than a single number. The counter-argument you have to beat is a good one, and being able to answer it is what separates a monitoring proposal that ships from one that gets deferred.