Before you call it Grid flakiness, ask the Grid what it actually saw

Four hundred tests run, six fail, and all six failed at almost exactly the five minute mark with the same line:

org.openqa.selenium.SessionNotCreatedException: Timed out creating session

By the time anyone opens the Grid console the queue has drained, every node is green, and a rerun passes. So the ticket gets closed as "Grid flakiness" and the same six tests fail again next week. The frustrating part is that the Grid knew exactly what happened, wrote it down in three separate places, and threw all of it away before anyone looked.

One exception, six places it could have come from

Selenium Grid 4 is a distributed system with named parts, and the documentation is specific about what each one does. A new session request travels through most of them:

The Router is the entry point and receives all external requests. For a new session it forwards to the New Session Queue, which holds new session requests in FIFO order with configurable timeout and retry parameters. The Distributor polls that queue, finds a suitable Node, creates the session, and records the session-to-Node relationship in the Session Map. The Node then executes commands; the docs are blunt that "A Node only executes the received commands, it does not evaluate, make judgments, or control anything other than the flow of commands and responses." Underneath the Node sits the driver process, and underneath that, the browser. The Event Bus carries registration and heartbeat messages between components.

That is six boundaries plus a browser, and your client sees one exception at the end of it. Treating that exception as a verdict is the mistake. It is a symptom that arrived from somewhere, and the whole job is finding out where.

The good news is that the somewhere is usually written into the message itself.

The error string names the component

Selenium's Grid source assigns distinct messages at distinct points in the pipeline. These are the ones worth memorising, because each maps to a different investigation.

The single most useful distinction in that table is the last row against the first two. A session creation failure has no session id. A session loss does. If your exception contains a UUID, the Grid successfully gave you a browser and then something took it away, which is a completely different investigation from never getting one.

Note also what is not in the table: This version of ChromeDriver only supports Chrome version N. That message comes from the driver on the Node, not from any Grid component, and it means the Grid worked perfectly and handed you a broken pairing. That case is covered separately in the SessionNotCreated version mismatch guide, and it is worth ruling out first because it is the cheapest to confirm.

The two commands that make the Grid answer

Almost all Grid triage runs through two documented endpoints, and neither requires installing anything.

GET /status is the liveness view. Reading Selenium's GridStatusHandler, the response is a value object containing ready, a message that is either "Selenium Grid ready." or "Selenium Grid not ready.", and a nodes array. Each node entry carries nodeId, externalUri, maxSessions, slots, availability, heartbeatPeriod, sessionTimeout, version, and osInfo. Each slot carries id, lastStarted, session, and stereotype. The availability value is one of UP, DRAINING, or DOWN.

#!/usr/bin/env bash
# grid-health.sh: what is the Grid's shape right now?
set -euo pipefail
GRID="${GRID:-http://localhost:4444}"

echo "== readiness =="
curl -sf "$GRID/status" | jq '{ready: .value.ready, message: .value.message}'

echo
echo "== nodes: availability, capacity, and version =="
curl -sf "$GRID/status" | jq -r '
  .value.nodes[]
  | "\(.availability)\t\(.externalUri)\tslots=\(.slots | length)\tmax=\(.maxSessions)\tv=\(.version)\tos=\(.osInfo.name)"
'

echo
echo "== busy slots (a slot with a non-null session is in use) =="
curl -sf "$GRID/status" | jq '
  [.value.nodes[].slots[] | select(.session != null)] | length
'

echo
echo "== what browsers does this Grid actually offer? =="
# Compare this against the capabilities your failing test requested.
# A stereotype your test does not match is the whole bug, most of the time.
curl -sf "$GRID/status" | jq -r '
  [.value.nodes[].slots[].stereotype
   | {browserName, browserVersion, platformName}]
  | unique[]
  | tostring
'

echo
echo "== version drift across nodes (all rows should be identical) =="
curl -sf "$GRID/status" | jq -r '.value.nodes[].version' | sort | uniq -c

That last check earns its place more often than you would expect. A Grid running mixed Selenium versions across nodes produces failures that appear to be random by node, and the /status version field is the fastest way to see it.

What /status cannot tell you is whether there was a queue. For that you need the GraphQL endpoint, whose schema the Grid docs publish in full: grid exposes uri, totalSlots, nodeCount, maxSession, sessionCount, version, and sessionQueueSize, and sessionsInfo exposes sessionQueueRequests alongside the running sessions.

#!/usr/bin/env bash
# grid-queue.sh: was there a queue, and what was in it?
set -euo pipefail
GRID="${GRID:-http://localhost:4444}"

q() {
  curl -sf -X POST -H 'Content-Type: application/json' \
    --data "{\"query\":\"$1\"}" "$GRID/graphql"
}

echo "== capacity vs demand =="
# sessionQueueSize > 0 while sessionCount == maxSession is textbook starvation.
q '{ grid { nodeCount, totalSlots, maxSession, sessionCount, sessionQueueSize, version } }' \
  | jq '.data.grid'

echo
echo "== the actual payloads sitting in the queue =="
# Each entry is the capabilities blob a client asked for. If nothing here
# matches any stereotype from grid-health.sh, you have a matching problem,
# not a capacity problem.
q '{ sessionsInfo { sessionQueueRequests } }' \
  | jq -r '.data.sessionsInfo.sessionQueueRequests[]'

echo
echo "== per-node occupancy and stereotypes =="
q '{ nodesInfo { nodes { id, uri, status, slotCount, sessionCount, stereotypes,
      sessions { id, capabilities, startTime, sessionDurationMillis } } } }' \
  | jq '.data.nodesInfo.nodes'

echo
echo "== raw queue endpoint (total plus request payloads) =="
curl -sf "$GRID/se/grid/newsessionqueue/queue" | jq '.'

The /se/grid/newsessionqueue/queue endpoint is documented as returning the total number of requests in the queue plus the request payloads. It overlaps with the GraphQL view and is easier to hit from a shell script, so use whichever fits your tooling.

Worked example one: starvation, which is a capacity problem wearing a timeout costume

Six failures clustered at five minutes, spread across the tests that happen to start last in a parallel run. Reruns pass because the rerun is not competing with 39 other tests.

The evidence pattern is unmistakable once you capture it at the right moment:

• grid.sessionQueueSize is greater than zero.
• grid.sessionCount equals grid.maxSession.
• Every node's availability is UP.
• The queued request payloads match stereotypes that exist on the Grid.
• The failures land at almost exactly --session-request-timeout, which defaults to 300 seconds.

All five together mean the Grid is working correctly and is simply too small for the concurrency you asked of it. The queue did its job: it held requests in FIFO order and expired them when they aged out.

Node capacity is the lever, and the defaults are worth knowing. --max-sessions defaults to the number of available processors, and the Grid docs note that Nodes create one slot per available CPU for Chromium-based browsers and Firefox by default, with Safari limited to one. --override-max-sessions defaults to false and exists specifically to let you exceed the recommended value, with the documentation warning that "Session stability and reliability might suffer as the host could run out of resources."

That warning is the trade-off and it is real. Raising --max-sessions past the CPU count on a node is how you convert a clean queue timeout into a mess of browser crashes and screenshot timeouts that look like application bugs. If you take that lever, take it in small steps and watch the failure classes change.

The lever people reach for instead, raising --session-request-timeout, is the wrong one. It does not add a single slot. It converts a five minute failure into a ten minute failure and lets the queue grow deeper before anything gives, which makes the next incident larger and slower to diagnose.

Worked example two: a request nobody can serve waits five minutes to find out

The identical symptom, a different cause, and the discriminator is one field.

A test asks for browserVersion: "beta", or platformName: "windows", or a se:name prefixed capability your Grid does not understand. No node has a matching stereotype. You might expect an immediate rejection. You get a five minute wait and Timed out creating session.

This is deliberate, and reading the Distributor makes the reasoning clear. When a request does not match, the Distributor constructs a SessionNotCreatedException reading Unable to find a node supporting the desired capabilities, sets a retry flag, and wraps the whole thing in a RetrySessionRequestException with the message Will re-attempt to find a node which can run this session. The source comment on that branch says it plainly: the last node may have drained and the Distributor has to wait for a new one to register. On a Grid that scales nodes on demand, immediately rejecting a request because no node exists right now would be wrong.

The cost is that on a static Grid, where no new node is ever going to appear, every unservable request burns the full timeout before failing.

The fix is one flag. --reject-unsupported-caps defaults to false, and the documentation describes it as allowing the Distributor to reject a request immediately if the Grid does not support the requested capability, noting that this "is suitable for a Grid setup that does not spin up Nodes on demand." With it enabled, the Distributor compares queued requests against currently UP nodes and fails the unmatched ones straight away with No nodes support the capabilities in the request, plus a log line naming the exact capabilities that found no home.

The trade-off is stated in the documentation itself and you must respect it. On an autoscaling Grid, on Kubernetes, or on any deployment where nodes come and go, this flag will reject requests that would have succeeded thirty seconds later. Enable it on static Grids. Leave it off on elastic ones, and accept the slow failure as the price of elasticity.

Either way, the capability comparison is something you can do yourself in thirty seconds, and it is the highest-yield check in this whole article:

# Left column: what tests asked for. Right column: what the Grid offers.
# If the intersection is empty, stop investigating capacity.
diff <(curl -sf "$GRID/se/grid/newsessionqueue/queue" \
        | jq -r '.value.requests[]? | fromjson? | .capabilities? // empty | tostring' | sort -u) \
     <(curl -sf "$GRID/status" \
        | jq -r '.value.nodes[].slots[].stereotype | tostring' | sort -u)

Payload shapes vary between Grid versions, so treat that as a starting point and print the raw endpoint output first if the jq path does not line up with what your Grid returns.

Worked example three: the session existed, then it did not

Third shape, and the one most often misfiled. The test does not fail at startup. It runs for a while, does real work, and then dies partway through with:

org.openqa.selenium.NoSuchSessionException: Cannot find session with id: 3f9c...

There is a session id, so the Grid succeeded at everything this article has covered so far. Something removed the session afterwards. Three causes account for nearly all of it.

The Node killed it for inactivity. --session-timeout defaults to 300 seconds, documented as: the Node will automatically kill a session that has not had any activity in the last X seconds, releasing the slot for other tests. The Node writes a matching log line, Session id <id> timed out, stopping..., and that line is the proof. This fires on tests that block for a long time without issuing a WebDriver command: waiting on an external batch job, sleeping through an email delivery, polling a slow report. The test is not hung, it just went quiet, and the Grid could not tell the difference.

The browser crashed. No timeout line on the Node, but the browser process disappeared. In containers the overwhelming cause is shared memory: Chrome needs more than Docker's default 64 MB /dev/shm and dies without it. The fix is shm_size: 2gb on the container, and the symptom before the fix is renderer crashes that look like application flakiness.

The Node went away. A node that stops heartbeating gets purged. --heartbeat-period defaults to 60 seconds, --purge-nodes-interval defaults to 30 seconds, and --healthcheck-interval defaults to 120 seconds. A node lost to a spot-instance reclaim or an OOM kill takes its running sessions with it, and every test on it fails at once. That last detail is the discriminator: crashes are usually one test, node loss is always several, simultaneously, on the same externalUri.

Telling the first two apart takes one grep, and it is worth putting into the failure hook:

# Which one was it? Search the node log for the session id from the exception.
SESSION_ID="3f9c..."
docker logs selenium-node-chrome 2>&1 | grep -F "$SESSION_ID"

# "Session id <id> timed out, stopping..."  -> idle timeout, raise --session-timeout
#                                              or make the test keep the session warm
# "Session id <id> is stopping on demand..." -> an orderly quit, the test asked for it
# nothing at all                             -> the browser or the node died underneath it

The second failure mode: the node that never joins

Everything above assumes nodes registered. When they do not, the symptom is a Grid that reports ready: false, an empty nodes array, and every single test failing at the queue timeout, which reads like a total outage rather than a configuration problem.

Registration runs over the Event Bus, and the documentation is explicit that a Node registers to the Distributor by sending a registration event through it. The relevant defaults form a narrow window that is easy to miss. --register-cycle defaults to 10 seconds, described as how often the Node will try to register itself for the first time. --register-period defaults to 120 seconds, described as how long the Node will keep trying, after which "the Node will not attempt to register again."

That last clause is the trap. A node whose event bus was unreachable for the first two minutes of its life gives up permanently. It stays running, it answers its own /status, it looks alive to any process supervisor, and it will never join the Grid. Restarting it fixes it, which makes it feel intermittent when it is deterministic.

--register-shutdown-on-failure, documented as causing the Node to shut down after the register period completes without a successful registration and described as "Useful in container environments to trigger a restart", turns that silent zombie into an exit code your orchestrator can act on. On any containerised Grid it should be on.

The checks, in the order that finds the problem fastest:

# 1. Does the node think it is alive? (default node port 5555)
curl -sf http://selenium-node:5555/status | jq '.value.ready'

# 2. Does the Grid know about it? Compare against the node's own externalUri.
curl -sf http://localhost:4444/status | jq -r '.value.nodes[].externalUri'

# 3. If (1) is true and (2) does not list it, the event bus is the suspect.
#    Publish 4442, subscribe 4443 are the documented defaults.
nc -zv selenium-hub 4442 && nc -zv selenium-hub 4443

# 4. Did the node give up? Look for registration attempts stopping.
docker logs selenium-node-chrome 2>&1 | grep -iE "regist|event bus|unable to"

A node passing check one and missing from check two, with the ports open, is nearly always a hostname problem: the node registered an externalUri that the Router cannot route back to. --grid-url and --bind-host exist for exactly this, and the documentation describes --bind-host as controlling whether the server binds to the host address or "only use it to report its reachable url", which is the Docker case.

There is one more flag worth knowing here. --node-down-failure-threshold defaults to 0, documented as the maximum number of consecutive session creation failures before the Node is marked as DOWN, with 0 disabling the feature and allowing unlimited retries. On the default, a node whose browser installation is broken will keep accepting session requests and keep failing them forever, poisoning a percentage of every run. Setting it to a small number takes the bad node out of rotation on its own.

Wiring the evidence so it exists next time

None of the above helps if the state is gone by the time anyone looks. The single highest-value change in this article is not a flag, it is a failure hook.

# docker-compose.yml: a Grid configured to be diagnosable
services:
  selenium-hub:
    image: selenium/hub:4.39.0          # pin it; :latest re-pairs browsers under you
    ports:
      - "4442:4442"                     # event bus publish
      - "4443:4443"                     # event bus subscribe
      - "4444:4444"                     # router, /status, /graphql, /ui
    environment:
      # Fail unservable requests immediately instead of after 300s.
      # ONLY correct because this Grid is static. Do not set this on autoscaling.
      - SE_REJECT_UNSUPPORTED_CAPS=true
      # Shorter than the 300s default so CI surfaces starvation fast.
      - SE_SESSION_REQUEST_TIMEOUT=120
      - SE_SESSION_REQUEST_TIMEOUT_PERIOD=10
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:4444/status"]
      interval: 15s
      timeout: 5s
      retries: 10

  chrome:
    image: selenium/node-chrome:4.39.0
    depends_on:
      selenium-hub:
        condition: service_healthy
    shm_size: 2gb                        # Chrome dies in the default 64m /dev/shm
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=2
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true
      # Turn a node that never joined into a container restart instead of a zombie.
      - SE_REGISTER_SHUTDOWN_ON_FAILURE=true
      # Take a node that keeps failing session creation out of rotation.
      - SE_NODE_DOWN_FAILURE_THRESHOLD=3
      # Tests that legitimately go quiet for a while need this above 300.
      - SE_NODE_SESSION_TIMEOUT=300

Environment variable names for the Selenium Docker images are set by those images rather than by the Grid CLI, so check the image documentation for the release you pin. The CLI flags they correspond to (--reject-unsupported-caps, --session-request-timeout, --register-shutdown-on-failure, --node-down-failure-threshold, --session-timeout) are the documented Grid options, and passing them through a TOML config file is the version that is guaranteed stable across images.

Then the hook that actually saves the next incident:

# .github/workflows/ui.yml (excerpt)
      - name: Run UI suite
        id: suite
        run: pytest -q tests/ui --junitxml=report.xml

      # The Grid's state at the moment of failure. Ten seconds later it is gone.
      - name: Capture Grid evidence
        if: failure()
        run: |
          set +e
          mkdir -p grid-evidence
          curl -sf http://localhost:4444/status > grid-evidence/status.json
          curl -sf http://localhost:4444/se/grid/newsessionqueue/queue > grid-evidence/queue.json
          curl -sf -X POST -H 'Content-Type: application/json' \
            --data '{"query":"{ grid { nodeCount, totalSlots, maxSession, sessionCount, sessionQueueSize, version } }"}' \
            http://localhost:4444/graphql > grid-evidence/grid.json
          docker logs selenium-hub          > grid-evidence/hub.log          2>&1
          docker logs selenium-node-chrome  > grid-evidence/node-chrome.log  2>&1

      - name: Upload Grid evidence
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: grid-evidence
          path: grid-evidence/

Four files and two logs. Every diagnosis in this article can be made from that bundle without reproducing anything, which is the difference between a fifteen minute investigation and a week of "let's add more logging and wait for it to happen again".

If you need more than that, the Grid is instrumented with OpenTelemetry and traces every request end to end. The documentation notes that tracing is on by default, that the console exporter logs spans at FINE while the server prints INFO and above by default, and that --log-level FINE is how you see them. There is also --http-logs, which the CLI options page notes requires tracing to be enabled. Both are debugging tools, not steady-state configuration, for reasons covered below.

The costs, stated honestly

--reject-unsupported-caps will reject valid requests on an elastic Grid. The flag's own documentation scopes it to Grids that do not spin up Nodes on demand. Enabling it on Kubernetes with autoscaling nodes converts a slow success into a fast failure, which is strictly worse.

Shorter timeouts surface real capacity problems as red builds. Dropping --session-request-timeout from 300 to 120 does not make the Grid smaller; it makes an existing shortage visible sooner. That is the point, and it will still feel like the change broke CI. Say so up front when you make it.

--log-level FINE is expensive on a busy Grid. Per-request span logging is a large multiple of the default volume, and it lands in whatever log ingestion you pay for. Put it on a debug deployment or behind a flag you flip during an incident.

The evidence bundle contains capabilities, which contain whatever you put in them. Session capabilities can carry cloud provider access keys, build names, tunnel identifiers, and internal hostnames. Redact before uploading, or scope the artifact so it is not world-readable in a public repository.

--override-max-sessions trades a clean failure for a dirty one. More slots than CPUs means the queue timeout stops firing and browser instability starts. Move in small increments and watch which failure class replaces which.

Rolling it out

Add the failure hook first, change nothing else. It is the only step with no downside, and after one week of real failures you will have data instead of theories. Most teams find their actual root cause here without touching a single flag.

Classify a month of failures by error string. Use the table near the top. The distribution is the plan: mostly Timed out creating session with a full queue means capacity; mostly the same message with an empty queue means capability matching; mostly Cannot find session with id means node or browser stability. These need different fixes and you should not guess which one you have.

Pin the image versions. Do this before tuning anything, because tuning a Grid whose components re-pair themselves on every rebuild is not tuning, it is gambling.

Then change one flag at a time. --reject-unsupported-caps on a static Grid, or --register-shutdown-on-failure on a containerised one, are the two highest-yield single changes. Make them separately so you can attribute the effect.

Put the error-string table in the runbook. The lasting win is that the next person to see Timed out creating session knows within thirty seconds whether to look at capacity or capabilities. That is worth more than any configuration change here.

For the adjacent territory: zero-downtime node draining covers the DRAINING state you will see in /status during upgrades, event bus connectivity debugging goes deeper on registration failures, and multi-region Grid architecture covers what changes when the Router and Nodes are not on the same network.

When not to do this

When you have not ruled out the client side. If every test fails identically, including from a laptop, the Grid may be innocent. A DNS change, an expired proxy credential, a corporate TLS interception, or a wrong command_executor URL all produce total failure with no Grid involvement. curl -sf $GRID/status from the machine running the tests answers this in one second, and if it fails you are debugging networking, not Selenium.

When the failure names two version numbers. This version of ChromeDriver only supports Chrome version N is a driver refusing a browser on the Node. Grid endpoints will show you a perfectly healthy Grid, because it is. Go fix the pairing.

When you do not run the Grid. On a cloud provider, /status and /graphql are usually not exposed to you, the queue is theirs, and the concurrency limit is contractual rather than technical. The correct investigation there is their dashboard, their session logs, and their support channel. Building local tooling against endpoints you cannot reach is time spent for nothing.

When the real problem is test design. A suite that needs 40 concurrent browsers to finish in time, where 30 of those tests are asserting business logic that has an API, does not have a Grid problem. Moving those tests down the stack removes the queue pressure permanently and makes the remaining browser tests faster and more stable. Scaling the Grid to serve tests that should not be browser tests is expensive and it never ends.

When you are mid-incident and reaching for --log-level FINE on production. Turning on span logging during an outage adds log volume to a system already under stress, and the thing you need is the evidence bundle, which is four cheap HTTP calls. Save the tracing for the reproduction afterwards.

FAQ

Why does a request for a browser nobody has take five minutes to fail?

Because the Grid assumes a Node might still show up. The Distributor marks an unmatched request as retryable and puts it back on the queue rather than rejecting it, which is correct behaviour when Nodes scale on demand and one is mid-registration. The request then ages out at the session request timeout, 300 seconds by default. Setting --reject-unsupported-caps makes the Distributor check the request against currently UP Nodes and fail it immediately with No nodes support the capabilities in the request.

How do I tell a queue timeout apart from a browser crash?

Look for a session id. A queue timeout never gets one: the message is Timed out creating session or New session request timed out, and the Grid has nothing to show you because nothing was ever allocated. A browser that died mid-test produces NoSuchSessionException with an actual id in it, and the Node log will carry a matching Session id <id> timed out, stopping... line if it was killed for inactivity.

Is /status or the GraphQL endpoint the better diagnostic?

They answer different questions and you want both. GET /status is the liveness view, giving you value.ready, a message, and per-Node availability, slots, and stereotypes, which is what you need for "is this Grid healthy right now". The GraphQL endpoint exposes grid.sessionQueueSize and sessionsInfo.sessionQueueRequests, which is what you need for "was there a queue when my test failed", and there is no /status field that answers that.

Should I capture evidence on every run or only on failure?

Failure only, with one exception. A snapshot on every run is cheap to write and expensive to store, and nobody reads the 400 passing ones. Capture the bundle when a job fails and additionally once at the start of every run, because the starting state tells you whether the Grid was already degraded before your tests touched it.

Does raising the session request timeout fix queue starvation?

It hides it. A longer timeout converts a fast failure into a slow one and lets requests pile up further, which makes the next incident worse rather than better. If sessionQueueSize is consistently above zero while every slot is busy, the honest reading is that the Grid is undersized for the concurrency you are asking of it, and the fix is more slots or less parallelism.

What does --log-level FINE actually cost?

Volume, mostly. The Selenium docs note that the console trace exporter logs spans at FINE while the server prints INFO and above by default, so turning it on gives you per-request tracing detail and a much larger log stream. On a busy Grid that is real disk and real ingestion cost, which is why it belongs on a debug deployment or behind a flag rather than on permanently in production.

Practise the triage

Take a failure into the QABattle arena with nothing but the exception string. Name the component that emitted it, the single endpoint that would confirm your call, and the one field in that response you would read first. Three answers, thirty seconds. That is what separates a Grid incident that closes in five minutes from one that closes as "flaky".