Monitor Selenium Grid Health and Readiness Endpoints

Selenium Grid health readiness endpoint monitoring should query the documented /status route for health, then apply an operator-owned readiness policy that also checks eligible Nodes, compatible capacity, queue pressure, and session ownership. Selenium does not define a separate formal readiness endpoint, so readiness is a derived admission decision rather than a second Selenium API.

That distinction prevents two costly mistakes. A monitor should not call a Grid dead merely because every slot is busy, and it should not call a Grid ready merely because an HTTP server returned 200. The first state can be healthy saturation. The second can hide an empty Grid model, missing browser stereotypes, an unreachable Session Map, or a browser that cannot start.

The official Grid endpoints reference identifies /status for Grid state and direct Node health checks. It describes availability, sessions, and slots, but it does not publish an independent readiness contract for an orchestrator. This guide turns those supported signals into a measurable policy, adds queue and capacity evidence, and tests each failure boundary without changing Selenium's documented semantics.

Define Health and Readiness as Different Decisions

Health asks whether the component can respond and report its current state. Readiness asks whether this deployment should receive a particular class of new work now. The answers overlap, but they are not interchangeable. A Router can answer /status while no Node supports Firefox. A Grid can have no idle Chrome slot while all active Chrome sessions execute normally. A Node can be DRAINING, serve its existing owner sessions, and correctly reject new allocation.

Start by writing three policies rather than one green light:

1. Liveness policy: the monitored process responds within a bounded time and returns parseable status. Failure restarts or removes only the component whose process is no longer viable.
2. General readiness policy: the Router can describe a fresh Grid model and at least one required service lane is eligible for admission. Failure removes this Router endpoint from new client traffic without claiming every component is dead.
3. Workload readiness policy: a named capability shape, such as Chrome on Linux with a team extension capability, has eligible capacity or an acceptable measured queue delay. Failure pauses that workload or sends it to another pool.

The word "ready" can appear in a status payload, but an infrastructure controller still needs a declared interpretation. Record the Grid version and policy version beside every decision. During upgrades, response fields and defaults can change, while the operational meaning your clients require should remain reviewable. Teams moving from an older topology should first confirm the component boundary in the Selenium 3 to 4 capabilities and Grid migration guide.

A useful first assertion is therefore not "status equals true." It is "the Router returned a fresh valid status model containing the minimum eligible service lane required by this probe." The first failure boundary is the Router request itself. Capacity, queue, and browser startup are later boundaries and should produce different diagnostic reasons.

Map Each Signal to Its Owning Grid Component

The official Grid architecture explains why one endpoint cannot prove every path. The Router accepts client traffic. New session requests wait in the New Session Queue. The Distributor matches requests to free slots recorded in its Grid Model. The Session Map tells the Router which Node owns an established session. A Node owns the browser process and executes commands.

Use that ownership map to avoid drawing a broad conclusion from a narrow observation. The following decision table is a policy template, not a Selenium protocol definition.

Do not count every advertised slot as executable concurrency. A Node can expose several browser stereotypes while its maximum session count limits simultaneous execution. Likewise, a free Edge slot says nothing about a queued Firefox request. The Grid GraphQL capacity dashboard guide develops that distinction for long-running monitoring.

Session ownership deserves its own signal. A Grid may accept new sessions but fail commands for existing IDs if the Session Map path is unavailable or stale. During diagnosis, inspect Session Maps and queues separately so an allocation failure does not get mislabeled as a browser command failure.

Establish a Baseline Before Setting Thresholds

A threshold copied from another Grid rarely describes your workload. Measure ordinary status latency, Node registration convergence, session creation time, queue residence time, and canary cleanup across at least one representative demand cycle. Slice the data by browser, platform, Node pool, region, and deployment state. The slow lane often disappears inside a global average.

For /status, record response duration and payload age at the monitor. Because the status document is a point-in-time model, stamp it when received and reject late out-of-order samples. Permit only one request in flight per target. If a poll takes longer than the interval, skip or cancel the older poll instead of accepting responses in reverse order. Jitter monitor start times so ten replicas do not strike the Router simultaneously.

For readiness, define the minimum registered stereotypes rather than only a minimum Node count. Two Nodes can both advertise Chrome and leave a required Firefox lane absent. Compare a normalized capability inventory with an approved deployment manifest. Treat planned drains differently from unexplained disappearance. The Node draining upgrade workflow shows why DRAINING is a valid maintenance state but not new-session capacity.

Queue policy needs duration and compatibility. A queue size of one for two seconds during burst startup can be ordinary. A queue of one Firefox request for five minutes while Chrome slots sit idle indicates a missing compatible lane. Set warning and critical conditions from percentiles and the new-session timeout policy. The session queue timeout debugging guide helps separate slow admission from requests that will expire.

Finally, decide what a monitor failure does. Liveness failure may restart a local process. General readiness failure may remove a Router from a load balancer. Workload readiness failure may pause only one CI browser matrix. Avoid letting a browser-capacity policy trigger an endless Router restart, because restarting a responsive control plane does not create a missing Node.

Model the output as more than pass or fail. healthy, degraded, not-ready, and unknown can drive different actions when their definitions are precise. A fresh status response with one redundant Node missing may be degraded but ready. An expired sample is unknown, even if its last value was green. No compatible lane is not-ready for that workload. A transport failure is unhealthy for the queried endpoint. Dashboards can preserve those states while an orchestrator still consumes a Boolean projection.

Baseline recovery as carefully as failure. Measure how long a replacement Node takes to register, how long GraphQL and /status take to converge, and when the first successful canary appears. Set the recovery threshold from the slowest required evidence, not from Pod start time. This prevents traffic from returning while the Java process is up but the required browser lane is still absent.

Implement a Status-Based Readiness Probe

The following workflow creates an auditable probe around the supported status route. It deliberately keeps policy inputs in environment variables so each Grid pool can declare its own minimum lane.

1. Send a bounded GET request to the Router or Standalone /status route used by clients.
2. Require HTTP success, valid JSON, and the expected value object before evaluating Grid state.
3. Confirm the payload's Grid-level ready field is true, while treating it as one input rather than the entire policy.
4. Select Nodes whose availability is UP; exclude DRAINING and unavailable Nodes from new-session capacity.
5. Find a free slot whose stereotype matches the probe's required browser name.
6. Emit one reason code and a sanitized evidence record, then exit nonzero when the local policy fails.

Code 1: Bash and jq readiness policy derived from /status

#!/usr/bin/env bash
set -euo pipefail

: "${GRID_URL:=http://localhost:4444}"
: "${REQUIRED_BROWSER:=chrome}"

status_file="$(mktemp)"
trap 'rm -f "$status_file"' EXIT

curl --fail --silent --show-error \
  --connect-timeout 2 --max-time 5 \
  "${GRID_URL%/}/status" >"$status_file"

jq -e --arg browser "$REQUIRED_BROWSER" '
  .value as $value
  | ($value.ready == true)
    and any(
      $value.nodes[]?;
      .availability == "UP"
      and any(
        .slots[]?;
        .session == null
        and .stereotype.browserName == $browser
      )
    )
' "$status_file" >/dev/null

jq --arg browser "$REQUIRED_BROWSER" '{
  observedAt: (now | todateiso8601),
  browser: $browser,
  gridReadyField: .value.ready,
  message: .value.message,
  upNodes: ([.value.nodes[]? | select(.availability == "UP")] | length),
  freeMatchingSlots: ([
    .value.nodes[]?
    | select(.availability == "UP")
    | .slots[]?
    | select(.session == null and .stereotype.browserName == $browser)
  ] | length)
}' "$status_file"

This probe is intentionally strict for a browser-specific admission target. It should not be used as process liveness, because ordinary saturation makes freeMatchingSlots zero. A queue-backed service may choose to remain ready while saturated if its service objective permits waiting. In that case, replace the free-slot condition with a queue-age or capacity-reservation rule.

Test the exact JSON shape against the promoted Selenium release. The official CLI documentation warns that configuration documentation can lag executable help, so run the deployed server's config help when a field or option differs. Keep the probe parser versioned with the Grid image.

Add GraphQL Capacity and Queue Context

Status is useful for registered Node and slot state. GraphQL gives a narrower query for aggregate session count, maximum concurrency, queue size, and Node status. The official GraphQL support page documents grid, nodesInfo, and sessionsInfo, including sessionQueueSize and Node status values.

Code 2: GraphQL query for readiness context

query ReadinessContext {
  grid {
    version
    nodeCount
    totalSlots
    maxSession
    sessionCount
    sessionQueueSize
  }
  nodesInfo {
    nodes {
      id
      uri
      status
      maxSession
      slotCount
      sessionCount
      stereotypes
    }
  }
}

Post this query to the Router's /graphql route from a trusted monitor. Validate HTTP status, JSON shape, GraphQL errors, required fields, and nonnegative finite numbers before replacing the last good sample. HTTP 200 with an errors array is not a valid capacity observation. A failed parse should mark the sample stale, not replace counts with zero.

Calculate global concurrency headroom as maxSession - sessionCount, bounded at zero. Display totalSlots separately because advertised browser choices and simultaneous execution are different quantities. At Node level, exclude statuses other than UP from new admission, then limit apparent free slots by remaining Node concurrency. When the queue is nonzero, compare queued capability shapes with eligible stereotypes before declaring spare capacity.

Do not fetch complete queued request payloads for a simple readiness decision. Capabilities may contain test names, provider options, URLs, or credentials placed there by a client. Queue size is enough for a general pressure signal. If a protected diagnostic classifier needs request shapes, sanitize and retain them briefly. The existing capacity dashboard implementation provides a fuller polling and stale-sample model.

GraphQL and /status should agree on stable facts such as registered Node count over a settled interval, but they need not be sampled at the same instant. Attach observation timestamps and avoid failing a release on a one-poll race during registration. Repeated disagreement after convergence is evidence worth preserving.

Exercise Bounded Failure Cases

Readiness becomes trustworthy only after the team has watched it fail for the intended reasons. Run experiments in a disposable staging Grid with bounded blast radius. Preserve before, during, and recovered samples so the assertion includes transition behavior rather than one final screenshot.

Use this minimum test set:

For multi-component outages, change one dependency at a time first. A simultaneous Router, Event Bus, and Node kill proves only that a large outage is bad. Once individual expectations are stable, combine failures that match real fault domains. The multi-region Grid architecture guide is useful when the test must distinguish local pool readiness from global routing policy.

Recovery assertions matter. A Node restart should not mark the required lane ready until registration has converged and a canary succeeds. A transient status failure should clear only after several fresh passing observations if the alert policy uses hysteresis. A drained Node should not be counted while it still owns sessions. Track time to detect and time to recover separately.

Never use the queue-clear or session-delete endpoints as a probe. Those are mutating control operations. A monitor that clears waiting work to see whether the queue is reachable destroys the evidence and harms clients. Use GET status, narrow GraphQL queries, and a uniquely labeled canary session that always quits.

Place Probes in Kubernetes and CI Carefully

In Kubernetes, a startup probe can wait for the Java process and Grid role to initialize. A liveness probe can test the local component's bounded status response. A readiness probe can apply the deployment's traffic policy, but capability-specific admission is often better handled by an external monitor than by restarting or removing every Pod that lacks one browser lane.

For disposable browser workers, separate Node Pod readiness from whole-Grid readiness. A Node should not receive registration or scheduling assumptions until its driver and browser prerequisites exist, while the Router remains available as other Nodes rotate. The guide to running Grid on Kubernetes with disposable Nodes covers lifecycle choices that affect this boundary.

CI should run two classes of check. A fast configuration gate parses the probe, validates expected stereotypes against deployment configuration, and checks that secrets are not printed. A staging smoke gate starts one session for each critical capability through the public Router, asserts a small browser action, records the session ID, and quits in finally. Never use a shared production customer's session as the canary.

Pin the Grid container or server version, probe revision, browser image, and policy manifest in evidence. During a rolling change, compare old and new pools independently before shifting traffic. A global success can hide that all canaries landed on the old pool. Give each canary a capability label or trace attribute that identifies the candidate deployment.

The Grid CLI options reference documents Router authentication, sub-paths, registration secrets, health-check intervals, session queue controls, and the Distributor's Node checks. Account for a configured sub-path when constructing both /status and /graphql. Keep registration secrets away from read-only probes, and never expose direct Node control endpoints to the public network.

Debug False Green and False Red Results

A false green begins with scope. Confirm the monitor queried the same Router and environment as clients. Then verify the sample timestamp, Grid version, Node identities, required stereotype, and GraphQL errors. A cached reverse-proxy response or stale service discovery target can present an attractive status document from the wrong pool. Disable caching for operational routes and include a Grid identity in monitor configuration.

If /status passes but session creation fails, move downstream in order: queue acceptance, Distributor matching, Node reservation, driver startup, browser startup, Session Map insertion, and response routing. Preserve a test ID across logs and traces. The Grid trace correlation guide shows how to connect that chronology without guessing from timestamps alone.

A false red often comes from treating capacity as liveness. Check whether the policy failed because all matching slots were occupied, the required browser name differed in case or spelling, a Node was intentionally draining, or the status sample arrived during registration. Compare the raw response with the normalized decision record. The monitor must expose which predicate failed, not only an exit code.

Reproduce a disputed result from the saved sample before querying live state again. A later healthy response cannot explain why an earlier policy failed, and a screenshot omits fields the evaluator may have used. Store the sanitized input, evaluator version, normalized lane key, and resulting predicates together. That bundle lets reviewers distinguish a parser defect from a real transition without replaying a destructive outage.

Protect the monitoring surface. Put the Router behind authenticated HTTPS where required, restrict GraphQL and status access to operational networks, apply rate controls, and redact Node URIs if incident artifacts leave the platform team. Do not include registration secrets in curl command lines that process listings or CI logs can reveal. Queue request details and capabilities need stricter handling than aggregate counts.

Alert on sustained, fresh conditions with an owner and runbook. Include target, policy version, failed predicate, first observed time, last good time, browser lane, Node status summary, queue size, and canary result. Page for loss of a critical service lane or control path. Ticket or trend short saturation that remains inside the agreed queue objective.

Frequently Asked Questions

Does Selenium Grid provide a separate readiness endpoint?

No. Selenium documents the /status route for Grid and Node state, not a second formal readiness endpoint. An operator can derive readiness from that response, compatible capacity, queue conditions, sample freshness, and a canary, but the resulting pass or fail decision belongs to the deployment policy.

Should a full Selenium Grid fail its liveness check?

Usually no. A Grid with every compatible slot occupied may be healthy but temporarily unable to admit another matching session. Keep liveness focused on process and control-path response, then represent saturation through readiness, capacity alerts, or client-side queue expectations according to the service objective.

What should a Selenium Grid readiness policy check?

Check that the /status sample is fresh and valid, required Nodes are UP, expected stereotypes remain registered, and the target workload has admissible capacity or an acceptable queue wait. For stronger evidence, create and quit a bounded canary session through the same Router used by test clients.

Can HTTP 200 from /status prove that new sessions will start?

No. HTTP 200 proves that the request reached a handler and produced a response. The payload may describe unavailable Nodes, missing stereotypes, or no free capacity. Session creation also depends on the Router, queue, Distributor, Session Map, matching Node, browser driver, and browser startup path.

How often should Selenium Grid monitoring poll?

Choose an interval shorter than the detection objective but long enough to avoid control-plane pressure. Start with measured response time, allow only one bounded request in flight, add jitter across monitors, and alert from several fresh observations. A single failed poll during deployment should not automatically page an operator.

Which evidence belongs in a Grid readiness incident?

Preserve the raw sanitized /status response, GraphQL capacity snapshot, observation time, Grid version, Router address, queue size, required stereotype, Node availability, and canary result. Add correlated Grid logs or traces when session creation fails, while removing credentials, internal secrets, and sensitive queued capabilities.

Put the Monitoring Policy Into Practice

Begin with one Router, one required browser lane, and four recorded outcomes: healthy idle, healthy saturated, missing compatible Node, and failed browser startup. Make the reason code differ for each outcome. Then run the probe during a controlled Node drain and verify that active sessions remain owned while new allocation excludes that Node.

Use QABattle's automation battles to practice turning an observed system state into a precise test decision. Pair that exercise with the session and queue inspection guide and the trace-correlation workflow. The monitoring design is ready when an operator can tell which component boundary failed, which workload is affected, and which safe action follows from the evidence.