Drain Selenium Grid Nodes for Zero-Downtime Browser Upgrades

A browser upgrade on Selenium Grid is an allocation change, not a container restart with nicer timing. The safe operation has five distinct states: preserve spare capacity, mark one node as draining, let its owned sessions finish, replace the browser image, and prove that the new slots can accept matching requests. Skip any state and a rolling deployment can become either a session outage or a silent loss of browser coverage.

The key constraint is immovable session ownership. A running WebDriver session remains attached to its node and browser process. Grid can stop routing new work to that node, but it cannot transfer the live session to another host. Zero downtime therefore means continuous admission for new sessions while old sessions complete, not uninterrupted survival after a node is killed.

Treat Draining as an Allocation State

The Grid architecture reference separates the Distributor's model of available slots from the sessions actually running on Nodes. A node whose availability is draining should receive no new sessions. Its existing slot sessions continue until clients quit them or Grid ends them under the configured lifecycle.

That distinction makes draining safer than removing. Removal tells the Distributor to forget a node, while the process and its current sessions can continue outside the model. Draining keeps the intended shutdown visible and allows the node to stop after its final session. Use removal for recovery from a lost or permanently broken node, not as the normal first step of an upgrade.

Establish Capacity and Identity Before the Roll

Inventory nodes from the Router's /status response before changing anything. Record each node ID, external URI, availability, slot stereotypes, and active session IDs. Node identity matters because the Distributor drain endpoint targets a UUID, not a host label that an orchestrator may reuse. Save the pre-change response with the deployment record so a later mismatch is diagnosable.

Capacity must be evaluated by stereotype, not only by total slots. Ten idle Chrome slots do not protect a Firefox-only lane. For each capability shape used by CI, check that the remaining up nodes can accept the expected concurrent starts while one node is absent. Also inspect the new-session queue and stop the rollout when queue age or creation failures begin rising. A rolling operation is conditional on headroom; it is not a timer that advances regardless of demand.

Choose the unit of failure deliberately. Drain one node at a time when nodes are the independent failure domains. In an availability zone or host pool, avoid selecting several nodes that remove the same browser stereotype together. The deployment controller should persist the selected node ID and replacement workload ID so a retry continues the same operation rather than draining a second node.

Issue the Authenticated Drain Request

The official Grid endpoints documentation defines a Distributor drain endpoint and a direct Node drain endpoint. Prefer the Distributor route through the Router because the control plane already knows the node ID and can update its model. Configure a registration secret and keep it in the deployment system's secret store; do not print it in CI output.

set -euo pipefail

: "${GRID_URL:?Set the Router URL}"
: "${NODE_ID:?Set the exact node UUID}"
: "${GRID_REGISTRATION_SECRET:?Set the Grid registration secret}"

curl --fail-with-body --silent --show-error \
  --request POST \
  --header "X-REGISTRATION-SECRET: ${GRID_REGISTRATION_SECRET}" \
  "${GRID_URL}/se/grid/distributor/node/${NODE_ID}/drain"

Treat a successful HTTP response as acceptance of the transition, not proof that shutdown finished. Make the operation idempotent at the deployment layer by recording that drain was requested. If a retry receives an unexpected response, query status before deciding whether to repeat, wait, or fail. Never compensate for ambiguity by killing the node immediately.

Observe Sessions Until Ownership Ends

Poll Grid status and locate the selected node by ID. During the drain, its availability may report draining, and its slots reveal which sessions remain. Completion is reached when the node has no active slot sessions and shuts down, or when it disappears after the expected graceful exit. A node that disappears while sessions were still active is a failed upgrade, not a successful drain.

status_json="$(curl --fail --silent --show-error "${GRID_URL}/status")"

active_sessions="$(
  jq --arg id "${NODE_ID}" '
    [.value.nodes[]? | select(.id == $id) | .slots[]? | select(.session != null)]
    | length
  ' <<<"${status_json}"
)"

node_count="$(
  jq --arg id "${NODE_ID}" '[.value.nodes[]? | select(.id == $id)] | length' \
    <<<"${status_json}"
)"

printf 'node_present=%s active_sessions=%s\n' "${node_count}" "${active_sessions}"

Set an operational deadline longer than normal test completion, but do not turn deadline expiry into an automatic hard kill. First identify owners from session IDs, find the CI jobs, and decide whether they are legitimately long-running or abandoned. A client that never calls quit() can hold the drain open; fixing lifecycle cleanup is preferable to hiding it behind forceful infrastructure recycling.

Replace the Browser Image Without Opening a Gap

Once graceful shutdown is confirmed, replace the workload using an immutable browser image and the same declared Grid configuration. Keep the old image reference available for rollback. Browser binary, matching driver behavior, fonts, locale, certificates, shared memory settings, and container limits all belong to the tested unit; changing several of them without recording the image digest makes regressions hard to isolate.

The Grid CLI option reference includes --drain-after-session-count for nodes that should recycle after a configured number of sessions. That policy is useful for disposable environments, but it solves a different trigger. An upgrade controller still needs to coordinate spare capacity, image replacement, readiness, and rollout order.

Do not start draining the next old node as soon as the replacement process is merely running. Wait for registration and capability verification. Process health says Java is alive; Grid readiness says the Distributor can see usable slots; test readiness says a client can complete a session through the actual Router path.

Re-register and Prove the Replacement Slots

The replacement Node registers by heartbeat and status exchange with the Distributor. Confirm its reported URI is reachable from Grid components, especially when NAT, proxies, or container networks change addresses. Compare its slot stereotypes with the pre-change node. A typo in browserName, platform metadata, or vendor-prefixed capability can leave the total slot count looking healthy while real requests remain queued.

Run a canary through the Router using one representative request per changed stereotype. The canary should create a session, navigate to a controlled page, perform a basic assertion, and quit. Then verify that the session disappears from the status model. This tests admission, routing, browser startup, command execution, and cleanup without pretending to replace the full suite.

Only after the canary succeeds should the controller mark that node complete and select the next candidate. If it fails, preserve the new node logs and status snapshot, remove or drain the bad replacement as appropriate, restore the previous image, and repeat the same readiness checks.

Analyze Failures by Transition

If new sessions still land on the selected node, confirm that the request targeted the correct node ID and Router, that authentication succeeded, and that the Distributor status shows draining. A hostname match in deployment tooling is not enough when a restarted process has a different UUID.

If draining never completes, inspect remaining slot sessions and client ownership. Common causes are missing quit() calls, hung tests, a session timeout that does not match expectations, or a monitoring script counting slots rather than non-null sessions. Preserve evidence before deleting a session because forced deletion changes the failure you are trying to understand.

If the replacement registers but receives no work, compare stereotypes and requested capabilities. Also validate the external URL and component connectivity. If the queue grows during every node rotation, the supposed spare capacity is not spare under real arrival patterns; lower rollout concurrency or schedule the operation during a measured demand window.

Accept the Tradeoffs Explicitly

Draining protects active sessions but can make an upgrade wait for the slowest test. Forced termination gives a bounded deployment but breaks in-flight jobs. A sound policy names who may force termination, after what investigation, and how affected jobs are reported or retried. It does not quietly redefine a killed session as zero downtime.

Fine-grained nodes reduce the amount of capacity removed per step, but increase registration, scheduling, and orchestration overhead. Larger nodes are simpler to operate, yet each drain removes more simultaneous slots. Choose topology from workload isolation and failure-domain needs, then design the rollout around that topology.

Automatic --drain-after-session-count recycling limits node age, while explicit drain endpoints support controlled maintenance. They can coexist, but the controller must recognize nodes already draining for policy reasons and avoid counting them as rollout headroom.

Use a Rollout Checklist That Can Stop

• Snapshot Grid status, queue indicators, node IDs, slot stereotypes, and active sessions.
• Confirm healthy matching capacity remains after one selected node stops accepting work.
• Send one authenticated drain request and persist the operation identity.
• Observe the selected node until all owned sessions end and graceful shutdown is confirmed.
• Replace only that workload with the recorded browser image and Grid configuration.
• Wait for registration, compare stereotypes, and execute a Router-level canary session.
• Preserve logs and status on failure; rollback before touching another node.
• Advance one failure domain at a time, with queue growth and session-creation errors as stop signals.

Finish With Evidence, Not a Green Process Check

A reliable browser upgrade is complete when every intended node reports the new image, every required capability has healthy registered slots, canary sessions create and quit through the Router, and no pre-upgrade session was killed by the rollout. Keep the before-and-after status snapshots with the deployment. That evidence turns draining from a hopeful delay into a controlled Grid state transition.