Selenium Grid 4 Failure Domains and High-Availability Boundaries

High availability for Selenium Grid is not one replica count. A Grid request crosses ingress, queue, assignment, session lookup, event propagation, and a stateful browser Node. Redundancy at one layer protects only that layer; a live browser still disappears when its owning Node or browser process fails.

Design availability by naming state, recovery owner, and user-visible consequence for every component. Then test those boundaries with controlled failures instead of assuming a distributed deployment is resilient because its processes are separate.

Define the Service You Are Protecting

Grid has at least two availability objectives. Admission availability asks whether a new valid session request can be accepted and assigned. Command availability asks whether an existing session can continue receiving commands. They depend on overlapping but different components.

The official Grid architecture guide and component reference identify the Event Bus, New Session Queue, Distributor, Node, Session Map, and Router as distinct roles. Use those roles as the starting fault model. Add the load balancer, database, DNS, certificates, container scheduler, browser images, and client network because production requests depend on them too.

Separate Stateless Ingress from Stateful Routing

The Router is the Grid front end. New-session requests go toward the queue, while commands for active sessions must reach the Node recorded for that session. Multiple Router instances behind a health-aware load balancer can remove one ingress process as a single point, provided every Router can reach the same downstream components and routing state.

Do not use a shallow TCP check as the entire readiness contract. A Router can accept a connection while the queue, Session Map, or Distributor is unavailable. Probe the public /status path, then run a synthetic new session and a command through the same TLS, authentication, DNS, and proxy route used by clients.

Keep the public boundary narrow. The Selenium documentation cautions against broadly exposing Grid. Put authentication, encryption, request limits, and network policy at ingress, and keep component ports private. Availability controls that make an unauthenticated browser-control endpoint easier to reach are a security regression.

Treat Queue Survival as an Explicit Property

The New Session Queue owns requests that have been accepted but not assigned. A queue process failure can affect admission differently from active commands, because established sessions do not need to re-enter it. The key design question is what happens to requests already waiting.

Call the desired boundary durable only after proving the deployment's behavior. Does queued state survive restart? Are requests replayed, rejected, or lost? Can clients tell whether a timed-out request still exists? The architecture diagram expresses the reliability requirement; it is not a claim that component mode automatically supplies persistence or replication.

Client retry policy must match that answer. Blind resubmission can duplicate demand when the original request remains queued. A clear request deadline, correlation identity, and queue observation path let the runner decide whether to retry after failure.

Protect Distributor Correctness Before Replica Count

The Distributor maintains the Grid model and assigns queued requests to eligible slots. Availability requires current Node status, a functioning queue connection, and a topology that coordinates assignment correctly. Starting another process does not prove that multiple Distributors can safely share work under every deployment mode.

Use the topology documented and tested for the selected Selenium artifact. Verify startup dependencies, registration, Event Bus addresses, Session Map address, and queue address from rendered configuration. During failover testing, confirm that each request receives at most one assignment and that released slots return to the model.

A stale model can show idle capacity that no longer exists or omit a newly registered Node. GraphQL inventory, Distributor logs, Node status, and Event Bus evidence should share deployment and correlation metadata. Restarting all control-plane processes together may restore service but erases the failing relationship.

Understand What an External Session Map Saves

The Session Map stores the relationship between a session ID and the Node URI. Moving that mapping to an external database or Redis-backed implementation can remove process-local routing state as a recovery obstacle. The Selenium external datastore guide documents supported external Session Map approaches.

Grid external Session Map configuration:

[sessions]
implementation = "org.openqa.selenium.grid.sessionmap.jdbc.JdbcBackedSessionMap"
jdbc-url = "jdbc:postgresql://session-db.internal:5432/selenium_sessions"
jdbc-user = "selenium_grid"
jdbc-password = "REDACTED_AT_RUNTIME"

This fragment shows the ownership boundary, not a complete secret-management design. Supply credentials through the deployment's protected rendering process and do not commit real values. Make the database itself highly available according to its operator's practices, including backups, connection limits, TLS, monitoring, and schema lifecycle.

An external map does not contain the browser's heap, DOM, profile, process handles, or network connections. If the mapped Node is dead, preserving its URI cannot resurrect the session. It enables surviving Routers and map services to find sessions whose Nodes remain healthy.

Include the Event Bus in the Failure Model

Grid uses synchronous HTTP for some communication and asynchronous Event Bus messages for other coordination. Node registration and state propagation depend on that event path. If publish or subscribe connectivity breaks, components may keep running while their shared model stops reflecting reality.

Monitor Event Bus reachability from each role, not only from an external dashboard. Network policy, DNS, certificate, and advertised-address errors can be directional. Capture component startup logs because an incorrect subscribe endpoint can look like an empty Grid rather than a crashed bus.

Recovery should prove convergence: expected Nodes register, statuses update, stereotypes match configuration, and the Distributor assigns a test session. A green Event Bus port alone does not establish that consumers are subscribed to the right channels.

Make Nodes Replaceable but Not Pretend Sessions Migrate

A Node owns browser and driver processes for its active slots. If the Node, container, host, driver, or browser fails, those sessions are lost. Replacement restores future capacity; it does not continue in-memory browser state. Test orchestration must mark affected attempts failed and decide whether a fresh attempt is allowed.

Build Nodes from immutable browser images, advertise deterministic stereotypes, and keep test data outside the Node when it must survive an attempt. Store artifacts continuously or flush at meaningful checkpoints when crash evidence matters, since a terminated ephemeral Node may never execute teardown.

Drain a healthy Node before planned maintenance so it stops receiving new sessions while existing work finishes under policy. Forced termination remains necessary after a deadline, but the resulting test failures should be classified as maintenance interruption rather than application defects.

Map Failures to Observable Consequences

Router loss should remove one ingress path while another serves traffic. Queue loss can reject or lose unassigned work according to proven semantics. Distributor loss blocks new assignment. Session Map loss prevents routing of active commands even when Nodes remain alive. Event Bus loss can stale registration and slot state. Node loss terminates its own sessions.

Write one fault-injection case for each consequence. Capture request outcome, recovery time, queue state, active-session behavior, and resource cleanup. Do not invent a universal objective; set budgets from the suite's feedback requirements and infrastructure constraints.

Run failures during idle periods and under controlled load. Some defects appear only with queued work or active sessions. Ensure tests cannot reach production or uncontrolled external targets, and isolate experiments from normal CI capacity.

Design Recovery Ownership and Runbooks

Every alert should name the operator and the first evidence source. Ingress belongs to the platform edge owner. Grid control-plane components belong to the Grid operator. Browser images and driver compatibility belong to the Node image owner. Test retry and data reset belong to framework owners. Database recovery belongs to its service owner.

Runbooks should avoid global restarts as the default. Preserve logs and GraphQL state, isolate the failed component, replace it, and verify the smallest end-to-end flow. For active-session routing incidents, test a known surviving session before creating only new ones; admission success does not prove command continuity.

Document degraded modes. The organization may choose to stop accepting new sessions when the external map is unhealthy, or reject specialized pools while general pools remain available. Explicit degradation is safer than returning intermittent failures with no ownership signal.

Weigh High-Availability Tradeoffs Honestly

Redundancy adds load balancers, state stores, certificates, service discovery, monitoring, upgrades, and coordination. It can reduce interruption from a process failure while increasing configuration and observability demands. A small team may get better reliability from a simpler Standalone deployment that is quickly replaced than from an untested distributed topology.

Use measured failure cost to choose boundaries. If browser startup is cheap and tests are retryable, fast replacement may matter more than preserving queued requests. If a long-running regulated flow cannot be replayed, Selenium browser automation may need checkpointing at the business-process layer because Grid cannot migrate the live session.

High availability also changes upgrade procedures. Components sharing protocol and state contracts should be promoted through a compatibility plan, with rollback that accounts for database and queue state rather than only container images.

Close with Tested Failure Boundaries

Grid resilience is the sum of explicit guarantees: reachable ingress, known queue survival, correct assignment, recoverable session routing, convergent event propagation, and replaceable Nodes. None of those guarantees implies that a lost browser session will move elsewhere.

Draw the dependency path, assign each stateful boundary an owner, and inject one failure at a time. The architecture is highly available only where observed recovery proves it. Everywhere else, fail clearly, preserve evidence, and start a fresh browser attempt under a deliberate policy.