When Selenium Grid forgets where a live browser is running

A Router restarts during a long regression run, comes back healthy, and immediately returns “no such session” for browsers that are visibly still open on the Nodes. Re-running the tests hides the incident, but it also throws away twenty minutes of valid work. The lost state is not the page, cookie jar, or browser process. It is the small ownership record that tells Grid where each session lives.

Follow the command path before changing storage

Every WebDriver command after session creation contains a session ID. The Router cannot broadcast that command to every Node and hope one claims it. It asks the Session Map for the URI associated with the ID, then sends the command to that Node. The map therefore sits on the command path even though it does not execute browser automation itself.

That distinction matters during incident review. The Distributor decides where a new session should start. The Node owns the browser and executes commands. The Session Map records the relationship between the new session ID and the Node address. Replacing the map's storage changes the durability and availability of that relationship. It does not change slot matching, browser startup, queue ordering, Node health checks, or test isolation.

In a local or Standalone deployment, keeping the map in memory is reasonable. The Grid and its state share one process boundary. If that process dies, the browser sessions are usually treated as lost too. Fully distributed deployments create a different situation. Routers, the Session Map service, the Distributor, and Nodes can restart independently. A local map inside a replaceable Session Map process turns a routine control-plane restart into a session-routing outage.

The useful mental model is a directory, not a browser backup. For session 8d1..., the directory says “send commands to http://node-7:5555.” If the directory disappears while node-7 and Chrome remain healthy, the browser exists but cannot be found through the Router. If the directory remains after node-7 dies, the Router faithfully sends traffic to an address that cannot complete it. Durability solves the first failure and can expose the second more clearly.

Selenium's external datastore support provides JDBC and Redis-backed Session Map implementations. Both are loaded as extensions and run behind the normal Session Map HTTP service. Routers and Distributors continue using the Session Map service URL. They do not connect directly to PostgreSQL or Redis. That boundary is valuable: datastore credentials stay with the Session Map component, while the rest of Grid depends on its stable HTTP contract.

A successful new session creates several observations close together:

1. The client receives a W3C new-session response containing a session ID.
2. The Node reports an occupied slot for that same ID.
3. The Session Map contains an entry for the ID and the owning Node URI.
4. A command sent through the Router reaches that Node.
5. Calling quit removes the browser session and its map entry.

Those observations should share the same session ID in your evidence. A database row from one test and a Node screenshot from another prove very little. Capture the ID at the client, query the table while the test is paused, and inspect the Node status before cleanup.

The inverse lifecycle is just as important. A normal quit should terminate the driver and remove the mapping. An entry is not intended as a permanent audit record. If compliance requires session history, export lifecycle events or logs elsewhere. Treating the live table as history encourages operators to retain stale routing data, which is exactly the wrong behavior for the Router.

Build a baseline you can actually prove

Start with PostgreSQL in a disposable environment, even if production will use a managed database. The goal is to see the full lifecycle before adding failover, connection pools, secrets, and network policy. Selenium's documented JDBC schema has five fields: the session ID, capabilities, Node URI, stereotype, and start time. Use the schema expected by the Selenium version you deploy rather than adding speculative columns or triggers.

The following local Compose file creates the documented table and waits for PostgreSQL to accept connections. The password is intentionally suitable only for a local exercise. Production credentials belong in the secret mechanism already used by your platform.

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: selenium
      POSTGRES_PASSWORD: local-only-password
      POSTGRES_DB: selenium_sessions
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U selenium -d selenium_sessions"]
      interval: 2s
      timeout: 2s
      retries: 20
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro

Create init.sql with Selenium's published column names and start the database. The primary key below is an operational hardening choice: two live owners for one session ID would be a data integrity fault, not useful redundancy. Confirm that your deployed Selenium artifact accepts the schema before applying such a constraint to an existing database.

set -euo pipefail

cat > init.sql <<'SQL'
CREATE TABLE IF NOT EXISTS sessions_map (
  session_ids varchar(256) PRIMARY KEY,
  session_caps text,
  session_uri varchar(256),
  session_stereotype text,
  session_start varchar(256)
);
SQL

docker compose up -d postgres
until docker compose exec -T postgres pg_isready -U selenium -d selenium_sessions; do
  sleep 2
done
docker compose exec -T postgres psql -U selenium -d selenium_sessions -c '\d sessions_map'

Next, configure only the Session Map component to know about JDBC. The implementation class and configuration keys below are the ones documented by Selenium. Keep the Selenium server, selenium-session-map-jdbc extension, and JDBC driver versions under change control. A server upgrade with an old extension on the classpath is not a clean compatibility test.

set -euo pipefail

SE_VERSION=4.41.0
JAR="selenium-server-$SE_VERSION.jar"

cat > sessions.toml <<'TOML'
[sessions]
implementation = "org.openqa.selenium.grid.sessionmap.jdbc.JdbcBackedSessionMap"
jdbc-url = "jdbc:postgresql://localhost:5432/selenium_sessions"
jdbc-user = "selenium"
jdbc-password = "local-only-password"
TOML

EXTENSIONS="$(coursier fetch -p \
  org.seleniumhq.selenium:selenium-session-map-jdbc:$SE_VERSION \
  org.postgresql:postgresql:42.7.5)"

java -jar "$JAR" --ext "$EXTENSIONS" sessions \
  --publish-events tcp://localhost:4442 \
  --subscribe-events tcp://localhost:4443 \
  --port 5556 \
  --config sessions.toml

Pin the sample version to the same version as your downloaded server jar. The number is not a recommendation to ignore later releases. It makes the classpath relationship visible and prevents “latest” from changing halfway through a diagnostic run.

Bring up the other distributed components with the Session Map address set to http://localhost:5556. The Router and Distributor must both point at the same service. If one still uses an embedded or different remote map, creation can appear successful while later commands fail through another Router. Compare effective startup arguments in process logs, not just a configuration repository that may not match the running containers.

A small Java probe gives you a repeatable lifecycle. It records the session ID before holding the browser open, which lets an operator query PostgreSQL and the Node while ownership is live. The final block always calls quit so a passing run also verifies deletion.

import java.net.URI;
import java.time.Duration;
import org.openqa.selenium.By;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

public final class SessionMapProbe {
  public static void main(String[] args) throws Exception {
    URI grid = URI.create(System.getenv().getOrDefault(
        "GRID_URL", "http://localhost:4444"));

    ChromeOptions options = new ChromeOptions();
    options.addArguments("--headless=new");

    RemoteWebDriver driver = new RemoteWebDriver(grid.toURL(), options);
    String sessionId = driver.getSessionId().toString();
    System.out.println("SESSION_ID=" + sessionId);

    try {
      driver.get("https://www.selenium.dev/selenium/web/web-form.html");
      String heading = driver.findElement(By.tagName("h1")).getText();
      if (!"Web form".equals(heading)) {
        throw new AssertionError("Unexpected heading: " + heading);
      }

      System.out.println("Inspect the Session Map now; waiting 45 seconds");
      Thread.sleep(Duration.ofSeconds(45).toMillis());

      driver.navigate().refresh();
      System.out.println("ROUTED_AFTER_WAIT=true");
    } finally {
      driver.quit();
      System.out.println("QUIT_SENT=true");
    }
  }
}

While the probe waits, query by the printed ID:

set -euo pipefail

SESSION_ID="$1"
docker compose exec -T postgres psql \
  -U selenium \
  -d selenium_sessions \
  -P pager=off \
  -c "SELECT session_ids, session_uri, session_start FROM sessions_map WHERE session_ids = '$SESSION_ID';"

A live run should return one row whose session_ids value exactly matches the client. The session_uri should match a registered Node, not the Router. After quit, the same query should return zero rows. Record both checks. Teams often prove insertion and forget deletion, then discover months later that their “persistent” map is full of dead ownership records.

The database is now on every routed command's dependency path through the Session Map service. Measure that cost before declaring success. Run a fixed number of simple commands through the Router with the local map, then with JDBC, and compare command latency percentiles. Do not benchmark page load, which is dominated by the application and network. A title read or current URL request is a cleaner routing probe.

Diagnose missing and stale mappings differently

Two incidents can produce the same test-level sentence, “the driver stopped responding,” while requiring opposite actions. A missing mapping means the Router cannot locate an otherwise live session. A stale mapping means it locates an owner that no longer owns a usable session.

For a missing record, the evidence lines up like this:

• The client still has a session ID.
• The target Node's status lists that session in an occupied slot.
• A direct owner check on the Node returns true.
• The Session Map lookup or backing-table query has no row.
• A command through the Router fails before the Node receives it.

Selenium exposes a Node ownership endpoint and a Session Map lookup route. Use them only on the internal network and include the configured registration secret where required. These are operational endpoints, not browser-test APIs.

set -euo pipefail

SESSION_ID="$1"
NODE_URL="$2"
SESSION_MAP_URL="$3"
REGISTRATION_SECRET="$4"

curl --fail-with-body --silent --show-error \
  -H "X-REGISTRATION-SECRET: $REGISTRATION_SECRET" \
  "$NODE_URL/se/grid/node/owner/$SESSION_ID"
printf '\n'

curl --fail-with-body --silent --show-error \
  "$SESSION_MAP_URL/se/grid/session/$SESSION_ID"
printf '\n'

When the first response is true and the second request reports that the session cannot be found, storage or lifecycle handling around the map is the failing boundary. Restarting the browser Node would destroy useful evidence. Preserve the Session Map logs, datastore availability events, and the exact create-session interval first.

For a stale record, the map returns a Node URI but that URI is wrong for current reality. There are several versions:

• DNS now resolves the hostname to a replacement Node with a different identity.
• The old Node process died and nothing listens at the stored address.
• A replacement process listens there but does not own the session.
• The Node is reachable and owns the ID, but its browser process has already exited.
• A network policy blocks Router-to-Node traffic while health checks originate from a different path.

Check the stored URI, then query that exact Node's status and owner endpoint. A connection refusal is not proof that PostgreSQL is wrong. It proves only that the Router cannot reach the recorded address at that moment. If the owner endpoint returns false, the record is stale. If it returns true and a direct WebDriver command works, investigate Router networking, TLS, or proxy configuration.

Representative evidence for a missing mapping is short and decisive:

SESSION_ID=8d1d18d6b9a64e7e
NODE_OWNER=true
DATABASE_ROWS=0
ROUTER_RESULT="no such session"
NODE_SESSION_COUNT=1

Representative stale-map evidence has a different shape:

SESSION_ID=43bb648931ce4fd5
DATABASE_ROWS=1
STORED_NODE_URI=http://grid-node-17:5555
NODE_STATUS_HTTP=000
ROUTER_RESULT="connection refused"

Those blocks are evidence formats, not commands to synthesize in production. Populate them from the same incident timestamp. The difference tells the responder whether to recover map state, remove a bad entry by ending the session, repair Node reachability, or accept that the session is already gone.

Never “fix” a stale map by manually changing session_uri to another healthy Node. A WebDriver session is stateful and belongs to the Node and browser process that created it. Another Node cannot adopt the ID merely because the database points there. Manual edits can convert an obvious routing failure into commands landing on the wrong process or a confusing ownership rejection.

Test the restart you actually care about

A basic insertion test proves configuration, not resilience. Name the component whose restart is meant to become safe and kill only that component while a session remains active.

For a Router restart, the sequence is straightforward. Create a browser through Router A, keep the Node and Session Map alive, stop Router A, start Router B with the same Session Map URL, then send another command using the existing client connection or an attached HTTP probe. The client may need to reconnect at the TCP layer, but the ownership lookup should still resolve. If a load balancer fronts multiple Routers, remove one backend rather than changing the public URL.

A Session Map service restart tests something different. With JDBC storage, stop the Java Session Map process but leave PostgreSQL and the Node running. During the outage, existing commands should fail or wait according to your client and proxy timeouts. Start a fresh Session Map process against the same database, wait for its readiness response, then issue a command for the original session. Recovery proves that the new process reads existing ownership rather than starting empty.

The pass criteria must not say “test eventually passed.” Capture:

• session ID before the restart;
• Node ID and URI before and after;
• database row before, during, and after;
• timestamps for component stop, readiness, and first successful routed command;
• client exception for commands attempted during the outage;
• row removal after normal quit.

A realistic test also includes an in-flight command. If the Router or map disappears after the Node receives a click but before the response reaches the client, retrying that click may perform the business action twice. External storage cannot provide exactly-once WebDriver semantics. The safe expectation is that later commands can be routed after recovery, not that an ambiguous command can be replayed without application-level consequences.

Run a second example with abnormal Node loss. Start a session, confirm its row, then terminate the Node process rather than the Router. The row may remain until Grid processes the lifecycle event, or it may be removed as part of cleanup. Either way, the browser died with the Node. A durable row must not be interpreted as a recoverable browser. Your alert should say “mapping points to unavailable Node,” not “session recovered.”

A third example should exercise clean client quit during a brief database interruption. If quit cannot remove the record because storage is unavailable, observe what happens when connectivity returns. Does the implementation retry, does another lifecycle event remove it, or does the row remain? Test the deployed version rather than assuming. Set an operational threshold for rows whose Node no longer reports ownership, and alert on that condition. Avoid a blind age-based deletion job because a legitimately long soak test can exceed an arbitrary lifetime.

The restart test belongs in an infrastructure verification pipeline, not every feature-test job. It is intentionally disruptive and adds minutes. Run it before Grid upgrades, datastore changes, and control-plane deployment changes. A lighter continuous check can create one session, confirm the row, execute a command, quit, and confirm deletion.

Separate lookalike failures before touching the database

A Router can return “no such session” when the test itself already quit the driver. Parallel test code may share one static RemoteWebDriver, an after-method may run early, or a framework retry may reuse an object from the previous attempt. In that case the map is correctly empty and the Node correctly has no owner. The differentiating evidence is a successful DELETE for the same session ID before the failing command.

Search client and Router logs by ID. If the timeline is create, command, delete, then another command, fix test ownership. Restoring the deleted database row would be wrong. Make each test or worker own one driver, call quit once in a finally block, and do not hand a live driver across retry attempts.

A Node crash is another near miss. Both a lost map and a dead Node break subsequent commands, but the Node's status separates them. If the Node is absent or unavailable and the browser process is gone, persisting the map did its job by retaining the last owner. There is simply no session left to route. The recovery action is to start a fresh test, then diagnose Node stability using container exit status, memory pressure, driver logs, and host events.

Version mismatch can masquerade as datastore corruption. A new Selenium server paired with an extension artifact from another release may fail during startup, deserialize records differently, or reject configuration. The strongest clue appears before any browser starts: class-loading errors, missing methods, or configuration exceptions in the Session Map process. Align artifacts before inspecting rows.

Network asymmetry also misleads responders. The Session Map can read PostgreSQL, and an operator laptop can reach the Node, while the Router Pod cannot reach the Session Map or Node. Use probes from the actual component network namespace. Compare DNS resolution and HTTP response from Router to Session Map, then from Router to the stored Node URI. A green database dashboard says nothing about those hops.

Finally, check whether multiple environments share one table. Session IDs are designed to be unique, but shared storage also mixes Node hostnames, credentials, lifecycle policies, and blast radius. A staging Router reading a production map is a configuration error even if no key collides. Use separate databases or strict datastore isolation per Grid environment.

Roll out without hiding session loss

Treat the move as a state migration, not a config toggle on all components at once. Existing sessions in an in-memory map do not magically appear in a new JDBC table. A cutover during active work can strand them even though the new design is correct.

First, inventory session duration and concurrency. Find the longest legitimate suite, the normal active-session count, and the maintenance window in which new sessions can be paused. This determines whether you can drain naturally or need a temporary overlap strategy. For most QA Grids, draining is safer than attempting to copy live rows from an internal map.

Second, deploy PostgreSQL or Redis with backups, monitoring, and access restricted to Session Map instances. Test authentication, TLS if used, connection limits, and restoration outside Selenium. A Session Map that can write only until the database reaches its connection ceiling is not production-ready.

Third, start a canary distributed Grid that uses the external map end to end. Do not point one production Router at it yet. Run the lifecycle probe, Router restart, Session Map restart, abnormal Node loss, and database interruption. Record expected errors as well as recoveries.

Fourth, stop admitting new sessions to the old Grid and allow active sessions to finish. Verify both the Grid session count and the old map state, not just the CI queue. Then switch Routers and Distributors together to the external Session Map service. A split configuration is more dangerous than a brief planned pause.

Fifth, ramp traffic by job class. Short smoke tests go first because their cleanup cycles quickly. Next add ordinary regression work. Move long-running or stateful suites after a full observation period. Compare new-session latency, routed-command errors, database connections, query latency, and stale-row count at each step.

Keep rollback explicit. If the external map fails before production sessions exist, route new work back to the old Grid. Once sessions have been created in the external map, rolling Routers back to an empty local map abandons those sessions. The rollback may therefore mean finishing or terminating current work, then starting a clean old Grid. Write that cost into the change plan.

Add a CI verification job that fails on lifecycle inconsistency, not merely HTTP readiness. The outline below starts the Java probe in the background and records its process ID, because the job has to wait for that exact process later. It then reads the session ID the probe prints, checks for one live row while the probe is still holding the browser open, waits for the probe to finish its normal quit, and checks deletion. Setting PROBE_PID at the point the probe starts is what makes the later wait valid; under set -euo pipefail, referencing an unset variable aborts the job with an unbound-variable error rather than a useful lifecycle failure.

set -euo pipefail

java -jar session-map-probe.jar > probe.log 2>&1 &
PROBE_PID="$!"

SESSION_ID=""
for _ in $(seq 1 60); do
  SESSION_ID="$(sed -n 's/^SESSION_ID=//p' probe.log | head -n 1)"
  if test -n "$SESSION_ID"; then
    break
  fi
  sleep 1
done
test -n "$SESSION_ID"

live_count="$(docker compose exec -T postgres psql \
  -U selenium -d selenium_sessions -At \
  -c "SELECT count(*) FROM sessions_map WHERE session_ids = '$SESSION_ID';")"
test "$live_count" = "1"

wait "$PROBE_PID"

dead_count="$(docker compose exec -T postgres psql \
  -U selenium -d selenium_sessions -At \
  -c "SELECT count(*) FROM sessions_map WHERE session_ids = '$SESSION_ID';")"
test "$dead_count" = "0"

In a shared environment, do not grant the CI job broad delete rights just to clean up. Its test should end its own session through WebDriver and leave unrelated rows untouched. Use a dedicated database user for read-only verification if the pipeline does not start the Session Map itself.

Know the cost and when to keep it local

JDBC adds a database, credentials, schema management, network calls, connection management, backups, and an on-call dependency. Redis adds a separate service, memory and persistence decisions, authentication, and failover behavior. Both add latency to ownership reads and writes. The absolute number may be small, but every WebDriver command needs routing, so tail latency and brief datastore pauses matter more than an average measured once.

Availability also becomes more complicated. Two Router replicas do not help if both depend on one unavailable Session Map service. Two Session Map processes do not help if they share a saturated database. A managed database does not help if DNS, certificates, or network policy block the Java processes. Draw the actual dependency chain and alert at each boundary.

The operational gain is narrow but valuable: replaceable Grid control-plane processes can recover the mapping for browser sessions that remain alive on Nodes. That is worth paying for when suites run for hours, control-plane deployments are frequent, or losing all active sessions has a measurable cost.

Do not adopt external storage for a developer's one-process Standalone Grid. Restarting that process already kills the useful session boundary, so a durable map retains pointers to browsers that are gone. The added services make local failure harder to understand without preserving work.

Avoid it as a cure for flaky tests. Element timing, shared application data, browser crashes, and bad locators do not improve because session ownership sits in PostgreSQL. If most failures happen inside healthy routed sessions, spend the effort on those causes.

Do not use it to resume a test after a Node replacement. The session's browser state is inside the old Node. A database record cannot transfer the process, open windows, cookies, network connections, or driver state. Build test-level checkpoints if a business workflow must recover across fresh sessions.

Finally, do not choose a store only because another team already runs it. JDBC is inspectable and familiar to many operations teams, but database maintenance can be heavier. Redis can be fast and simple for ephemeral ownership, but its durability and eviction settings deserve deliberate review. Pick the failure behavior your team can test and operate. The correct design is the smallest one that preserves the specific state you cannot afford to lose.