Stop dead test runners from holding Grid slots

A runner is cancelled halfway through a browser test, but the Grid dashboard still shows its Chrome session occupying the only slot. Ten minutes later, every new test is waiting behind a client process that no longer exists. Restarting the pipeline adds more requests; it does not release the browser left on the Node.

Understand what survives when the runner dies

Remote WebDriver has two processes with independent lifetimes. Test code and the Selenium client library run on the CI worker. The browser, browser driver, and Grid Node run somewhere else. Calling driver.quit() sends the WebDriver delete-session command across that boundary. The Node then terminates the browser resources and frees the slot.

A normal assertion failure should not prevent that command. JUnit, pytest, and other test frameworks have teardown mechanisms that run after a failed test. A process crash, forced container deletion, machine loss, or SIGKILL is different. No language-level finally block can execute after the process has ceased to exist. The Node has no immediate way to know whether the missing client is dead, paused, or temporarily disconnected.

Selenium's Node session timeout is the built-in backstop. The documented --session-timeout option makes a Node kill a session after that many seconds without activity. “Activity” here means commands reaching the session. It is not browser page activity, network traffic from the page, or a heartbeat from the test runner. A test that sleeps for ten minutes after its last WebDriver command can look exactly like an abandoned client to this timer.

Several Grid records participate in the lifecycle:

• the Session Map associates a session ID with the Node that owns it;
• the Node owns the browser process and the slot;
• the Distributor keeps its model of registered Nodes and slots;
• the Router uses the Session Map to send session commands to the right Node.

Those records explain why “I still see a row in the UI” is not enough evidence. The session can be live and usable. Its Node can be unreachable. The Grid can be converging after a Node failure. The browser process can even outlive a control-plane record when its host is isolated. Cleanup depends on which state you actually have.

Use a strict operational definition: a session is orphaned when its authorized client can no longer use it, no replacement client is expected to adopt it, and ending it cannot interrupt valid work. The third condition prevents an age-based janitor from killing long-running tests, local debugging sessions, or a job whose log stream was merely delayed.

Explicit teardown remains the primary control. This JUnit 5 example creates the remote driver in setup, prints the session ID for correlation, and attempts to quit after every test. A teardown failure is reported with the session ID so operations can inspect the exact remote object instead of searching by timestamp.

import java.net.MalformedURLException;
import java.net.URL;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

class AccountSettingsTest {
  private RemoteWebDriver driver;
  private String sessionId;

  @BeforeEach
  void openBrowser() throws MalformedURLException {
    URL gridUrl = new URL(System.getenv("SELENIUM_GRID_URL"));
    ChromeOptions options = new ChromeOptions();
    options.addArguments("--headless=new");
    driver = new RemoteWebDriver(gridUrl, options);
    sessionId = driver.getSessionId().toString();
    System.out.printf("grid_session_id=%s%n", sessionId);
  }

  @AfterEach
  void closeBrowser() {
    if (driver == null) {
      return;
    }

    try {
      driver.quit();
      System.out.printf("grid_session_closed=%s%n", sessionId);
    } catch (WebDriverException error) {
      System.err.printf(
          "grid_session_close_failed=%s error=%s%n",
          sessionId,
          error.getMessage());
    } finally {
      driver = null;
    }
  }

  @Test
  void updatesTheTimezone() {
    driver.get(System.getenv("APP_URL") + "/settings");
    driver.findElement(By.id("timezone")).sendKeys("Asia/Kolkata");
    driver.findElement(By.cssSelector("button[type='submit']")).click();
  }
}

The catch block does not claim cleanup succeeded. That distinction matters. Swallowing quit() errors keeps test reports tidy while leaving the operator with no session identity. Failing every product test because cleanup hit a transient network error is also noisy. Log teardown as a separate infrastructure outcome and let a bounded reconciler handle confirmed leftovers.

Confirm an orphan before deleting anything

Start with the CI system, not the Grid. Find the job that created the session and establish its terminal state. “No recent log lines” is weak evidence. “Job 81342 was cancelled at 10:14:08 UTC and its worker pod no longer exists” is much stronger. The test should emit its Grid session ID as soon as creation succeeds, ideally alongside a run ID, job ID, attempt number, and worker identity.

Next, query the session through Grid GraphQL. Session details include the session ID, start time, duration, Node ID, and Node URI. The following script lists current sessions and captures status in the same evidence directory. It performs no deletion.

#!/usr/bin/env bash
set -euo pipefail

GRID_URL="$SELENIUM_GRID_URL"
EVIDENCE_DIR="artifacts/orphan-check"
mkdir -p "$EVIDENCE_DIR"

curl --fail --silent --show-error \
  -H 'Content-Type: application/json' \
  --data '{"query":"{ sessionsInfo { sessions { id startTime sessionDurationMillis nodeId nodeUri capabilities } } }"}' \
  "$GRID_URL/graphql" |
  tee "$EVIDENCE_DIR/sessions.json"

curl --fail --silent --show-error "$GRID_URL/status" \
  > "$EVIDENCE_DIR/grid-status.json"

Locate the exact session, then check its Node. The documented owner endpoint returns whether that Node owns the session. It requires the Grid registration secret when one is configured. Use the Node URI returned by GraphQL, not a guessed port or a stale host list.

#!/usr/bin/env bash
set -euo pipefail

: "$SESSION_ID"
: "$NODE_URL"
: "$REGISTRATION_SECRET"
: "$SELENIUM_GRID_URL"

printf 'Checking Node ownership for %s\n' "$SESSION_ID"
curl --fail --silent --show-error \
  -H "X-REGISTRATION-SECRET: $REGISTRATION_SECRET" \
  "$NODE_URL/se/grid/node/owner/$SESSION_ID"
printf '\n'

read -r -p "Type the exact session ID to delete it: " confirmation
if [ "$confirmation" != "$SESSION_ID" ]; then
  echo "Confirmation did not match; no request sent." >&2
  exit 2
fi

curl --fail --silent --show-error \
  --request DELETE \
  "$SELENIUM_GRID_URL/session/$SESSION_ID"
printf 'Delete-session request completed for %s\n' "$SESSION_ID"

That script deliberately requires a human confirmation. Automation can replace the prompt only when it has a trustworthy ownership source, such as a CI API that proves the originating job is terminal. A time threshold is a useful filter for candidates, never the final authorization to delete.

Look at recent commands or trace data when available. A session with repeated commands is not idle even if its start time is old. A session with no commands since its runner disappeared is a strong candidate. Grid event logs and distributed traces can show a clean quit, timeout, Node removal, or failed route. Preserve those records before cleanup because the successful delete removes some of the easiest state to inspect.

Node status separates another case. If the Node is up, owns the session, and accepts commands, a targeted delete through the Grid is appropriate after client death is proven. If the Node is down or unreachable, a Router delete may fail because it cannot reach the owner. Do not keep firing deletion requests and calling them cleanup. Repair, restart, or retire the Node, then verify that the Distributor and Session Map converge.

A concise incident record should make the decision auditable:

session_id: "6f1cfa98c8ec4d43a84ca9f22b943a7d"
created_at: "2026-08-04T10:03:41Z"
last_known_command_at: "2026-08-04T10:13:55Z"
ci:
  run_id: "81342"
  attempt: 1
  state: "cancelled"
  worker_exists: false
grid:
  node_id: "1f5825df-2e3d-4ca7-9b75-a7f590eac312"
  node_availability: "UP"
  node_reports_owner: true
decision:
  action: "delete_session"
  approved_at: "2026-08-04T10:18:22Z"

Do not copy these sample values into a production tool. The useful part is the set of facts and their timing. A reconciler should build the same record from authoritative systems and keep it with the incident.

Read the diagnostic fields as one joined record

The GraphQL session object answers location and identity questions, not abandonment. Start with id and require an exact match with the value emitted by the runner. A timestamp-adjacent match is unsafe when retries or parallel tests use the same browser capabilities. Then read nodeId and nodeUri together. They identify the Node that Grid associates with the session, but a plausible URI does not prove the Node is reachable or that the browser can still execute commands.

startTime and sessionDurationMillis are easy to misuse. A healthy, busy session and an abandoned session can both have a large duration. The duration describes how long the session has existed, not how long it has been idle and not when the last WebDriver command completed. Treat it as context for a timeline. Do not convert it into a deletion threshold.

Join that object to the current status snapshot by Node identity, then use the owner endpoint for the exact session check. In a healthy active case, the Node is available, a slot is occupied, the owner check agrees, and the CI job is still running. In the usual orphan case, those Grid-side values may look identical. The separating facts come from outside Grid: the recorded job is terminal, its worker no longer exists, and no later command appears after the worker disappeared.

A different pattern points to stale control-plane state. GraphQL still returns the session, but the named Node does not report ownership or its current slot data has no matching session. Do not translate that mismatch into “the browser is definitely gone.” It proves only that the views disagree. Capture both responses close together, repeat the read to rule out normal convergence, and inspect the Node or its host before choosing a lifecycle action. One old JSON file beside one fresh response is a misleading comparison, even when both files are individually valid.

Work through the failure modes that look alike

The common case begins with cancellation. A test creates a session at 10:03, CI cancels its worker at 10:14, and no delete-session request appears. The Grid still shows a healthy Node and one occupied slot. The CI job is terminal, the worker is gone, the Node owner endpoint returns true, and the session has no later commands. A targeted delete is safe. The preventive fix is framework teardown plus a Node timeout sized to the largest legitimate command gap.

The cost of a shorter timeout is false cleanup. Consider a visual test that tells the browser to start a server-side export, then polls the export API from test code for eight minutes without touching WebDriver. From the Node's perspective, the browser session is inactive. A five-minute session-timeout ends it even though the runner is alive. The next browser command fails because the session no longer exists.

That near-miss has different evidence from an orphan. CI remains running, the worker exists, application polling continues in its logs, and the session ends at the configured inactivity boundary. Raising the timeout or issuing meaningless browser commands are not equally good fixes. Prefer restructuring the test so it does not reserve a browser while doing long API-only work. If the browser must stay, set the timeout above the observed legitimate gap and accept slower recovery from real client death.

Another case begins with a Node network partition. The runner cannot route commands, the Distributor eventually marks or removes the Node from its model, and the browser process may continue on the isolated machine. From the Grid's usable-capacity perspective, the slot is already gone. Deleting the session through the Router cannot kill a process it cannot reach. The remediation belongs at the host or container boundary: restore connectivity, terminate the isolated workload, or replace the Node. Confirm it does not re-register with stale work before returning it to service.

The evidence is Node-wide rather than session-specific. Heartbeats stop, Node status is unreachable, multiple sessions on that Node fail together, and unrelated Nodes remain healthy. A single orphan cleanup request cannot repair that pattern. Treat it as Node health and account for the lost capacity while recovery runs.

A third look-alike comes from a slow command. The runner is alive, the Node is up, and a WebDriver command is still in flight. An external janitor sees no completed command for several minutes and assumes inactivity. Deleting the session can turn one slow page load into a misleading invalid session id error. Track command start and completion where tracing supports it. “No completed command” and “no in-flight command” are different statements.

Interactive debugging creates a policy exception. An engineer may intentionally leave a remote browser idle while inspecting a failure. Production CI pools should not carry indefinite debugging sessions because they have no automatic ownership boundary. Use a separate pool with a longer timeout, a visible owner, and an expiry. Do not weaken the main pool's recovery just to support occasional manual work.

Finally, a stale dashboard tab can impersonate an orphan. The UI may show a snapshot that is older than the session deletion, or a reverse proxy may cache a response it should not. Re-query /status and GraphQL directly with a current timestamp. If the APIs no longer list the session and the slot is free, there is nothing to delete. Refreshing evidence is cheaper than restarting Grid components.

Separate a missing quit from a stalled remote teardown

Two cleanup defects can produce almost the same runner log. In both, teardown prints the session ID, reports that quit() failed, and the job ends while the slot remains occupied. The first root cause is that the delete-session request never reached Grid. Cancellation closed the process or its network path before the command crossed the client boundary. The second root cause is remote: Grid received the command, but browser or driver termination did not complete and the Node kept the session.

The boundary evidence decides which defect you have. Search Router request records or a distributed trace for a delete-session operation carrying the exact session ID, then follow that operation to the owning Node. No matching ingress evidence, combined with a dead worker, points to runner shutdown, client networking, or CI cancellation order. A matching operation at Grid ingress and a corresponding Node-side teardown attempt moves ownership away from the fixture. The platform team then needs the Node logs and surviving process state, not another finally block in test code.

Do not use the text of the client exception as the separator. A connection failure can mean the request never arrived, or that the response path failed after remote work began. Likewise, the continued dashboard row does not reveal whether teardown was never requested or started and stalled. Preserve both sides of the request boundary before retrying cleanup. A retry may release the slot, but it also removes the evidence needed to assign the permanent fix.

Prevent leaks at the client, Node, and platform layers

Client cleanup should be boring and universal. Put driver creation and teardown in one framework-owned fixture or extension. Do not scatter new RemoteWebDriver across tests. Emit the session ID immediately, and associate it with the CI run before navigating to the application. If setup fails after the session is created, the same fixture must still quit it.

Cancellation behavior deserves a real test. Start a disposable CI job, create a remote session, then cancel the job through the same mechanism developers use. Observe whether the runner receives a graceful termination window and whether framework teardown runs. Many teams test assertion failures but never test job cancellation, even though cancellation is the path most likely to skip cleanup.

Grace periods help with SIGTERM, but they do not solve machine loss or SIGKILL. A shutdown hook may attempt a quit during graceful termination, yet it cannot be your only control. Keep the Node timeout as an independent lease on inactive sessions.

Set that lease from data. Extract the longest gaps between WebDriver commands for representative suites. Separate genuine browser interactions from long API calls, sleeps, human debugging, and application batch jobs. Choose a timeout above the legitimate percentile plus margin, then calculate the worst-case orphan cost: slot count multiplied by the timeout. A thirty-minute timeout on a four-slot pool can lose two slot-hours after all four runners die together.

Selenium accepts the timeout as a Node option in seconds. Verify the options against the server artifact you deploy by asking that component for configuration help. This avoids copying a flag from a different Grid version.

#!/usr/bin/env bash
set -euo pipefail

SELENIUM_JAR="$1"
HUB_HOST="$2"

java -jar "$SELENIUM_JAR" node --help | sed -n '/session-timeout/,+3p'

exec java -jar "$SELENIUM_JAR" node \
  --hub "http://$HUB_HOST:4444" \
  --session-timeout 420 \
  --max-sessions 2

A seven-minute value is only an example. A suite with ten-minute legitimate command gaps needs a different value or a redesigned flow. Roll the change to one Node pool first. Watch unexpected session terminations, test error types, recovered slot time, and overall queue duration. Then expand browser by browser.

Disposable Nodes reduce the residue left by browser and driver processes. The documented --drain-after-session-count option can make a Node drain and shut down after a configured number of sessions. One session per Node gives strong isolation, but it trades density for startup overhead. It also moves reliability requirements to the scheduler that replaces Nodes.

Do not force-delete old records directly from a Session Map datastore. The Router, Node, Distributor, and Session Map each own part of the lifecycle. Bypassing Grid's documented endpoints can leave the browser alive, free the wrong mapping, or create a state that later registration cannot reconcile. Use the delete-session command for an owned, reachable session. Handle unreachable Nodes through their platform lifecycle.

A production reconciler should default to report-only. During the first rollout, have it list candidates with session ID, Node URI, age, last activity, CI ownership, and proposed action. Compare its decisions with humans for several weeks. Only then enable deletion for the narrow state where the owning job is conclusively terminal and the Node still reports ownership.

Protect the reconciler itself. Limit which Grid it can reach, require authentication where supported, keep the registration secret outside logs, rate-limit deletes, and write an immutable audit event. An accidental loop that deletes every session is worse than the leak it was meant to fix.

Roll the controls into an existing suite in dependency order

Begin with an instrumentation-only release. Record the session ID, CI run and attempt, worker identity, fixture scope, and teardown outcome without changing when sessions close. This establishes which fixture actually owns each driver. It also exposes tests that construct a second driver outside the shared fixture, a common reason a seemingly correct teardown closes one session while another survives. If correlation is introduced after timeout changes, the first invalid-session failures will lack the data needed to tell false cleanup from a client leak.

Next, move cleanup into the owner at its existing scope. A class-scoped driver should initially close at class teardown, and a suite-scoped driver at suite teardown. Changing everything to per-test sessions in the same patch adds a separate behavioral and performance migration. Tests that retain a driver reference past the declared scope, depend on browser state from an earlier test, or replace the fixture's driver without registering it will break first. Those failures are useful because they reveal hidden lifecycle ownership, but they should not be confused with Grid instability.

Land cancellation handling only after ordinary assertion and setup failures produce correlated close outcomes. Then exercise graceful cancellation and abrupt worker loss separately. The former tests the runner's shutdown path. The latter proves the Node lease without pretending client code can run after process death. Keep the destructive scenario in a disposable pool so its expected orphan cannot consume production capacity.

Only after the suite's legitimate command gaps are visible should one Node pool receive the timeout policy. Route a representative slice there and compare it with an unchanged pool. The useful signals are the elapsed point at which invalid-session errors appear, how long confirmed dead-runner sessions retain slots, and whether the next compatible request starts after release. A lower queue alone is not success if valid long-idle tests are now being killed.

Enable the reconciler last. Its report-only candidates should contain enough evidence for the same decision a human would make. Automatic deletion can then be restricted to the state the rollout has already validated. This order costs calendar time and temporarily leaves manual cleanup in place, but it prevents a fixture change, a timeout change, and a destructive controller from obscuring one another's failures. Keeping broader driver scopes also preserves existing test behavior, at the specific cost of slower isolation when a test contaminates shared browser state.

Verify the fix without creating another leak

Use three controlled scenarios. First, run a test that fails an assertion. The session should close immediately through normal teardown. Second, terminate a runner gracefully and verify whether the cancellation grace period lets teardown complete. Third, kill a disposable runner without cleanup and measure how long the Node timeout takes to release the slot.

For each case, capture the CI state, client session ID, Grid status, GraphQL session list, Node logs, and the next request's wait time. The proof is not merely that a row disappears. The original browser must stop, the slot must become eligible, and a new compatible session must start successfully.

Expected outcomes can be written as an executable check. This Python script polls GraphQL until a known session disappears, then exits nonzero if cleanup exceeds the agreed recovery window. It does not delete the session, so it can verify either explicit teardown or timeout-based cleanup.

#!/usr/bin/env python3
import os
import sys
import time
import requests

grid_url = os.environ["SELENIUM_GRID_URL"].rstrip("/")
session_id = os.environ["SESSION_ID"]
deadline = time.monotonic() + int(os.environ.get("RECOVERY_SECONDS", "480"))
query = "{ sessionsInfo { sessions { id } } }"

while time.monotonic() < deadline:
    response = requests.post(
        f"{grid_url}/graphql",
        json={"query": query},
        timeout=10,
    )
    response.raise_for_status()
    sessions = response.json()["data"]["sessionsInfo"]["sessions"]
    active_ids = {session["id"] for session in sessions}
    if session_id not in active_ids:
        print(f"session_released={session_id}")
        sys.exit(0)
    time.sleep(5)

print(f"session_still_active={session_id}", file=sys.stderr)
sys.exit(1)

Run this check in a disposable environment because one scenario intentionally abandons a session. The test must know the configured timeout and allow for the Node's cleanup interval without granting an unbounded wait.

Add a fourth scenario for a temporary client network break. Block traffic between one runner and the Router for less than the configured session timeout, then restore it. If the next WebDriver command succeeds, the lease tolerated a recoverable interruption. Repeat with a break longer than the timeout and confirm that the client receives a clear session-loss error while the slot is released. This pair establishes the operational boundary instead of assuming every disconnect means abandonment.

Measure recovery from the moment the last valid command reached the Node, not from when an engineer noticed the dashboard. Also record how long a compatible request waited after release. A session can disappear on schedule while the replacement test still queues because the Node or browser process did not recover cleanly. Slot reuse is the end-to-end proof.

Watch for error migration after rollout. If queue time falls but tests begin failing with invalid-session errors at nearly the same elapsed time, the timeout is reclaiming valid sessions. If sessions disappear from Grid but browser containers remain, platform cleanup is incomplete. If the Node drops out entirely, the change may be triggering process termination rather than session cleanup.

Know when forced cleanup is the wrong action

Do not delete a session because it is old. End-to-end tests, soak tests, and debugging sessions can be legitimately long-lived. Age can trigger investigation, but ownership and activity decide the action.

Do not use orphan cleanup to handle a saturated but healthy pool. If every occupied session belongs to a running job and continues receiving commands, the Grid is doing useful work. Capacity, suite duration, or admission control is the correct conversation.

Do not shorten --session-timeout to compensate for tests that forget quit() on ordinary failures. That hides a client lifecycle defect and makes every long command gap risky. Fix the fixture first, then keep the timeout for failures the client cannot handle.

Do not delete through a Node when the same request through the Grid Router works. The Grid-level delete follows the session mapping and gives one stable entry point. Direct Node operations belong in diagnosis and controlled administration, particularly when proving ownership.

Do not clear the New Session Queue as a session cleanup technique. Queued requests do not own browser sessions yet. Clearing that queue rejects waiting clients and leaves occupied slots untouched. It is a separate emergency action with a much wider blast radius.

Do not restart the entire Grid to recover one identified session. A restart can interrupt unrelated tests, obscure the original cause, and leave browser processes behind depending on how Nodes are managed. Prefer the narrowest documented lifecycle action that matches the evidence.

Assign ownership at the evidence boundary

The suite team owns driver construction, fixture scope, and proof that ordinary teardown attempted the exact session. The CI platform team owns cancellation order, worker termination grace, and the authoritative record that a run and worker are gone. The Grid team owns routing, session records, Node configuration, and interpretation of Grid traces. The compute or container team owns a browser process that survives after the Node can no longer control it.

A handoff should include the session ID, CI run and attempt, worker identity and terminal timestamp, Node ID and URI, close outcome from the client, Grid ingress evidence for the delete request, owner-check result, relevant status and GraphQL captures, and whether the browser process still exists. It should also state which cleanup action has already been attempted. Without that package, each team can truthfully show that its own component looks healthy while the request boundary containing the defect remains unexamined.

Session reconciliation does not catch an untracked browser-process leak. If Grid has removed the session and freed the slot but a browser or driver process remains on the host, there is no session ID left for this technique to find or delete. Host-level process and container lifecycle controls must detect that failure. Broadening a session janitor's deletion rules cannot repair it.

Every prevention layer has a cost. Strict fixtures add framework code and teardown reporting. Short leases recover slots faster but can kill intentional idle periods. Disposable Nodes isolate residue but increase startup latency and compute churn. A reconciler improves recovery but holds destructive authority. Choose the smallest combination that covers your actual failure paths, and keep the evidence needed to prove it did.