The zero-session gate that makes Selenium Node replacement safe

An autoscaler selects a Selenium Node for replacement because its CPU is low. The Node's health probe is green, so automation terminates it. One browser was sitting idle on a payment confirmation page, and “idle CPU” said nothing about whether that WebDriver session was finished.

Define a termination predicate instead of trusting one probe

A Node can be healthy, busy, draining, empty, unreachable, or gone. Those states answer different questions. The word “ready” becomes dangerous when a maintenance script uses it without saying ready for what.

Kubernetes readiness commonly asks whether a Pod can receive service traffic. A process check asks whether Java is running. Selenium's Node status tells the Distributor whether the Node is UP, DRAINING, or DOWN. Those three values are the entire Status enum in the Grid GraphQL schema, so a gate that filters on a friendlier-sounding string such as UNAVAILABLE matches nothing and silently treats every unhealthy Node as fine. GraphQL shows registered Nodes and active sessions. None of those observations alone proves that a machine may be destroyed.

Use a conjunction for safe termination:

1. The target is identified by Selenium's immutable Node ID, not only hostname or Pod name.
2. A drain request for that ID has succeeded.
3. Grid has observed the Node in DRAINING state, proving new sessions should not be assigned.
4. The same Node reports zero active sessions.
5. No monitoring or GraphQL error has been converted into an empty result.
6. The observation is recent enough for the maintenance action.
7. Termination targets the same process, Pod, or host whose Node ID was drained.

The order closes a real race. If automation observes sessionCount zero while the Node is UP, the Distributor can allocate another session one millisecond later. Draining first closes admission. Zero then becomes stable for the remaining life of that Node.

Node ID matters because addresses are reusable. A Pod can restart at the same DNS name with a new Grid identity. Automation that drains old ID A and later sees zero sessions at URI node-5 may actually be looking at replacement ID B. Killing by URI would terminate the replacement while claiming evidence from the old process.

Selenium's documented Distributor endpoint is:

POST /se/grid/distributor/node/{node-id}/drain

The Distributor passes the operation to the chosen Node. The direct Node endpoint is:

POST /se/grid/node/drain

Both require the registration-secret header when Grid is configured with one. Prefer the Distributor route when an external maintenance controller knows the Node ID and reaches the Router. Prefer the direct route when a local lifecycle hook can only reach its own Node. Keep the registration secret out of logs.

The drain operation is session-aware. It stops accepting new session requests, finishes current sessions, then stops the Node. It does not decide how long a hung test may block maintenance. Your controller needs a deadline and a policy for what happens afterward.

A conservative controller should have at least four outcomes:

• READY: DRAINING was observed and session count reached zero;
• WAITING: active sessions remain before the deadline;
• TIMEOUT: sessions remain at the deadline, so termination is not authorized;
• UNKNOWN: GraphQL failed, the Node disappeared too early, identity changed, or the response was invalid.

Only READY authorizes termination. Treating UNKNOWN as empty is the most common dangerous shortcut in homegrown scripts.

Read live Grid state by Node ID

The GraphQL schema exposes nodesInfo.nodes with id, URI, status, sessionCount, stereotypes, and session details. Query only fields needed by the gate, but include session IDs in diagnostic snapshots. An operator must be able to tell which test is blocking a drain.

set -euo pipefail

GRID_URL="$1"
GRID_NODE_ID="$2"

body='{"query":"{ nodesInfo { nodes { id uri status sessionCount sessions { id startTime } } } }"}'
snapshot="$(curl --fail-with-body --silent --show-error \
  -H 'Content-Type: application/json' \
  --data "$body" \
  "$GRID_URL/graphql")"

jq --arg id "$GRID_NODE_ID" \
  '.data.nodesInfo.nodes[] | select(.id == $id)' \
  <<< "$snapshot"

A busy Node before drain may produce a record shaped like this:

{
  "id": "7a2d1dc2-7f91-4e80-b491-6c9cf9eb8d15",
  "uri": "http://selenium-node-chrome-4:5555",
  "status": "UP",
  "sessionCount": 2,
  "sessions": [
    {"id": "18f71b...", "startTime": "2026-08-04T09:42:11Z"},
    {"id": "a9902c...", "startTime": "2026-08-04T09:51:03Z"}
  ]
}

After a successful drain request, status should become DRAINING while the sessions remain. Existing commands should continue reaching them. A new-session request should land elsewhere or wait in the queue. Once both sessions call quit, the count reaches zero and the Node may stop quickly.

GraphQL can return HTTP 200 with an errors array. Check it. An invalid query or schema mismatch must become UNKNOWN, not an empty nodes list. Likewise, a reverse proxy can return an HTML login page or a cached response. Validate content and timestamp observations at the caller.

The Node may disappear from GraphQL before a polling controller captures a zero sample. Graceful completion can produce that sequence because Selenium stops a drained Node after the last session. A crash can produce the same sequence. If session preservation is the goal, absence after a last-known count greater than zero is not sufficient evidence.

There are ways to add a trusted completion signal. A local wrapper can capture the Selenium process exit code and a drain-complete log. A platform controller can correlate Node DRAINING, the final session-removal events, and a clean container termination. A sidecar can own the drain protocol and publish a completion condition. Each adds complexity. The conservative alternative is to reject ambiguous disappearance and let an operator inspect it.

Do not use queue size as the Node's owned-session count. The New Session Queue contains requests not yet assigned to any Node. A drained Node can have zero sessions while the queue is large. That may make maintenance a bad capacity decision, but it does not mean the target still owns a browser.

Slots require care too. A Node can advertise more stereotypes or slot records than it may run concurrently. Count sessions, and retain session IDs. Do not subtract free slots from total slots and call the result ownership without validating the deployed schema and Grid behavior.

Implement a conservative drain gate

The next Python program uses only the standard library. It queries the exact Node, posts the documented Distributor drain endpoint, waits for DRAINING, and returns success only after seeing sessionCount zero on that same ID. It uses separate exit codes for timeout and unknown state.

#!/usr/bin/env python3
import argparse
import json
import sys
import time
import urllib.error
import urllib.parse
import urllib.request

QUERY = """
{
  nodesInfo {
    nodes {
      id
      uri
      status
      sessionCount
      sessions {
        id
        startTime
      }
    }
  }
}
"""

class UnknownState(RuntimeError):
    pass

def graphql(grid_url: str, timeout: float) -> dict:
    request = urllib.request.Request(
        grid_url.rstrip("/") + "/graphql",
        data=json.dumps({"query": QUERY}).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        if response.status != 200:
            raise UnknownState(f"GraphQL HTTP {response.status}")
        payload = json.load(response)

    if payload.get("errors"):
        raise UnknownState("GraphQL errors: " + json.dumps(payload["errors"]))

    try:
        nodes = payload["data"]["nodesInfo"]["nodes"]
    except (KeyError, TypeError) as error:
        raise UnknownState("GraphQL nodes payload is missing") from error
    if not isinstance(nodes, list):
        raise UnknownState("GraphQL nodes payload is not a list")
    return payload

def find_node(payload: dict, node_id: str) -> dict | None:
    nodes = payload["data"]["nodesInfo"]["nodes"]
    matches = [node for node in nodes if node.get("id") == node_id]
    if len(matches) > 1:
        raise UnknownState(f"duplicate Node ID {node_id}")
    return matches[0] if matches else None

def request_drain(
    grid_url: str,
    node_id: str,
    registration_secret: str,
    timeout: float,
) -> None:
    encoded_id = urllib.parse.quote(node_id, safe="")
    url = (
        grid_url.rstrip("/")
        + "/se/grid/distributor/node/"
        + encoded_id
        + "/drain"
    )
    request = urllib.request.Request(
        url,
        data=b"",
        headers={"X-REGISTRATION-SECRET": registration_secret},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        if response.status < 200 or response.status >= 300:
            raise UnknownState(f"drain HTTP {response.status}")

def run(args: argparse.Namespace) -> int:
    initial = find_node(
        graphql(args.grid_url, args.http_timeout), args.node_id)
    if initial is None:
        raise UnknownState("target Node is not registered before drain")

    print(json.dumps({"phase": "before", "node": initial}))
    request_drain(
        args.grid_url,
        args.node_id,
        args.registration_secret,
        args.http_timeout,
    )

    deadline = time.monotonic() + args.wait_seconds
    saw_draining = False
    last_node = initial

    while time.monotonic() < deadline:
        payload = graphql(args.grid_url, args.http_timeout)
        node = find_node(payload, args.node_id)

        if node is None:
            raise UnknownState(
                "Node disappeared before a zero-session sample; "
                + "last observation was "
                + json.dumps(last_node)
            )

        status = node.get("status")
        session_count = node.get("sessionCount")
        if not isinstance(session_count, int):
            raise UnknownState("sessionCount is not an integer")

        last_node = node
        saw_draining = saw_draining or status == "DRAINING"
        print(json.dumps({
            "phase": "waiting",
            "status": status,
            "sessionCount": session_count,
            "sessions": node.get("sessions", []),
        }))

        if saw_draining and status == "DRAINING" and session_count == 0:
            print(json.dumps({
                "result": "READY",
                "nodeId": args.node_id,
            }))
            return 0

        time.sleep(args.poll_seconds)

    print(json.dumps({
        "result": "TIMEOUT",
        "nodeId": args.node_id,
        "lastNode": last_node,
    }), file=sys.stderr)
    return 1

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--grid-url", required=True)
    parser.add_argument("--node-id", required=True)
    parser.add_argument("--registration-secret", required=True)
    parser.add_argument("--wait-seconds", type=int, default=1800)
    parser.add_argument("--poll-seconds", type=float, default=2.0)
    parser.add_argument("--http-timeout", type=float, default=5.0)
    args = parser.parse_args()

    try:
        return run(args)
    except (
        UnknownState,
        urllib.error.URLError,
        json.JSONDecodeError,
    ) as error:
        print(json.dumps({
            "result": "UNKNOWN",
            "nodeId": args.node_id,
            "error": str(error),
        }), file=sys.stderr)
        return 2

if __name__ == "__main__":
    raise SystemExit(main())

The strict zero observation can create a false negative when a correctly drained Node stops between polls. That is intentional. Safety gates should prefer a delayed replacement over destroying an unproven active session. If the extra delay is unacceptable, add a reliable lifecycle-completion signal rather than turning absence into success.

There is another timing edge. An empty Node may transition from UP to stopped so quickly that DRAINING is never sampled. Increase polling frequency only within reasonable load. A better design records the drain acknowledgement or emits an event from the local Node wrapper. Polling can reduce uncertainty, not eliminate it.

Wire the gate so nonzero exits stop the terminating action. This local-host wrapper assumes SELENIUM_PID is the exact Java process previously associated with the Node ID. Selenium drain may already have stopped it; in that case no signal is necessary.

#!/usr/bin/env bash
set -euo pipefail

GRID_URL="$1"
GRID_NODE_ID="$2"
REGISTRATION_SECRET="$3"
SELENIUM_PID="$4"

if python3 drain_gate.py \
  --grid-url "$GRID_URL" \
  --node-id "$GRID_NODE_ID" \
  --registration-secret "$REGISTRATION_SECRET" \
  --wait-seconds 1800 \
  > "drain-$GRID_NODE_ID.jsonl"; then
  if kill -0 "$SELENIUM_PID" 2>/dev/null; then
    kill -TERM "$SELENIUM_PID"
  fi
else
  status="$?"
  printf 'Termination blocked for Node %s, gate exit %s\n' \
    "$GRID_NODE_ID" "$status" >&2
  exit "$status"
fi

A Kubernetes or VM controller should apply the same exit-code contract but terminate through its platform's normal disruption mechanism. Also preserve availability controls such as PodDisruptionBudgets and replacement limits. Session safety says this Node owns no browser; it does not say the fleet can afford to lose its capacity now.

Exercise busy, leaked, and replaced Nodes

The happy-path test needs an active browser long enough to observe. This Java program opens a session, prints the ID, waits, executes a command after drain has begun, then quits. It proves that DRAINING does not kill existing work and that the session can complete normally.

import java.net.URI;
import java.time.Duration;
import org.openqa.selenium.By;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

public final class DrainSessionProbe {
  public static void main(String[] args) throws Exception {
    URI grid = URI.create(System.getenv("GRID_URL"));
    ChromeOptions options = new ChromeOptions();
    options.addArguments("--headless=new");

    RemoteWebDriver driver =
        new RemoteWebDriver(grid.toURL(), options);
    try {
      driver.get("https://www.selenium.dev/selenium/web/web-form.html");
      System.out.println("SESSION_ID=" + driver.getSessionId());
      System.out.println("Start drain against the owning Node now");

      Thread.sleep(Duration.ofSeconds(90).toMillis());

      driver.navigate().refresh();
      String heading = driver.findElement(By.tagName("h1")).getText();
      if (!"Web form".equals(heading)) {
        throw new AssertionError("Unexpected heading: " + heading);
      }
      System.out.println("EXISTING_SESSION_AFTER_DRAIN=passed");
    } finally {
      driver.quit();
      System.out.println("SESSION_QUIT=true");
    }
  }
}

Locate the owner with the documented session query before starting the gate:

set -euo pipefail

GRID_URL="$1"
SESSION_ID="$2"

query="$(printf '{ session(id: "%s") { id nodeId nodeUri sessionDurationMillis } }' "$SESSION_ID")"
curl --fail-with-body --silent --show-error \
  -H 'Content-Type: application/json' \
  --data "$(jq -nc --arg query "$query" '{query:$query}')" \
  "$GRID_URL/graphql" | jq

For the busy-Node case, expected evidence is:

• initial Node status UP with sessionCount one;
• drain request succeeds;
• status becomes DRAINING and count remains one;
• the probe's refresh succeeds;
• quit removes the session;
• the gate sees DRAINING with count zero and returns READY.

Also start a second probe after DRAINING appears. It may run on another compatible Node or wait. Query its nodeId. It must not equal the draining target. This catches a drain request that returned success but did not update Distributor admission.

The leaked-session case changes only cleanup. Simulate a test process that stops sending commands without calling quit, while the Node's session timeout is long enough to keep ownership. The gate must remain WAITING and then TIMEOUT. The diagnostic JSON includes the blocking session ID and start time. An operator can find the CI job and decide whether to terminate that session.

Do not make timeout automatically call Selenium's delete-session endpoint unless policy explicitly says maintenance outranks test preservation. Forced cleanup destroys the evidence the gate was built to protect. If forced termination is authorized after a threshold, record the session ID, owner, duration, approver or policy, deletion response, and maintenance action as a different outcome such as FORCED, never READY.

The identity-replacement case catches URI reuse. Start Node A, record its ID and URI, then replace it so Node B registers at the same service hostname. Run the gate with A's ID. It should return UNKNOWN because A is absent before drain. It must not find B by URI and terminate it. Then run a fresh gate using B's ID.

A fourth test should break GraphQL while the Node is active. Return an HTTP error, invalid JSON, or a GraphQL errors array from a test proxy. The gate must exit 2 and block termination. This verifies fail-closed monitoring rather than browser behavior.

Distinguish drain problems from nearby failures

If the drain POST returns 404, first verify Node ID and URL mode. Standalone, Hub-Node, and fully distributed deployments use the Grid entry point differently, but the documented Distributor path remains routed through the appropriate front door. A stale ID after Pod restart is more likely than a browser issue.

A 401 or 403 response points to registration-secret or proxy policy, not session state. Confirm the header reaches Grid without logging its value. Do not retry with an empty header or disable authentication during maintenance.

If the Node stays UP after a successful-looking request, inspect Distributor and Node logs for the same ID. The request might have hit a different environment, a proxy may have cached an invalid response, or the Node may be unreachable from the Distributor. Verify live GraphQL URI and Grid version.

If status is DRAINING and sessionCount never falls, the drain mechanism is working. One or more clients have not ended their sessions. Query session IDs, duration, and capabilities. Correlate them with client logs and CI jobs. A browser may be running a legitimate soak test, or its owner may have crashed without quit.

If sessionCount falls but queue size rises, the Node is being removed safely while compatible fleet capacity is shrinking. That is a scheduling and maintenance-rate issue. Pause further drains, wait for replacement Nodes to register, or throttle new CI work. Do not reverse the target Node to UP midway unless the deployed Selenium lifecycle explicitly supports and tests that action.

If existing commands fail immediately after DRAINING, check whether automation terminated the Pod too soon. Selenium drain is meant to preserve ongoing sessions. A short platform termination grace period, a preStop hook that returns early, or a second controller deleting the Pod can bypass the wait.

A Node can report zero while the browser's test process is about to create a session but has not reached Grid yet. Draining first means that request cannot land on the target, so it is not a safety problem for this Node. It can still affect fleet capacity and queue delay. This is why the predicate separates local ownership from global service impact.

Finally, do not confuse a WebDriver session with a test case. One session may span many tests, and one test may create multiple sessions. The termination gate protects every session owned by the Node. Test reports alone are insufficient unless they log and retain all session IDs.

Separate an abandoned session from a command that is still running

The most deceptive drain incident has the same Grid snapshot as a leaked session but a different root cause. The target remains DRAINING, sessionCount stays at one, and repeated queries show the same session ID. In one case the test runner has exited without calling quit. In the other, the runner is alive and waiting for a WebDriver command that has not returned. The gate is correct to block termination in both cases, but the recovery action is not the same. Cleaning up an abandoned session may be authorized by policy. Deleting a session with an active command can interrupt valid work and can leave the tested system in an uncertain state.

Read the diagnostic record in layers. The Node id proves which Grid identity owns the work. The status field should move from UP to DRAINING after admission closes. The sessionCount field should be a nonnegative integer, and its healthy drain sequence is a stable or falling value followed by zero. A broken sequence is the same positive value through the entire deadline. In sessions, the id is the join key for client, Grid, and CI evidence. startTime tells when ownership began, not when the session last performed useful work. A session that started two hours ago may be an active soak test, while a five-minute-old session may already be abandoned. Treat age as a routing clue, never as proof of idleness.

The gate's JSONL output is deliberately unable to decide between these two cases. A healthy line can show status DRAINING, sessionCount one, and one session ID, then show the same Node with count zero after teardown. Both failure modes show the first line indefinitely. The misleading value is the unchanged count. It proves continuing ownership, but it does not prove that the browser, client process, or application request is idle. Likewise, an empty sessions array beside a positive count is inconsistent evidence, not permission to terminate. Preserve that response as UNKNOWN and investigate the producer or parser.

Join the session ID to the test runner's lifecycle record. For an active command, the runner should still have a live job and a command-start record without the corresponding command-finish record. A Grid trace or Node log for the same session can show that the command reached the remote end, while application or proxy logs can show whether the tested request is still progressing. For an abandoned session, the decisive evidence is outside GraphQL: the CI job ended or the runner process exited, its final records contain no successful teardown, and no later command is associated with that session. Silence alone is weak evidence because log delivery can fail. A recorded process exit or cancellation event, correlated by job and session ID, is much stronger.

This distinction changes escalation. A live runner waiting on a command goes first to the test owner, who can identify the expected command duration and decide whether cancelling it is safe. If the application call is stuck, the tested-service owner needs the request correlation data and the time the command began. An abandoned session goes to the framework owner responsible for teardown and cancellation handling. The Grid platform owner should not infer either cause from session age. That team owns admission control, accurate snapshots, and enforcement of the maintenance outcome. The infrastructure owner acts only after the gate authorizes termination or an explicit force policy is invoked.

A useful handoff contains the exact Node ID and URI, session ID, drain request time, last several status and count samples, CI job and attempt identifier, runner exit or heartbeat evidence, the last command boundary recorded by the client, relevant Grid trace or Node log references, and the maintenance deadline. Include the policy decision needed from the recipient, such as whether a known idempotent test may be cancelled. Exclude the registration secret and redact sensitive capabilities. A screenshot that says “one session stuck” forces the next team to rediscover identity and timing, which wastes the remaining drain window.

Roll this evidence out before letting the gate stop fleet maintenance. First, land session-ID capture at driver creation and teardown in the shared test framework. Next, make CI cancellation and process-exit events searchable by the same job identifier. Then run the drain controller in observation mode and measure how many positive counts can actually be assigned to a living job. Fix the unowned remainder before making TIMEOUT block an upgrade. Legacy global drivers, teardown that runs only on successful tests, and suites that reuse one session across unrelated cases usually break attribution first. They can leave a valid Grid session with no single test owner even though the gate itself is behaving correctly.

After attribution is reliable, canary enforcement on a pool with short, independently owned sessions. Land the platform check that prevents deletion on TIMEOUT or UNKNOWN before enabling unattended drain requests. Only then include pools with long commands or shared external fixtures. The working signal is not merely more READY results. It is a falling rate of sessions with no owner, a stable rate of successful commands during DRAINING, and maintenance records in which every termination can be tied to a recent zero observation. If upgrade duration improves while unexplained session loss rises, the rollout is failing.

The extra evidence has a concrete cost. Client command-boundary logging increases event volume, correlation fields must survive across CI, Grid, and application retention systems, and conservative deadlines extend node replacement. Centralized polling also consumes Router capacity during large maintenance waves. Keep the client record focused on timestamps, command category, outcome, job ID, and session ID rather than logging command payloads, which may contain secrets. The maintenance team gains a defensible decision at the cost of more telemetry plumbing and slower replacement when ownership is ambiguous.

This technique does not detect a browser process that Grid has already removed from its session registry. If Grid reports zero while an orphaned browser child remains on the host, the session gate can still return READY because it protects registered WebDriver ownership, not arbitrary operating-system processes. Process-leak detection needs a separate Node or host control, and its findings should not be silently rewritten as a nonzero Grid session count.

Introduce the gate without freezing every upgrade

Begin in observation mode. Compute what the gate would decide but do not terminate anything. Compare Node IDs, statuses, counts, and maintenance outcomes for a week. Find how often Nodes disappear between DRAINING and zero, how long real sessions take, and how many clients leak sessions.

Instrument clients to print session IDs and call quit in finally or teardown logic. A gate can name a blocker only if reports can find its owner. Add CI job ID as ordinary test metadata only through supported mechanisms and without secrets; otherwise join by session ID and timestamp in your logging system.

Canary one browser pool with short tests. Set a generous deadline and require manual confirmation after READY. Then let automation act on READY while TIMEOUT and UNKNOWN remain manual. Expand to long-running and specialized pools after their session durations are measured.

Limit concurrent drains by compatible pool. Two empty Nodes may both be safe individually while removing both violates service capacity. Fleet coordination should wait for replacement Nodes to register with expected stereotypes before selecting the next target.

Store the complete JSONL evidence for each operation: target ID, URI, initial sessions, drain response time, every status transition, zero observation, gate result, and platform termination result. Redact credentials and potentially sensitive capabilities. Retention should cover infrastructure incident review.

Test upgrades against the deployed Selenium version. GraphQL fields and endpoints in the article are documented, but automation must still validate response shape. Run parser fixtures plus one live canary before changing all Nodes.

Choose deadlines by distribution, not guesswork. If 99 percent of sessions finish within twelve minutes but nightly accessibility runs take twenty-five, a ten-minute deadline guarantees routine escalation. You may schedule the long class separately or give it a different pool policy. One global timeout is simple and often wrong.

Accept conservative false negatives, and skip forced automation when needed

Polling adds load to the Router and delays maintenance by up to one poll interval after zero. Two-second polling is reasonable for a handful of drains, not for hundreds of controllers independently querying every Node. Centralize snapshots or stagger operations at fleet scale.

Failing UNKNOWN protects sessions but creates operator work. Clean Nodes that stop quickly may be flagged because zero was never observed. Add a trusted drain-completion signal when that cost becomes material. Do not weaken the predicate based on impatience.

Long deadlines preserve tests and extend patch windows. Short deadlines keep infrastructure moving and create more forced decisions. Make that trade visible to release and security owners. QA automation cannot choose the business priority alone.

Do not run this gate for a local disposable Grid where losing the only browser is acceptable and no automated host replacement exists. A manual quit and process stop are clearer.

Avoid automatic force for sessions that can change production-like state, perform payments, or manage scarce records. An interrupted command can have completed even when its response was lost. Starting a replacement session does not make replay safe.

Do not use the gate as a substitute for independent tests. Suites should own drivers, call quit, avoid shared sessions, and tolerate clean infrastructure scheduling. Drain automation handles platform maintenance, not poor test lifecycle.

The pattern is valuable when Nodes are replaced automatically and an active session is worth more than a few minutes of maintenance delay. Its strength comes from refusing to guess. Admission stopped, exact identity, observed zero, then terminate. Any other state remains work to investigate.