Every node says ready, every session times out: rehearsing a Selenium Grid event bus partition

At 09:12 on a release morning, a platform team applied a default-deny egress policy to the selenium namespace as part of a routine hardening pass. Nothing crashed. The Grid UI kept serving, curl http://grid.internal:4444/status kept returning HTTP 200, and every Node pod stayed Running with a green readiness probe. Over the next twenty minutes the regression suite went from 40 parallel sessions to zero, and the only error anyone could find in the test logs was a client-side timeout waiting for a new session.

That is the signature of an event bus partition, and it is worth learning to recognise because almost every other Grid symptom looks louder. A crashed Node throws connection refused. A bad capability throws SessionNotCreatedException with a readable message. A bus split throws nothing at all: it just stops the components from agreeing with each other, and lets your tests discover that fact through a timeout thirty seconds later.

What the bus is actually carrying

The Selenium documentation is unusually direct about this. The components page describes the Event Bus as a component that "Serves as a communication path between the Nodes, Distributor, New Session Queue, and Session Map." The architecture page adds that it is "Used for sending messages which may be received asynchronously between the other components."

Read that membership list carefully, because it is the whole article. Four things are bus clients, and the New Session Queue is one of them. It is not a passive buffer the Router pokes over HTTP; it is a first-class participant in the same asynchronous fabric as the Nodes, the Distributor and the Session Map. When the bus splits, it is not only node registration that stops. The path by which a queued request gets matched to a slot is degraded in the same stroke.

The Router is the exception, and this is why the failure is so quiet. The architecture page describes the Router as the component that "Acts as the front-end of the Grid. This is the only part of the Grid which may be exposed to the wider Web." Your monitoring almost certainly points at the Router, on 4444, over HTTP. That path is untouched by a bus partition. Your health check keeps passing while the Grid quietly loses the ability to place work.

Two flags define the bus wiring, and both are documented in the Grid CLI reference:

• --publish-events, a connection string for publishing events, default tcp://*:4442
• --subscribe-events, a connection string for subscribing to events, default tcp://*:4443

They are separate ports because they are separate sockets. That detail matters more than it looks, and we will come back to it in the second failure mode.

A third flag decides who owns the bus. --bind-bus is a boolean, default false, and the reference describes it precisely: "When true, the component will be bound to the Event Bus (as in the Event Bus will also be started by the component, typically by the Distributor and the Hub). When false, the component will connect to the Event Bus." So in the common Hub and Node topology the Hub binds and the Nodes connect. In fully distributed mode the getting-started guide shows a dedicated event-bus role on its own port, with the Distributor started using --bind-bus false so it connects rather than binds.

The default implementation is named in the reference too: --events-implementation defaults to org.openqa.selenium.events.zeromq.ZeroMqEventBus. That is useful context when you are reading stack traces, and it is a reminder that the bus is a messaging layer with its own liveness handling, not a request-response API you can curl.

The timers that decide how long the lie lasts

A partition is not instantly visible because several documented timers have to elapse first. These are the four that shape the shape of the outage:

Notice that node heartbeats travel over the bus, while the Distributor health check is described as pinging the Nodes. Those are two different transports carrying two different opinions about the same Node. During a bus partition they can disagree for a while, which is the mechanical reason your Grid can hold contradictory beliefs at the same time.

Two more Node flags decide what happens to a Node that comes up on the wrong side of the split:

• --register-cycle, default 10 seconds: "How often, in seconds, the Node will try to register itself for the first time to the Distributor."
• --register-period, default 120 seconds: "How long, in seconds, will the Node try to register to the Distributor for the first time. After this period is completed, the Node will not attempt to register again."

That second one is the sentence that ruins weekends. A Node that starts during a partition will retry for a bounded window and then stop, permanently, while the process stays alive and healthy from your orchestrator's point of view. There is a documented remedy: --register-shutdown-on-failure, a boolean defaulting to false, described as "If enabled, the Node will shut down after the register period is completed without a successful registration. Useful in container environments to trigger a restart." If you run Nodes in Kubernetes and you have not set that flag, your drill will find out why you should.

Cutting the bus on purpose

The manifest below is the smallest honest partition I know of. It is an egress NetworkPolicy scoped to Node pods that permits DNS and the Router's HTTP port, and permits nothing else. Because Kubernetes NetworkPolicy is allow-list based, leaving 4442 and 4443 out of the allow list is what produces the cut.

# chaos/eventbus-partition.yaml
# Severs Node -> Event Bus traffic while leaving DNS and Router HTTP intact,
# so /status keeps answering and you can watch the Grid disagree with itself.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: chaos-eventbus-partition
  namespace: selenium-chaos
  labels:
    chaos.internal/experiment: eventbus-partition
  annotations:
    chaos.internal/owner: "sdet-platform"
    chaos.internal/expires-at: "2026-08-04T11:30:00Z"
    chaos.internal/abort: "kubectl -n selenium-chaos delete networkpolicy chaos-eventbus-partition"
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: selenium-node
  policyTypes:
    - Egress
  egress:
    # DNS survives. Without this you are testing name resolution, not the bus.
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # Router HTTP survives, so the Grid keeps looking healthy from outside.
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: selenium-router
      ports:
        - protocol: TCP
          port: 4444
    # 4442 (publish) and 4443 (subscribe) are omitted deliberately.

The Compose equivalent, if you are rehearsing locally before you touch a cluster, is to put the Nodes on a second network and detach them from the bus network mid-run. The official Docker images publish 4442, 4443 and 4444 from the hub container and expect Nodes to reach the first two via SE_EVENT_BUS_HOST, SE_EVENT_BUS_PUBLISH_PORT and SE_EVENT_BUS_SUBSCRIBE_PORT, so docker network disconnect on the bus network reproduces the same split without any policy engine.

Whichever you use, write the abort command into the manifest itself, as an annotation, before you apply it. During an incident nobody remembers the label selector.

Reading the Grid while it is lying to you

Here is the triage sequence. It is deliberately built from two independent vantage points, because the entire diagnostic problem is that one vantage point looks fine.

#!/usr/bin/env bash
set -euo pipefail
ROUTER="${ROUTER:-http://localhost:4444}"
NODE="${NODE:-http://localhost:5555}"

echo "== 1. W3C status as seen by the Router (this is what your monitor checks) =="
curl -sS --fail "$ROUTER/status" | jq '{ready: .value.ready, message: .value.message}'

echo "== 2. What the Distributor still believes exists =="
curl -sS --fail -X POST -H 'Content-Type: application/json' \
  --data '{"query":"{ grid { nodeCount totalSlots maxSession sessionCount sessionQueueSize version } }"}' \
  "$ROUTER/graphql" | jq '.data.grid'

echo "== 3. Per-node view, including status and stereotypes =="
curl -sS --fail -X POST -H 'Content-Type: application/json' \
  --data '{"query":"{ nodesInfo { nodes { id uri status slotCount maxSession sessionCount } } }"}' \
  "$ROUTER/graphql" | jq '.data.nodesInfo.nodes'

echo "== 4. What is actually queued, straight from the New Session Queue =="
curl -sS --fail "$ROUTER/se/grid/newsessionqueue/queue" | jq '.'

echo "== 5. Ask the Node directly. This path does not touch the bus. =="
curl -sS --fail "$NODE/status" | jq '{ready: .value.ready, message: .value.message}'

Steps 1 and 5 are W3C WebDriver status calls, so value.ready and value.message are specification-defined and safe to depend on across versions. Steps 2 and 3 use field names documented on the Grid GraphQL support page. Step 4 uses the documented New Session Queue endpoint, which the endpoints page describes as returning the total request count and the payloads.

The diagnosis lives in the contradiction between step 5 and steps 2 through 4. A Node that answers ready: true on its own port while the Router's GraphQL view has stopped listing it is telling you, unambiguously, that the process is alive and the coordination path is not. No other common Grid failure produces that pair of observations.

An illustrative snapshot of the shape you are looking for, with numbers invented purely to show the relationship between the fields:

{
  "before": {
    "grid": { "nodeCount": 12, "totalSlots": 48, "maxSession": 48, "sessionCount": 31, "sessionQueueSize": 0 }
  },
  "during": {
    "grid": { "nodeCount": 4, "totalSlots": 16, "maxSession": 16, "sessionCount": 9, "sessionQueueSize": 137 },
    "node_direct_status": { "ready": true, "message": "Node has capacity available" }
  },
  "note": "Figures are illustrative and exist only to show which fields move together. Measure your own."
}

nodeCount collapsing while a directly queried Node still reports ready is the fingerprint. sessionQueueSize climbing is the consequence, not the cause, and chasing it is how teams end up adding capacity to a Grid that has plenty.

How to tell it is this and not a look-alike

Four failures land on the same help desk ticket. Here is the field that separates them.

Capacity exhaustion. The queue grows, but totalSlots and nodeCount hold steady and sessionCount stays near maxSession. Sessions are still completing, just not fast enough. The distinguishing evidence is that step 5 and step 2 agree with each other. If both views tell the same story, you have a sizing problem, not a partition, and the fix is capacity or scheduling rather than networking.

Registration secret mismatch. The endpoints page shows X-REGISTRATION-SECRET on the distributor administration calls, and the server section documents --registration-secret as a "Shared secret used to authenticate Node registration requests. Must match the value set on the Hub/Distributor." When this is wrong the Node also fails to appear, but the Node's own logs will show registration being attempted and rejected rather than attempted into silence. Look for whether the Node believes it is talking to anything at all.

A Node that was drained or removed. The distributor endpoints include DELETE /se/grid/distributor/node/<node-id> and POST /se/grid/distributor/node/<node-id>/drain. A drained Node is documented to stop "after all the ongoing sessions are complete" and to accept no new requests. That produces a shrinking nodeCount too, but it shrinks by exactly the nodes someone acted on, in the order they acted, and the Node process exits. A partition takes them in a group and leaves the processes running.

DNS failure. This is the one that most often masquerades as a bus split, especially in Kubernetes, because it also produces a Node that is alive and unregistered. The tell is that a partition leaves HTTP working to the Router while DNS failure breaks everything by name. Resolve the bus hostname from inside a Node pod. If the name does not resolve, you are debugging CoreDNS, not the bus. This is also why the chaos manifest above explicitly allows port 53: a drill that breaks DNS as a side effect teaches you nothing about the bus.

The single highest-value habit here is to never diagnose from the Router alone. One curl against a Node's own port, on 5555, costs nothing and eliminates half the hypothesis space.

Worked example one: the node that registered once and then vanished

A team ran Nodes as a Kubernetes Deployment with a readiness probe on GET /status against port 5555. During a cluster upgrade, a node pool drained and rescheduled the Selenium Node pods while the Hub pod was itself being rescheduled. Every Node pod came back Ready, because the probe only asks the Node about itself.

What actually happened is described entirely by two flags. --register-cycle defaults to 10 seconds and --register-period defaults to 120 seconds, and the reference states that after the register period completes, "the Node will not attempt to register again." The Hub took longer than two minutes to become reachable. Every Node exhausted its registration window against nothing, stopped trying, and then sat there passing its readiness probe indefinitely.

The Grid recovered when someone deleted the pods by hand, which is a fix that works and teaches nothing. The durable fix was two lines of configuration:

# Node startup, with registration failure made visible to the orchestrator.
java -jar selenium-server-<version>.jar node \
  --publish-events tcp://selenium-hub.selenium-chaos.svc.cluster.local:4442 \
  --subscribe-events tcp://selenium-hub.selenium-chaos.svc.cluster.local:4443 \
  --grid-url https://grid.internal \
  --register-period 300 \
  --register-shutdown-on-failure true \
  --max-sessions 4 \
  --session-timeout 300

--register-shutdown-on-failure is documented as causing the Node to "shut down after the register period is completed without a successful registration. Useful in container environments to trigger a restart." Turning it on converts a silent, permanent, invisible failure into a crash loop, and a crash loop is something your existing alerting already understands. Widening --register-period buys the Hub more time to come back before that happens.

The cost is real and you should say it out loud when you propose it: with this flag on, a genuinely long control-plane outage will restart your entire Node fleet rather than letting it wait. If your Nodes take a long time to become useful, you have traded a silent failure for a slow one. Measure both against your own startup time before choosing.

Worked example two: the queue that filled while nothing was wrong

A second team saw the queue climb during their nightly run and reacted the way most teams do, by scaling the Node deployment. The queue kept climbing. They doubled it again. Nothing changed, which is the point at which someone should have stopped adding capacity.

The relevant behaviour is the New Session Queue's own timers, documented in the SessionQueue section:

• --session-request-timeout, default 300 seconds: "A new incoming session request is added to the queue. Requests sitting in the queue for longer than the configured time will timeout."
• --session-request-timeout-period, default 10 seconds: "How often, in seconds, the timeout for queued new session requests is checked."
• --sessionqueue-batch-size, default 20: "Maximum number of session requests that can be consumed from the queue at a time, based on the available slots."
• --session-retry-interval, documented in the CLI reference as a retry interval in milliseconds, with the description "If all slots are busy, new session request will be retried after the given interval."

That last one is worth flagging carefully: the CLI reference and the Docker image documentation state this default differently, so read the value for the exact distribution you run rather than trusting either from memory.

None of those timers were the problem. The problem was that new Nodes were joining a network segment that could not reach 4442, so scaling the deployment added slots the Distributor never learned about. totalSlots in the GraphQL view had not moved despite the replica count tripling, and that single observation, checked first, would have saved four hours.

The rule that came out of it is now in their runbook, and it is a good one: before adding capacity, prove that the last capacity you added is visible. nodeCount and totalSlots must move when replicas move. If they do not, you have a coordination problem, and every additional replica makes the incident harder to read.

Turning the drill into a CI gate

A drill you run by hand once is a story. A drill that runs on a schedule and fails a pipeline is a control. This is the probe, written against the documented GraphQL fields, using the Java HTTP client and Jackson so it runs anywhere your existing test tooling runs.

package dev.example.grid.chaos;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.time.Instant;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertTrue;

class EventBusPartitionDrillTest {

  private static final String ROUTER = System.getProperty("grid.router", "http://localhost:4444");
  private static final String GRID_QUERY =
      "{\"query\":\"{ grid { nodeCount totalSlots sessionQueueSize } }\"}";

  private static final HttpClient HTTP = HttpClient.newBuilder()
      .connectTimeout(Duration.ofSeconds(5))
      .build();
  private static final ObjectMapper JSON = new ObjectMapper();

  private JsonNode gridSnapshot() throws IOException, InterruptedException {
    HttpRequest request = HttpRequest.newBuilder(URI.create(ROUTER + "/graphql"))
        .header("Content-Type", "application/json")
        .timeout(Duration.ofSeconds(10))
        .POST(HttpRequest.BodyPublishers.ofString(GRID_QUERY))
        .build();
    HttpResponse<String> response = HTTP.send(request, HttpResponse.BodyHandlers.ofString());
    if (response.statusCode() != 200) {
      throw new IllegalStateException("Router GraphQL returned HTTP " + response.statusCode());
    }
    return JSON.readTree(response.body()).path("data").path("grid");
  }

  private static void kubectl(String... args) throws IOException, InterruptedException {
    String[] command = new String[args.length + 1];
    command[0] = "kubectl";
    System.arraycopy(args, 0, command, 1, args.length);
    Process process = new ProcessBuilder(command).inheritIO().start();
    if (process.waitFor() != 0) {
      throw new IllegalStateException("kubectl failed: " + String.join(" ", command));
    }
  }

  @Test
  void gridReturnsToItsOwnSteadyStateAfterTheBusIsSevered() throws Exception {
    JsonNode before = gridSnapshot();
    int expectedNodes = before.path("nodeCount").asInt();
    int expectedSlots = before.path("totalSlots").asInt();
    assertTrue(expectedNodes > 0, "refusing to run a drill against an already-empty Grid");

    kubectl("-n", "selenium-chaos", "apply", "-f", "chaos/eventbus-partition.yaml");
    try {
      Thread.sleep(Duration.ofMinutes(3).toMillis());
      JsonNode during = gridSnapshot();
      assertTrue(during.path("nodeCount").asInt() < expectedNodes,
          "the partition did not take effect; the drill proved nothing");
    } finally {
      kubectl("-n", "selenium-chaos", "delete", "networkpolicy", "chaos-eventbus-partition");
    }

    // Recovery budget is a local measurement, not a universal constant.
    Duration budget = Duration.ofMinutes(
        Long.getLong("grid.recoveryBudgetMinutes", 6L));
    Instant deadline = Instant.now().plus(budget);
    JsonNode latest = null;
    while (Instant.now().isBefore(deadline)) {
      latest = gridSnapshot();
      if (latest.path("nodeCount").asInt() >= expectedNodes
          && latest.path("totalSlots").asInt() >= expectedSlots) {
        return;
      }
      Thread.sleep(Duration.ofSeconds(15).toMillis());
    }
    throw new AssertionError(
        "Grid did not return to steady state within " + budget
            + ". expected nodeCount>=" + expectedNodes
            + " totalSlots>=" + expectedSlots
            + " but last observed " + latest);
  }
}

Two things about that test are deliberate. It refuses to run against an empty Grid, because a drill that starts from zero cannot detect anything. And it asserts that the partition took effect before it asserts recovery, because the worst possible outcome is a green drill that never actually cut anything. A chaos test that cannot fail on its own injection step is decoration.

The recovery budget is read from a system property with no hardcoded expectation baked into the article. Derive yours by running the drill three or four times against a quiet Grid and taking the slowest observed recovery plus margin. Your number is a function of your heartbeat and purge settings, your pod scheduling latency and your image pull time, and none of those are knowable from here.

Second failure mode: the half-open split

Everything above assumes a clean cut in both directions. The more interesting failure, and the one that a NetworkPolicy cannot produce, is asymmetric: publish reachable, subscribe blocked, or the reverse.

This matters because --publish-events and --subscribe-events are separate connection strings on separate ports. A firewall rule, a security group, or a partially applied policy can easily allow one and deny the other. When that happens a component can be broadcasting perfectly while hearing nothing, which produces a Grid whose components hold genuinely different models of the same fleet rather than one model that is simply stale.

The diagnostic is to test each direction independently from inside a Node, rather than assuming they share a fate:

# Run from inside a Node container. Tests reachability of each bus port separately.
BUS_HOST="${SE_EVENT_BUS_HOST:-selenium-hub}"
PUB="${SE_EVENT_BUS_PUBLISH_PORT:-4442}"
SUB="${SE_EVENT_BUS_SUBSCRIBE_PORT:-4443}"

getent hosts "$BUS_HOST" || echo "FAIL: $BUS_HOST does not resolve (this is DNS, not the bus)"

for port in "$PUB" "$SUB"; do
  if timeout 3 bash -c "cat < /dev/null > /dev/tcp/$BUS_HOST/$port" 2>/dev/null; then
    echo "OK   tcp://$BUS_HOST:$port reachable"
  else
    echo "FAIL tcp://$BUS_HOST:$port unreachable"
  fi
done

# Asymmetry is the finding. One OK and one FAIL is a different bug from two FAILs.

Two FAILs and a resolving hostname is the clean partition covered above. One OK and one FAIL is the half-open case, and it deserves its own runbook entry because the symptoms are less consistent: some coordination appears to work, which sends people looking for capability or version problems that are not there.

To rehearse this deliberately you need something that can drop traffic per port and per direction. A service mesh sidecar or a chaos controller with fault injection can do it. So can a plain iptables rule inside a container you own, if the container has the capability to modify its own rules and you have accepted that this is a privileged operation. Whichever route you take, the acceptance criterion is the same as before: prove the injection took effect, then measure the return to a steady state you measured yourself.

Rolling this out without frightening anyone

The order below is the one I would defend in a planning meeting, because each step produces something useful even if the next step never happens.

Step one, baseline. Before any injection, record nodeCount, totalSlots, maxSession, sessionCount and sessionQueueSize from the Router GraphQL endpoint every fifteen seconds for a full working day. You cannot assert a recovery to steady state without knowing what steady state looks like on a Tuesday afternoon.

Step two, monitor the right thing. Add nodeCount and totalSlots to whatever dashboard your on-call rotation actually looks at, and alert on a sustained drop rather than on the queue. Do this before the first drill. If the drill triggers no alert, the drill has already found a defect.

Step three, drill in an isolated copy. A separate namespace, a separate Compose project, its own Router, and a Node fleet nobody is running tests against. Not a shared Grid with a "please avoid" message in chat.

Step four, drill during business hours with an owner present. The purpose of a chaos exercise is to observe humans reading dashboards, not only to observe software recovering. Running it at 3am against nobody proves the least interesting half.

Step five, promote to a schedule. Weekly or per-release, in the isolated environment, with the JUnit probe above as the gate. At this point failures are regressions in your recovery behaviour, which is exactly what you wanted to be able to detect.

Step six, add the asymmetric case. Only once the clean partition passes reliably. Half-open failures are harder to reason about and you want a known-good baseline before you introduce them.

What this costs, honestly

The engineering time is modest: a manifest, a probe and a dashboard panel, call it a couple of days for the first version. The ongoing costs are the ones people underestimate.

You need a second Grid. Not a scaled-down toy, because timing behaviour is what you are measuring, and a two-node Grid recovers differently from a thirty-node Grid. That is a real infrastructure line item, and the honest justification for it is not chaos testing alone, it is that you also need somewhere to test Grid version upgrades.

You will spend time on false starts. NetworkPolicy semantics vary by CNI plugin, and a policy that partitions cleanly on one cluster may be partially enforced on another. Budget for the first two or three attempts producing no partition at all, which is precisely why the probe asserts that the injection worked.

You will surface pre-existing bugs. That is the point, but it is worth setting expectations before you start, because the first drill often produces a list of unrelated fixes rather than a clean pass. --register-shutdown-on-failure being unset is the most common of these.

And the recovery behaviour you tune has a cost of its own. Shorter heartbeat and purge intervals detect a partition faster and also make the Grid more sensitive to ordinary network noise, which can churn your node model during a busy run. Faster detection is not free, and the correct setting depends on how stable your network genuinely is rather than on how stable you would like it to be.

When not to do this

Skip this entirely if you run Standalone mode. The getting-started guide's Standalone command starts everything in one process, and there is no network hop between components to sever. A partition drill against Standalone is testing the loopback interface.

Skip it if your Grid is ephemeral per pipeline. If every CI run spins up a fresh Grid, runs for eleven minutes, and tears it down, the failure mode you are rehearsing is bounded by the run itself. Your budget is better spent on making startup deterministic and on failing fast when a Node does not register within the first minute.

Delay it if you do not yet monitor nodeCount. Chaos engineering assumes you can observe the system. Injecting a failure into a Grid you cannot see is not an experiment, it is an outage with extra steps. Build the dashboard first; you may well discover partitions you have been having all along.

Delay it if you have no owner. These drills leave residue, whether that is a stale NetworkPolicy, a drained Node, or a fleet stuck past its registration window. Someone has to be accountable for the cleanup and for the runbook that results. Without that, the drill runs once, produces a screenshot for a slide, and the environment quietly rots.

And do not run it against a Grid that shares a network segment with anything else you care about. Egress policies are blunt instruments. Verify your podSelector matches exactly the pods you intend, in a namespace that contains nothing else, before you apply anything.

Related reading

For the connectivity basics that sit underneath all of this, start with the guide on debugging Selenium Grid event bus connectivity and the breakdown of event bus failure modes. If your Nodes are containers, the registration timing discussed above interacts directly with the patterns in running Selenium Grid on Kubernetes with disposable nodes and running Selenium tests in Docker. For a refresher on the topology itself, the Selenium Grid tutorial covers Hub and Node before you get to distributed mode.

FAQ

Which ports does the Selenium Grid event bus use, and can I change them?

Two connection strings control it. The CLI reference documents --publish-events with a default of tcp://*:4442 and --subscribe-events with a default of tcp://*:4443, and both accept any TCP URI you give them. In the Docker images the same values are set through SE_EVENT_BUS_HOST, SE_EVENT_BUS_PUBLISH_PORT and SE_EVENT_BUS_SUBSCRIBE_PORT. Changing them is fine as long as every bus client is changed together, because a Node pointed at the old port will never be corrected by the Router.

How do I tell a bus partition apart from a Grid that is simply out of slots?

Compare two numbers that come from different places. sessionQueueSize from the Router GraphQL endpoint tells you demand; nodeCount and totalSlots from the same query tell you what the Distributor still believes exists. Capacity exhaustion shows a full queue with totalSlots unchanged and sessions completing steadily. A partition shows a full queue while nodeCount falls, even though a direct call to the Node's own /status answers normally.

Does blocking the event bus ports stop tests that are already running?

Not immediately, and that is exactly what makes the failure confusing. A session that already has a slot talks to the Node over HTTP through the Router, which is a separate path from the bus. So in-flight tests often finish cleanly while every new session request piles up in the New Session Queue behind them.

Is it safe to run a partition drill against a shared Grid?

No. Run it in a namespace or Compose project that nothing else routes to, with a hard expiry on the chaos manifest and a Node fleet you are willing to lose. The reason is not politeness: --register-period documents that a Node stops attempting first registration after the period elapses, so a drill can leave Nodes running but permanently invisible.

What should the drill actually assert, given that I cannot invent recovery numbers?

Assert a return to your own measured steady state, not to a number from an article. Record nodeCount, totalSlots and sessionQueueSize before you cut the bus, then require the same node and slot counts within a recovery budget you derived from your own baseline runs. The budget is a property of your --heartbeat-period, --register-cycle and --purge-nodes-interval settings plus your scheduler, so it has to be measured locally.

Do I need a service mesh or a chaos platform to do this?

A NetworkPolicy or a Compose network is enough for the first version. The value of a mesh or a chaos controller comes later, when you want to inject one-way loss or latency rather than a clean cut, because a partial partition is the failure mode that the simple manifest cannot reproduce.

Practice this

Take the last Grid incident your team had where the fix was "restart the nodes" and ask which of the five triage commands above would have named the cause in the first minute. If the answer is none of them, that is the gap. Then try the same reasoning under time pressure in the QABattle battle arena: pick a Selenium infrastructure scenario, state which single observation would separate a partition from capacity exhaustion, and commit to it before you look at the answer.