Why healthy Selenium Grid pods still cannot create a session

Every Grid pod is Ready, yet a new session sits in the queue until the client times out. The failure appeared the moment a default-deny policy reached the namespace. Reopening every port makes the suite green, but it also removes the isolation the policy was meant to provide.

That pattern is common because a distributed Grid is not one service. It is a set of processes with different traffic directions, and a Kubernetes Service does not grant network access by itself. The practical job is to permit the smallest complete conversation, then prove each part of that conversation from the identity that actually uses it.

Why readiness and Grid readiness disagree

A pod can pass a readiness probe while the Grid is unusable. A Router probe may call its local /status endpoint. An Event Bus probe may confirm that the status server on port 5557 responds. A Node probe may only confirm that the Node process accepts HTTP. None of those checks necessarily crosses the policy boundary that is blocking real work.

Selenium documents the Router as the Grid entry point. New session requests go from the Router to the New Session Queue. The Distributor polls that queue, chooses a slot, asks a Node to create the browser session, and records the session-to-Node relationship in the Session Map. Commands for an existing session return through the Router and are forwarded to the owning Node.

Node registration has another important shape. A Node sends a registration event through the Event Bus. After the Distributor receives that event, it reaches the Node over HTTP to confirm that the Node exists. One-way access to the Event Bus is therefore insufficient. The event can arrive while the Distributor's callback to port 5555 is dropped, leaving a Node process that looks healthy but never becomes usable capacity.

The default ports in a fully distributed Grid make the dependency graph concrete:

Those are defaults, not promises about your deployment. A Helm values file, TOML file, or command-line argument can change them. The Selenium CLI documentation is the right contract for the server options, while the running pod specification is the truth for a particular cluster. If the two disagree, write policy for the effective value and fix the configuration drift separately.

Kubernetes adds two rules that regularly surprise test teams. First, policies are additive. A new allow policy does not cancel a default deny, it contributes another allowed path. Second, both sides matter when egress and ingress are restricted. Router egress to the Session Map can be allowed while Session Map ingress from the Router remains denied. The packet still does not complete.

Service discovery is a third boundary. A default-deny egress policy can block DNS before any Selenium port is attempted. In that case, logs name a Service that cannot be resolved, and a port probe using the Service DNS name fails before opening a socket. Treat DNS as infrastructure shared by the namespace, not as an accidental side effect of allowing broad egress.

Which connections a distributed Grid actually needs

Draw the flow from observed process arguments, not from pod names. Teams often call one Deployment hub, another grid, and a third workers. Those labels do not tell a policy reviewer whether the pod is acting as Router, Distributor, Event Bus, or Node. Put the role in a stable pod label and use that label in both NetworkPolicy selectors and diagnostic commands.

The following role labels are examples owned by the deployment, not Selenium configuration keys:

metadata:
  labels:
    qa-grid/component: router
spec:
  template:
    metadata:
      labels:
        qa-grid/component: router

Apply the label to the pod template. Labeling only the Deployment object does not label existing pods, and a NetworkPolicy selects pods rather than workload controllers. Confirm the result with:

kubectl -n selenium get pods -L qa-grid/component -o wide

A useful traffic inventory has four columns: source role, destination role, destination port, and reason. Add a fifth column for the command-line option that establishes the address. That last column catches a class of bugs that policy changes cannot fix.

Do not copy the table blindly if you combine roles. A Hub packages Router, Distributor, Session Map, Queue, and Event Bus into one process. Standalone adds a Node. Traffic between roles inside the same process may never cross a pod boundary, so a policy for a Hub-and-Node topology is smaller than one for a fully distributed topology.

The reverse mistake is more damaging: assuming that a Service account or Service object grants connectivity. NetworkPolicy is primarily based on pod and namespace selectors plus ports. A ClusterIP gives a stable virtual address, but traffic is eventually delivered to selected pod endpoints and is still subject to policy enforcement by the cluster network implementation.

Keep test runners outside the Grid namespace when possible. Then the public rule is easy to review: a namespace with an explicit access label may reach only Router port 4444. The browser Nodes do not need inbound access from every test worker. They need access from the internal Grid roles that route and schedule sessions.

DNS deserves an explicit line in the inventory when egress is denied. Many clusters label the DNS namespace kubernetes.io/metadata.name: kube-system, but the DNS pods and ports can vary. Inspect the cluster before committing a selector. A policy that permits UDP 53 but not TCP 53 also has an edge case: larger DNS responses and retries can use TCP.

How to write the policy without guessing

The first worked example builds the perimeter in layers. It assumes the selenium namespace exists, each Grid pod has one qa-grid/component label, and test runners live in namespaces labeled qa-access=grid-clients. It uses the standard distributed Grid ports. If your live arguments differ, change the manifests before applying them.

Start with namespace-wide deny rules and a narrow client entry point:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: selenium
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: selenium
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: clients-to-router
  namespace: selenium
spec:
  podSelector:
    matchLabels:
      qa-grid/component: router
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              qa-access: grid-clients
      ports:
        - protocol: TCP
          port: 4444

Next, allow the Router to reach only the four HTTP roles it calls. Separate rules cost more YAML, but they make a review meaningful. A single rule listing every internal port for every destination would let the Router reach unrelated listeners that happen to use those ports.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: router-egress
  namespace: selenium
spec:
  podSelector:
    matchLabels:
      qa-grid/component: router
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: distributor
      ports:
        - protocol: TCP
          port: 5553
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: session-map
      ports:
        - protocol: TCP
          port: 5556
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: session-queue
      ports:
        - protocol: TCP
          port: 5559
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: node
      ports:
        - protocol: TCP
          port: 5555
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: router-to-internal-ingress
  namespace: selenium
spec:
  podSelector:
    matchExpressions:
      - key: qa-grid/component
        operator: In
        values: [distributor, session-map, session-queue, node]
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              qa-grid/component: router
      ports:
        - protocol: TCP
          port: 5553
        - protocol: TCP
          port: 5555
        - protocol: TCP
          port: 5556
        - protocol: TCP
          port: 5559

That combined ingress rule is readable, but it is intentionally a little wider than a policy per destination. It allows a Router packet to any listed port on any selected internal role. Normally only one of those ports is listening in each container, so the practical exposure is limited. A high-assurance cluster should split it into four destination-specific policies to prevent a future sidecar or debug listener from inheriting access.

This destination-specific rule is the Node portion of that stricter fix. Only Router and Distributor pods can open the Node's WebDriver port:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: node-http-from-grid-control
  namespace: selenium
spec:
  podSelector:
    matchLabels:
      qa-grid/component: node
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: qa-grid/component
                operator: In
                values:
                  - router
                  - distributor
      ports:
        - protocol: TCP
          port: 5555

The Distributor requires queue, map, Node, and Event Bus access. Nodes need the Event Bus, while Router traffic does not. The following example completes those paths:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: distributor-egress
  namespace: selenium
spec:
  podSelector:
    matchLabels:
      qa-grid/component: distributor
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: session-queue
      ports:
        - {protocol: TCP, port: 5559}
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: session-map
      ports:
        - {protocol: TCP, port: 5556}
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: node
      ports:
        - {protocol: TCP, port: 5555}
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: event-bus
      ports:
        - {protocol: TCP, port: 4442}
        - {protocol: TCP, port: 4443}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: distributor-to-internal-ingress
  namespace: selenium
spec:
  podSelector:
    matchExpressions:
      - key: qa-grid/component
        operator: In
        values: [session-map, session-queue, node]
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              qa-grid/component: distributor
      ports:
        - {protocol: TCP, port: 5555}
        - {protocol: TCP, port: 5556}
        - {protocol: TCP, port: 5559}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: event-bus-client-egress
  namespace: selenium
spec:
  podSelector:
    matchExpressions:
      - key: qa-grid/component
        operator: In
        values: [node, session-map, session-queue]
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              qa-grid/component: event-bus
      ports:
        - {protocol: TCP, port: 4442}
        - {protocol: TCP, port: 4443}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: event-bus-ingress
  namespace: selenium
spec:
  podSelector:
    matchLabels:
      qa-grid/component: event-bus
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: qa-grid/component
                operator: In
                values: [distributor, node, session-map, session-queue]
      ports:
        - {protocol: TCP, port: 4442}
        - {protocol: TCP, port: 4443}

Some Grid versions or topologies connect additional components to the Event Bus. Inspect the component arguments and Selenium version before treating the last selector list as universal. The policy is correct only when it matches the clients you actually start.

The cost of this approach is visible. You now own role labels, port values, policies, and tests as one versioned contract. A Grid upgrade that changes a role or introduces a new connection can fail closed. That is safer than silently widening access, but it requires an operational owner and a rollout check.

What the failure evidence says

A WebDriver exception is the end of the chain, not enough evidence to blame policy. Diagnose from the last confirmed boundary. Start outside the Grid, then move inward only when the previous hop works.

This second worked example creates a short-lived probe in a client namespace and calls the Router. It does not need a browser or Selenium binding:

kubectl create namespace qa-runners --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace qa-runners qa-access=grid-clients --overwrite

kubectl -n qa-runners run router-check --rm -i --restart=Never --image=curlimages/curl:8.10.1 --command -- curl --fail --silent --show-error --connect-timeout 3 http://selenium-router.selenium.svc:4444/status

A healthy response contains a value.ready field. When it is false, read the accompanying message and Node list rather than declaring the Router unreachable. A policy failure at this boundary usually produces output shaped like:

curl: (28) Failed to connect to selenium-router.selenium.svc port 4444
after 3002 ms: Timeout was reached

If DNS is blocked, the message instead names resolution:

curl: (6) Could not resolve host: selenium-router.selenium.svc

Those outcomes have different owners. The first says the name resolved but the TCP connection did not complete. The second says the test never reached port 4444. Check the namespace label, Router pod label, Service selector, and both sides of the policy before changing Selenium timeouts.

For internal paths, use a probe image with nc and give the probe the same role label as the real source. The sample Service names below must match your manifests:

# Expected to succeed: a Node reaches both Event Bus data ports.
kubectl -n selenium run probe-node-publish --rm -i --restart=Never --image=nicolaka/netshoot:v0.13 --labels='qa-grid/component=node' --command -- nc -zvw3 selenium-event-bus 4442

kubectl -n selenium run probe-node-subscribe --rm -i --restart=Never --image=nicolaka/netshoot:v0.13 --labels='qa-grid/component=node' --command -- nc -zvw3 selenium-event-bus 4443

# Expected to succeed: the Distributor verifies a Node over HTTP/TCP.
kubectl -n selenium run probe-distributor-node --rm -i --restart=Never --image=nicolaka/netshoot:v0.13 --labels='qa-grid/component=distributor' --command -- nc -zvw3 selenium-node 5555

# Expected to fail: test clients are not allowed to bypass the Router.
kubectl -n qa-runners run probe-client-node --rm -i --restart=Never --image=nicolaka/netshoot:v0.13 --command -- nc -zvw3 selenium-node.selenium.svc 5555

A successful nc check prints a connection success and exits zero. A filtered path normally waits and exits nonzero with a timeout. A closed port tends to fail immediately with Connection refused. That distinction is valuable: refusal usually points to a missing listener, wrong targetPort, or endpoint mismatch, while a timeout is consistent with a drop policy or an unreachable route.

This diagnostic checks that the Node Service has endpoints, then launches a probe with the Distributor's policy identity to test DNS and TCP separately:

#!/usr/bin/env bash
set -euo pipefail

grid_namespace=${GRID_NAMESPACE:-selenium}
node_service=${NODE_SERVICE:-selenium-node}

endpoint_addresses=$(
  kubectl -n "$grid_namespace" get endpointslice \
    -l "kubernetes.io/service-name=$node_service" \
    -o jsonpath='{range .items[*].endpoints[*]}{range .addresses[*]}{.}{"\n"}{end}{end}'
)

if [[ -z "$endpoint_addresses" ]]; then
  printf 'no endpoint addresses for Service %s\n' "$node_service" >&2
  exit 1
fi

printf 'node_endpoints=%s\n' "$(tr '\n' ',' <<<"$endpoint_addresses")"

kubectl -n "$grid_namespace" run "distributor-check-$RANDOM" \
  --rm -i --restart=Never \
  --image=nicolaka/netshoot:v0.13 \
  --labels=qa-grid/component=distributor \
  --env="NODE_SERVICE=$node_service" \
  --command -- sh -ec '
    getent hosts "$NODE_SERVICE"
    nc -z -w 3 "$NODE_SERVICE" 5555
  '

Do not stop at the socket test. It proves that a TCP handshake can complete, not that Grid messages are exchanged correctly. Pair it with these read-only views:

kubectl -n selenium get networkpolicy
kubectl -n selenium describe networkpolicy event-bus-ingress
kubectl -n selenium get pods -L qa-grid/component
kubectl -n selenium get service
kubectl -n selenium get endpointslice
kubectl -n selenium logs deployment/selenium-distributor --since=10m
kubectl -n selenium logs deployment/selenium-event-bus --since=10m
kubectl -n selenium logs deployment/selenium-node --since=10m

EndpointSlices answer a near-miss that looks like policy: a Service resolves and accepts no connection because its selector found no Ready endpoints. Compare the endpoint addresses with the destination pods. If the list is empty, fix labels, readiness, or the Service selector. Adding an allow rule cannot create an endpoint.

A wrong Event Bus address is another near-miss. Port probes from the Node to the intended Event Bus Service pass, but the Node process was started with a different hostname or port. Read the actual container arguments and configuration mounted into that pod. If logs show attempts against the wrong destination, the network is doing exactly what the process requested.

Resource starvation can imitate lost registration. A Node publishes late because browser discovery or startup is stalled, or a Distributor falls behind under CPU pressure. In that case, policy probes stay green and Kubernetes metrics show throttling or restarts. Capture timestamps from Node and Distributor logs around the same restart. Do not loosen network rules to compensate for scheduling delay.

The strongest end-to-end check creates an actual session through the Router. This Python example avoids the application network by loading a data URL, so failure stays focused on the Grid control and command paths:

import os
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

grid_url = os.environ.get(
    "SELENIUM_REMOTE_URL",
    "http://selenium-router.selenium.svc:4444",
)

options = Options()
options.add_argument("--headless=new")

started = time.monotonic()
driver = webdriver.Remote(command_executor=grid_url, options=options)
try:
    creation_seconds = time.monotonic() - started
    driver.get(
        "data:text/html,"
        "<title>grid-policy-smoke</title>"
        "<main id='result'>session reached its node</main>"
    )
    assert driver.title == "grid-policy-smoke"
    assert driver.find_element("id", "result").text == "session reached its node"
    print(f"session_created_seconds={creation_seconds:.3f}")
    print(f"session_id={driver.session_id}")
finally:
    driver.quit()

Run it from the same namespace and service account used by CI. A laptop connected through port-forwarding takes a different path and can produce a false green result. Keep the session ID with Router, Distributor, and Node logs so the team can show whether creation reached a slot and whether commands reached the selected Node.

How to roll it out without breaking every shard

NetworkPolicy has no portable audit-only mode. A large production namespace is a poor place to discover undocumented flows by denial. Roll out in stages and give each stage a rejecting test.

First, add role labels without policies. Verify that every current and replacement pod receives exactly one role. A typo such as sessionmap versus session-map can make an allow selector match nothing. It can also make the intended pod escape a selective policy if a namespace-wide default deny is not yet active.

Second, record effective Grid addresses and ports. Collect the pod command, environment, and mounted configuration for every role. Compare those values with Services and EndpointSlices. This inventory is a deployment artifact, not a page in a wiki that can drift independently.

Third, apply destination ingress rules while egress remains unrestricted. Run the positive and negative TCP matrix. This stage narrows who can call Grid components but leaves each component able to initiate its old traffic. It is easier to identify an omitted source than when both directions change at once.

Fourth, apply role-specific egress rules. Keep DNS permitted and run the matrix again. Create several real sessions, not just one. A single session may reuse a registered Node and miss the Node-registration path that breaks during scale-up.

Fifth, add the namespace-wide egress deny and recycle one Node. Watch it register, accept a session, execute a command, and quit. Scale the Node Deployment up by one and down again if that matches normal operations. Autoscaling is where an incomplete policy often appears for the first time.

Use a canary namespace when Grid capacity is business-critical. Deploy the same manifests and browser image, direct one CI shard to the canary Router, and compare session creation latency plus failure classification. The canary must include fresh Node registration and session teardown, not only /status.

On a CI runner with cluster access, save the Python smoke example above as ci/grid_policy_smoke.py and wire both the allowed Router path and denied Node bypass into the job:

- name: Create a session through the Router
  env:
    SELENIUM_REMOTE_URL: http://selenium-router.selenium.svc:4444
  run: python ci/grid_policy_smoke.py

- name: Prove runners cannot bypass the Router
  run: |
    kubectl -n qa-runners run grid-node-deny-${{ github.run_id }}-${{ github.run_attempt }} \
      --rm -i --restart=Never \
      --image=nicolaka/netshoot:v0.13 \
      --command -- sh -ec '
        if nc -z -w 3 selenium-node.selenium.svc 5555; then
          printf "runner reached a Node directly\n" >&2
          exit 1
        fi
      '

- name: Collect policy diagnostics
  if: ${{ failure() }}
  run: |
    kubectl -n selenium get networkpolicy -o yaml
    kubectl -n selenium get endpointslice -o wide
    kubectl -n selenium logs deployment/selenium-router --tail=200
    kubectl -n selenium logs deployment/selenium-distributor --tail=200
    kubectl -n selenium logs deployment/selenium-node --tail=200

Preserve a rollback manifest rather than relying on memory during an outage. NetworkPolicy resources are additive, so an emergency allow policy can restore a specific path without removing every control. Make it narrow, time-bound through your incident process, and remove it after the missing flow is encoded correctly.

The migration has measurable cost. Every new role or port requires a policy review. Diagnostic pods need permission to run and may themselves be selected by default deny. CI gains extra smoke-test time, especially when it recycles a Node. Those costs buy a smaller east-west attack surface and faster fault localization, but only if the checks stay maintained.

Policy behavior also depends on the network implementation. The Kubernetes API can accept NetworkPolicy resources even when the cluster networking layer does not enforce them as expected. Prove one negative path. If a client that should be denied still reaches a Node, stop claiming isolation and work with the platform team on enforcement.

When network isolation is the wrong control

Do not split every Grid component merely to make a policy diagram impressive. A small team with one trusted CI namespace may get a clearer, safer result from a Hub-and-Node deployment. Fewer network boundaries mean fewer addresses, policies, and partial failure modes. Distributed mode earns its complexity when scaling and independent component operations matter.

Avoid a default-deny rollout when role ownership is unknown and no one can observe component traffic. The first action in that environment is inventory and instrumentation. Applying denial first creates an outage whose evidence is spread across several teams.

NetworkPolicy is also not encryption or application authentication. A permitted pod can still send malicious or malformed traffic to a Grid component. Use Selenium's registration secret where Node registration needs that control, keep the Router behind appropriate ingress authentication, and use transport security where the threat model requires it. Segmentation limits reachability; it does not make trusted traffic safe.

Do not use broad Router-to-Node access as a substitute for fixing stale Session Map data. If the Router resolves a session to the wrong Node, packets can reach that Node successfully and the command still fails. Session identity, Node URI, and Grid logs are the evidence for that problem.

Similarly, a policy is not the place to fix browser access to the application under test. Browser traffic originates from Node pods, so a strict Node egress rule can block the product even while Grid control traffic works. Decide explicitly which application hosts, proxies, certificate services, and DNS paths browsers require. Mixing those rules with Event Bus rules makes both harder to review.

Performance testing is another poor fit for the end-to-end smoke shown above. Session creation latency includes queueing, browser startup, image state, and Grid messaging. It can detect a severe regression, but it cannot attribute milliseconds to NetworkPolicy. Use tracing and platform telemetry for that question.

Finally, skip per-component isolation in a local disposable cluster if the policy never ships with the real deployment. A test that exercises a different CNI, different labels, and a Standalone Grid gives confidence in YAML syntax, not in production reachability. Put the meaningful checks next to the manifests and topology they protect.