A PodDisruptionBudget cannot drain a Selenium Node for you

A cluster upgrade evicts two browser Pods, and six tests fail halfway through checkout. The PodDisruptionBudget looked correct because it kept three Pods available. Unfortunately, all three remaining Pods offered Chrome, while the evicted Pod was the only Firefox capacity and still owned a live session.

Know what the budget can and cannot see

Kubernetes counts Pods. Selenium Grid counts Nodes, slots, stereotypes, and sessions. A PodDisruptionBudget works with the first model. It has no field for a WebDriver session ID, no concept of a slot being occupied, and no way to know that one browser session has been running for forty minutes.

A PDB limits how many selected Pods can be unavailable during voluntary disruptions that use the Kubernetes Eviction API. The API evaluates the selector, desired replica count, current health, and the budget's minAvailable or maxUnavailable rule. When an eviction would exceed the allowance, it rejects or delays that eviction. Kubectl drain uses this path and retries rejected evictions until they succeed or its timeout expires.

That protection is narrower than many maintenance runbooks assume. A PDB does not prevent:

• a machine, network, or hypervisor failure;
• Node pressure eviction by the kubelet;
• direct deletion of a Pod or its owning Deployment;
• every consequence of a workload-controller rollout;
• a Selenium process crash inside a Pod;
• a browser crash inside a healthy Selenium Node;
• termination after the Pod's grace period expires.

Involuntary failures count against availability, but the budget cannot stop them. If one of four selected Pods is already unready and maxUnavailable is one, another voluntary eviction should be blocked. This is useful because it prevents maintenance from adding a second planned loss to an existing one.

The PDB health calculation usually relies on the Pod Ready condition. A Selenium Node can be Kubernetes-ready while every browser slot is occupied. It can also remain HTTP-reachable while Grid marks it DRAINING. Readiness therefore cannot substitute for Grid session evidence.

Selenium has its own drain operation. The documented endpoint tells a Node to stop accepting new sessions, finish ongoing sessions, and then stop. That is the session-aware half of maintenance. Kubernetes eviction is the workload-aware half. Safe maintenance needs both in the correct order.

A concise ownership rule helps:

1. Selenium drain controls admission of new browser work and waits for owned sessions.
2. GraphQL or Node status proves the session count and Grid availability.
3. Kubernetes cordon prevents new Pods from scheduling onto the host under maintenance.
4. Eviction removes Pods while respecting PDB concurrency.
5. The workload controller creates replacement Pods on eligible hosts.
6. Grid registration proves replacement capacity is usable before the next eviction.

Skipping step one can terminate a live browser. Skipping step six lets maintenance consume disruption allowance faster than Grid restores compatible slots.

Budget roles and browser pools separately

A Grid deployment contains components with different failure and replacement behavior. One broad selector such as app=selenium is easy to write and hard to reason about. Five Chrome Nodes do not make one Router available, and two Routers do not replace the only Session Map process.

Create budgets around interchangeable Pods. Typical groups include:

• Router replicas that share the same configuration and front-door service;
• Distributor replicas, when the deployed architecture supports that topology;
• Session Map and New Session Queue components according to their persistence and replica model;
• Chrome Nodes with the same platform, version policy, and network access;
• Firefox Nodes as a separate pool;
• rare Windows, macOS, mobile relay, or private-network pools on their own.

Labels used by the PDB must appear on the Pod template, not only on a Deployment or Helm release object. Inspect live Pods. A selector typo can yield expectedPods zero, which looks harmless in YAML but protects nothing.

The next manifest gives a two-replica Router pool a minimum of one available Pod and a four-replica Chrome Node pool at most one unavailable Pod. It also creates a separate Firefox budget. The labels are examples and must match the workload templates in the same namespace.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: selenium-router
  namespace: grid
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: selenium-grid
      app.kubernetes.io/component: router
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: selenium-node-chrome
  namespace: grid
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: selenium-grid
      app.kubernetes.io/component: node
      grid.selenium.dev/browser: chrome
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: selenium-node-firefox
  namespace: grid
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: selenium-grid
      app.kubernetes.io/component: node
      grid.selenium.dev/browser: firefox

AlwaysAllow needs a deliberate explanation for browser Nodes. It permits eviction of an unhealthy selected Pod even when the healthy budget is tight. That prevents a stopped or broken Selenium process from blocking a Kubernetes node drain forever. It is safe only when your maintenance automation drains WebDriver sessions before the Pod becomes unhealthy or stops. If readiness turns false while sessions are still active, AlwaysAllow can let eviction finish before those sessions do.

The default unhealthy-pod behavior favors availability and may block eviction until enough healthy replicas exist. That can be appropriate when there is no trusted session-aware coordinator. Test both the happy path and a stuck unhealthy Pod before choosing the field. Do not copy it from an unrelated web service.

Check the cluster version before relying on the field at all. unhealthyPodEvictionPolicy arrived as a beta field in Kubernetes 1.27 and reached general availability in 1.31. An API server older than 1.27, or one with the feature gate turned off, drops the field silently during validation. The manifest applies without an error, kubectl get pdb -o yaml simply does not show it, and eviction keeps the old default behavior while your runbook assumes otherwise. Read the field back from the live object after applying it and treat its absence as an unmet prerequisite, not as a cosmetic difference between the file and the cluster.

MinAvailable and maxUnavailable express different operational intent. MinAvailable is intuitive for a small control-plane pool: keep at least one Router. MaxUnavailable scales naturally with a Node pool: maintenance may take one out at a time. Percentages can change rounding behavior as replicas scale, so use integer values when losing exactly one specialized Node is the rule.

A one-replica component with minAvailable one allows no voluntary eviction. That is not graceful high availability. It is a maintenance lock. If the component must survive maintenance, add a tested replica or accept a planned outage. Lowering the PDB during every upgrade simply moves the availability decision into an unreviewed manual step.

PDBs also do not guarantee topology. Four healthy Chrome Nodes on one Kubernetes worker satisfy a count until that worker fails. Spread replacement capacity across the failure domains that matter to your cluster. The budget limits voluntary concurrency; scheduling rules determine where the remaining Pods live.

Coordinate Selenium drain before Kubernetes eviction

Begin a host-maintenance operation by cordoning the Kubernetes node. Cordon stops new Pods from being scheduled there, but it does not evict existing Pods or stop Grid from assigning sessions to an existing Selenium Node. Those are separate schedulers.

Identify every Selenium Grid Node Pod on the host and map it to the Grid Node ID. Do not assume the Pod UID equals Selenium's Node ID. The Grid's /status response and GraphQL nodesInfo expose Grid identity, URI, status, sessions, and stereotypes. Store the mapping as evidence for the operation.

Post the documented drain request through the Grid Router or directly to the Node. The registration secret is required when configured. Once Grid reports DRAINING, new session requests should not be assigned there. Existing sessions continue.

The following coordinator accepts a Grid URL, registration secret, Kubernetes node name, and one or more Selenium Node IDs already verified to run on that host. It cordons first, drains all listed Grid Nodes, then polls GraphQL until every listed Node has been observed in DRAINING state with zero sessions. Only after that confirmation does it invoke kubectl drain. A Node that vanishes from the Grid model before its confirmation stops the operation for an operator to review, because absence alone does not prove a clean drain. It requires curl and jq.

#!/usr/bin/env bash
set -euo pipefail

GRID_URL="$1"
REGISTRATION_SECRET="$2"
KUBERNETES_NODE="$3"
shift 3

if test "$#" -eq 0; then
  printf 'Provide at least one Selenium Grid Node ID\n' >&2
  exit 64
fi

kubectl cordon "$KUBERNETES_NODE"

for GRID_NODE_ID in "$@"; do
  curl --fail-with-body --silent --show-error \
    --request POST \
    -H "X-REGISTRATION-SECRET: $REGISTRATION_SECRET" \
    "$GRID_URL/se/grid/distributor/node/$GRID_NODE_ID/drain"
done

confirmed=" "
deadline="$((SECONDS + 1800))"
while :; do
  snapshot="$(curl --fail-with-body --silent --show-error \
    -H 'Content-Type: application/json' \
    --data '{"query":"{ nodesInfo { nodes { id uri status sessionCount } } }"}' \
    "$GRID_URL/graphql")"

  pending=0
  for GRID_NODE_ID in "$@"; do
    case "$confirmed" in
      *" $GRID_NODE_ID "*) continue ;;
    esac

    status="$(jq -r --arg id "$GRID_NODE_ID" \
      '[.data.nodesInfo.nodes[] | select(.id == $id) | .status][0] // "ABSENT"' \
      <<< "$snapshot")"
    count="$(jq -r --arg id "$GRID_NODE_ID" \
      '[.data.nodesInfo.nodes[] | select(.id == $id) | .sessionCount][0] // "absent"' \
      <<< "$snapshot")"
    printf 'gridNode=%s status=%s sessions=%s\n' \
      "$GRID_NODE_ID" "$status" "$count"

    if test "$status" = "ABSENT"; then
      printf 'Node %s left the Grid before a zero-session confirmation\n' \
        "$GRID_NODE_ID" >&2
      exit 3
    fi

    if test "$status" = "DRAINING" && test "$count" = "0"; then
      confirmed="$confirmed$GRID_NODE_ID "
      continue
    fi

    pending="$((pending + 1))"
  done

  if test "$pending" -eq 0; then
    break
  fi
  if test "$SECONDS" -ge "$deadline"; then
    printf 'Timed out waiting for Selenium sessions to finish\n' >&2
    exit 1
  fi
  sleep 10
done

kubectl drain "$KUBERNETES_NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=30m

The script deliberately does not force deletion when the deadline expires. A force option would contradict the session-preservation goal. The operator should inspect long sessions, decide whether they are hung, and terminate them through an approved test or Grid procedure.

It is just as deliberate that a missing Node produces exit 3 rather than a zero session count. Selenium stops a drained Node once its last session ends, so absence from nodesInfo is a normal part of a successful drain. It is also what a crashed process, a lost host, a network partition, or a Pod deleted by another controller looks like, and GraphQL cannot tell you which one happened. A jq default such as // 0 turns every one of those into "safe to evict" and reads a possible mid-session kill as a clean finish. Requiring an explicit DRAINING observation with zero sessions costs a little maintenance time and occasionally flags a Node that really did stop cleanly between polls. That is the correct direction to be wrong in when the alternative is evicting a Pod whose browser state was never confirmed gone.

Calling Selenium drain through the Distributor is important when a Node remains reachable. Removing a Node from the Distributor is different: Selenium's remove endpoint makes the Distributor forget it but does not stop ongoing sessions or the Node process. Removal can be useful for other operations, but it is not a substitute for graceful drain.

The order between session zero and Pod eviction has a race, and DRAINING only closes half of it. Polling an ordinary ready endpoint without first changing Grid admission lets the Node receive another session between checks, so draining first is still mandatory. What DRAINING protects is a Node ID. It is not a property of the Pod.

That distinction is where the sequence can still lose a session. Selenium's drain stops the Node process after the last session ends, and a container running under a Deployment or StatefulSet has restartPolicy: Always, so the kubelet restarts it inside the same Pod. The restarted process registers with the Distributor as a brand new Node ID in the UP state, advertising its slots, on a Pod that kubectl drain has not finished evicting. kubectl cordon does nothing about this, because cordon only stops new Pods from being scheduled onto the host. The replacement Node can accept a fresh session and lose it to the eviction that is still in flight, which is precisely the mid-session kill the whole procedure exists to prevent.

Close that window by taking the Pod out of the registration path before you drain, not after. Scale the workload down so the terminating replica is not recreated, or evict the Pod through the Eviction API immediately after its Node reports the zero-session confirmation, or remove the label that puts the Pod on the Node's Event Bus and Router reachability path so a restarted container cannot register at all. Whichever mechanism you pick, the evidence to capture is the same three facts in order: the drained Node ID is gone from nodesInfo, no new Node ID has appeared advertising that Pod's URI, and the eviction completed after both of those were true. A maintenance run that never checks the second fact can pass its own assertions and still kill a browser.

The Pod termination grace period must exceed the legitimate session-drain time if any part of the sequence happens in a preStop hook. A 30-second default cannot protect a 20-minute browser run. A long grace period slows stuck maintenance and resource recovery. External coordination, as above, keeps termination from starting until sessions are gone, which makes the grace period a cleanup bound rather than the full test duration.

Prove maintenance with a live session

A maintenance test should hold a real session on a known Node, begin the drain, and issue another command before quitting. A smoke test that creates and immediately closes a browser never exercises the dangerous interval.

The Java probe below opens a browser, prints its session ID, waits two minutes for the operator or maintenance job to begin, then refreshes the page. Its finally block closes the session normally. Run it only in a disposable Grid and use GraphQL to locate the owning Node before triggering the coordinator.

import java.net.URI;
import java.time.Duration;
import org.openqa.selenium.By;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

public final class MaintenanceSessionProbe {
  public static void main(String[] args) throws Exception {
    URI grid = URI.create(System.getenv("GRID_URL"));
    ChromeOptions options = new ChromeOptions();
    options.addArguments("--headless=new");

    RemoteWebDriver driver =
        new RemoteWebDriver(grid.toURL(), options);
    try {
      driver.get("https://www.selenium.dev/selenium/web/web-form.html");
      String sessionId = driver.getSessionId().toString();
      System.out.println("SESSION_ID=" + sessionId);
      System.out.println("Begin maintenance against the owning Node");

      Thread.sleep(Duration.ofMinutes(2).toMillis());

      driver.navigate().refresh();
      String heading = driver.findElement(By.tagName("h1")).getText();
      if (!"Web form".equals(heading)) {
        throw new AssertionError("Unexpected heading: " + heading);
      }
      System.out.println("COMMAND_AFTER_DRAIN_STARTED=passed");
    } finally {
      driver.quit();
    }
  }
}

During the wait, GraphQL should show the session on the target Node. After the drain request, the Node status should become DRAINING while sessionCount remains one. The refresh should still pass because draining stops new allocation, not commands for the existing session. After quit, sessionCount reaches zero and the Node can stop.

Capture a second new-session request during the same interval. It should land on another compatible Node or remain queued. It must not appear on the draining Node. This is the admission assertion that prevents the zero-session race.

The complete evidence timeline looks like:

10:02:11 session=18f... node=7a2... status=UP sessionCount=1
10:02:20 kubernetesNode=worker-3 cordoned=true
10:02:22 gridNode=7a2... drainRequestHttp=200
10:02:24 gridNode=7a2... status=DRAINING sessionCount=1
10:03:01 secondSessionNode=4c9... targetDrainingNode=false
10:04:27 originalSessionRefresh=passed
10:04:31 gridNode=7a2... status=DRAINING sessionCount=0
10:04:33 workloadReplicas=4->3 podRecreate=disabled
10:04:35 nodeAtUri=http://selenium-node-chrome-4:5555 registered=false
10:04:36 podEviction=accepted

Use actual timestamps and IDs from the environment. The lines above show the shape of useful evidence, not a claim that every Grid emits those exact log messages.

Run another exercise where the browser never quits. The coordinator should hit its deadline and leave the Kubernetes drain incomplete. That is a pass for the safety control. The response procedure then decides whether to extend the window, contact the test owner, or terminate the session. Automation should not silently turn a timeout into force.

Diagnose blocked and over-permissive budgets

Before changing a PDB, inspect its live status and selector. Kubernetes reports expectedPods, currentHealthy, desiredHealthy, disruptionsAllowed, observedGeneration, and conditions. ObservedGeneration should match metadata.generation before you trust the calculation.

set -euo pipefail

NAMESPACE="$1"

kubectl get pdb -n "$NAMESPACE" \
  -o custom-columns='NAME:.metadata.name,EXPECTED:.status.expectedPods,HEALTHY:.status.currentHealthy,DESIRED:.status.desiredHealthy,ALLOWED:.status.disruptionsAllowed,GEN:.metadata.generation,OBSERVED:.status.observedGeneration'

for pdb in $(kubectl get pdb -n "$NAMESPACE" -o name); do
  printf '\n%s\n' "$pdb"
  kubectl describe -n "$NAMESPACE" "$pdb"
done

A common blocked case has replicas one, currentHealthy one, desiredHealthy one, and disruptionsAllowed zero. Kubectl drain reports that eviction would violate the disruption budget. Nothing is malfunctioning. The budget says the only Pod must remain. Add a ready replacement or schedule an outage.

Another blocked case has replicas four but only three healthy, with maxUnavailable one. The existing unhealthy Pod has consumed the allowance. Find why its replacement cannot become ready. Pending Pods may lack cluster capacity, image pulls may fail, or the Grid process may not register. Loosening the PDB treats the symptom by allowing maintenance to reduce capacity further.

A stale status is different. If observedGeneration trails metadata.generation, wait for the disruption controller or investigate control-plane health. Editing minAvailable repeatedly while the controller has not observed the previous edit makes diagnosis worse.

The over-permissive case often has expectedPods zero. The selector matches no Pods, perhaps because a chart changed labels. The budget then protects no Selenium component. Compare the selector with live Pod labels:

kubectl get pods -n grid --show-labels
kubectl get pdb -n grid selenium-node-chrome -o jsonpath='{.spec.selector}'
printf '\n'
kubectl get pods -n grid \
  -l 'app.kubernetes.io/name=selenium-grid,app.kubernetes.io/component=node,grid.selenium.dev/browser=chrome' \
  -o wide

A broad selector can be numerically correct and operationally wrong. Suppose it selects six Chrome Nodes and one Firefox Node with maxUnavailable one. Kubernetes may evict Firefox because six healthy Pods remain. The PDB kept the count while Grid lost an entire stereotype. Separate pools prevent that specific mistake.

Direct Pod deletion is the most convincing bypass. An operator runs kubectl delete pod and the Pod terminates even though disruptionsAllowed is zero. The PDB was not ignored; the operation did not use the Eviction API. Audit maintenance tools and permissions. Use kubectl drain or another Eviction API client for planned disruption.

Deployment rollouts are another near miss. PDBs do not replace Deployment strategy. Set maxUnavailable and maxSurge on the workload so application-driven updates preserve intended capacity, then add Selenium session drain. A perfect PDB cannot make a rolling image update wait for a WebDriver session by itself.

Separate session loss from failures with the same timing

A test may fail during maintenance because the application under test also moved, not because its browser Pod was evicted. The WebDriver session stays present in GraphQL, but navigation receives gateway errors or the page loses backend connectivity. Compare Grid session ownership with application deployment events before blaming the PDB.

A browser can crash on an unevicted Node. The Pod remains Running and Ready, Grid may remove the session, and no Kubernetes eviction event exists. Inspect browser and driver logs, Pod restart count, container state, and the Session Map timeline. PDB status is irrelevant to a process-level failure that never made the Pod unavailable.

Network policy changes can break Router-to-Node commands while all Pods remain healthy. The session remains mapped to its Node, and Kubernetes shows no disruption. Probe the stored Node URI from the Router network context. Preserving Pod count does not preserve every communication path.

A session request can time out in the queue because maintenance drained scarce capacity correctly. That is an availability trade rather than session destruction. Existing sessions finish, but new work waits for replacements. Queue snapshots and client construction time distinguish this from a mid-session loss. Increase surge capacity or pause CI admission if session-start objectives matter during maintenance.

An involuntary worker failure can occur inside the same upgrade window. PDBs cannot prevent it, though the failed Pod counts against the budget and should block another voluntary eviction. Node conditions and Pod disruption events identify the unplanned loss. Do not mark the maintenance procedure safe merely because the second eviction was blocked after the first Node already vanished.

Finally, terminationGracePeriodSeconds can expire. The eviction was allowed correctly, the preStop hook began Selenium drain, but a session exceeded the grace period and kubelet killed the container. Evidence shows an accepted eviction, a terminating Pod, sessions still above zero, and forced termination at the deadline. Increase the grace period, move drain before eviction, or impose shorter session limits. Changing minAvailable will not fix that race.

Roll out the policy and accept its limits

Inventory voluntary disruption sources: manual kubectl drain, managed-cluster upgrades, autoscaler consolidation, security patch automation, and internal operators. Confirm which ones use the Eviction API and what timeout or force behavior they apply. A PDB only helps when the actor honors it.

Label live workloads by role and compatible browser pool. Apply budgets in a staging Grid, then use server-side dry run and selector checks before production:

kubectl apply --server-side --dry-run=server -f selenium-grid-pdbs.yaml
kubectl apply -f selenium-grid-pdbs.yaml
kubectl get pdb -n grid

Run one maintenance probe against one worker. Measure time from cordon to Grid DRAINING, longest session completion, replacement Pod readiness, Grid registration, and host drain completion. These numbers set a realistic automation timeout and surge requirement.

Roll through one worker at a time until replacement capacity is both Kubernetes-ready and visible to Grid. Kubernetes readiness alone can precede Node registration or useful stereotype availability. Query GraphQL nodesInfo and confirm the expected pool count before continuing.

Alert on a PDB stuck at disruptionsAllowed zero only when maintenance needs allowance or availability is already degraded. Zero can be the intended steady state for a strict single-replica component. Add context rather than paging on the number alone.

PDBs cost maintenance time and spare capacity. A replacement needs somewhere else to schedule. If the cluster is full, strict budgets can stall upgrades indefinitely. Reserve headroom, allow temporary surge nodes, or accept a documented maintenance window.

Session-aware drain costs even more time because the longest browser controls the host. Cap individual session duration, find leaked sessions, and make test owners visible. Do not shorten the timeout until normal long tests are accounted for.

Skip a PDB when the cluster has no voluntary disruption mechanism and the Grid is a disposable personal environment. There is no operator benefit to a policy object nobody invokes.

Do not pretend a PDB makes a single replica highly available. It can block planned eviction, but it cannot survive the host failing. Replication, persistence, topology, and tested recovery solve that problem.

Avoid protecting one-shot browser Pods with a broad count if failed sessions are intentionally retried and the infrastructure treats each Pod as disposable. Protect the control plane and aggregate admission capacity instead, according to the actual architecture.

Most importantly, do not use a PDB as the first line of session lifecycle logic. It is the last concurrency check before voluntary eviction. Selenium drain makes a browser Node safe to remove; the PDB makes sure too many safe removals do not happen together. Both controls are necessary when a running session is worth preserving.