Stop sensitive Selenium sessions from landing in the wrong cloud

A payment test marked “restricted” lands on a public cloud browser because the local pool is full. The test passes, so the placement bug survives unnoticed until someone reads the provider history. Capacity fallback made the suite faster, but it violated the one rule the routing layer was supposed to enforce.

Write the placement policy before wiring providers

Hybrid capacity is useful because the two sides are different. An on-prem pool may have private network access, controlled images, and strict data boundaries. A cloud service may offer browser versions, operating systems, and burst capacity that are expensive to maintain locally. Routing should preserve those differences, not flatten them into one mysterious URL.

Start with inputs that a reviewer can reason about:

• data classification, such as synthetic, internal, or restricted;
• required browser, version, operating system, and device;
• network reachability, including private application endpoints;
• whether cloud execution is allowed for this workload;
• whether waiting is preferable to failing when the preferred pool is full;
• the maximum acceptable session-start delay and provider cost.

Do not use a test method name, package name, or an undocumented environment variable as a security label. Names change during refactoring and are rarely validated. Attach classification through a typed test annotation, suite manifest, or CI input owned by the same governance process as the test data.

Placement and capacity are different decisions. “This test is allowed in cloud” does not mean “always put it in cloud.” “On-prem is full” does not make a restricted test eligible for another provider. First determine the set of allowed destinations. Then choose among that set using availability, coverage, and cost.

Fail closed where the rule protects data or network boundaries. If a restricted test requires the on-prem pool and no compatible slot is available, queue it, reschedule it, or fail it with a placement error. Quietly relaxing the rule turns a capacity incident into a compliance incident.

Failing closed has a visible price. The release may wait for local capacity, and the platform team may need to reserve idle headroom. That is an honest trade. If the organization wants cloud overflow for the workload, change the data and access design so the workload becomes eligible. Do not hide the decision in retry code.

Also define who owns the policy. QA can specify coverage needs. Security and application owners define data and network restrictions. Platform engineers know pool capabilities and cost. A route table created by one group without the others usually encodes assumptions that break during an incident.

A useful decision record includes the rule version, normalized requirements, chosen pool, reason, and timestamp. It must not include secrets. Add the WebDriver session ID once creation succeeds. That one line becomes the join key between the test report, Grid GraphQL, and provider-side records.

Choose the routing boundary deliberately

There are two practical patterns, and they fail in different places.

The first pattern chooses the endpoint in the test infrastructure. A factory creates RemoteWebDriver against either the on-prem Router URL or the cloud WebDriver URL. The choice is explicit and easy to unit test. Each provider remains operationally separate, which limits shared failure. Credentials and provider-specific options can stay in provider adapters.

The cost is duplicated entry points. Client configuration must know both destinations, and policy changes require updating the shared test library or an external policy service. Cross-provider queue and capacity data are not naturally centralized.

The second pattern exposes one Selenium Router and adds a Relay Node for the external WebDriver service. The Relay advertises configured stereotypes. The Distributor compares a new-session request with local and relay slots, then sends it to a matching Node. Tests keep one Grid URL.

That central entry point simplifies clients, but the capability design now carries placement intent. If local and cloud slots advertise identical stereotypes, the Distributor is free to choose either compatible slot. Add a namespaced custom capability when pool identity is a hard requirement, and configure corresponding stereotypes. The namespace prevents the capability from pretending to be part of the W3C standard set.

A Relay does not make the external provider local. Commands still cross the network, provider authentication still matters, and vendor limits still apply. The Relay represents slots and forwards WebDriver traffic. Confirm that the external service's status endpoint behaves as Grid expects. Some providers require a gateway or adapter that presents a compatible health endpoint and handles credentials.

The two patterns can coexist during migration, but each session must have one authoritative route. Do not let a client choose the cloud URL and also send a capability intended for an internal Relay unless that combination is explicitly supported. Conflicting layers produce reports where the recorded policy says one thing and the actual endpoint guarantees another.

My default is client-side selection when data placement is the primary concern. The endpoint itself becomes a strong boundary, and cloud failure cannot make the on-prem Grid reinterpret a capability. I prefer Relay when the goal is centrally managed browser coverage and the external service has been tested as a clean WebDriver backend.

Make client-side routing auditable

The following Java example implements a small internal policy. Everything named RoutePolicy, Workload, RouteDecision, or RoutedDriver belongs to the example application; none of those names claim to be Selenium APIs. The Selenium calls are limited to ChromeOptions and RemoteWebDriver.

Restricted and internal work stays on premises. Synthetic work may use cloud only when the caller explicitly allows it and requests coverage unavailable locally. There is no automatic capacity fallback because this factory does not have authoritative pool state.

import java.net.URI;
import java.time.Instant;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

enum DataClass {
  SYNTHETIC, INTERNAL, RESTRICTED
}

record Workload(
    String testId,
    DataClass dataClass,
    String platformName,
    boolean cloudAllowed,
    boolean needsCloudCoverage) {}

record RouteDecision(
    String pool,
    URI endpoint,
    String reason,
    Instant decidedAt) {}

record RoutedDriver(
    RemoteWebDriver driver,
    RouteDecision decision) implements AutoCloseable {
  @Override
  public void close() {
    driver.quit();
  }
}

final class RoutePolicy {
  static RouteDecision decide(
      Workload workload, URI onPrem, URI cloud) {
    if (workload.dataClass() == DataClass.RESTRICTED) {
      return new RouteDecision(
          "onprem", onPrem, "restricted data", Instant.now());
    }
    if (workload.dataClass() == DataClass.INTERNAL) {
      return new RouteDecision(
          "onprem", onPrem, "internal network boundary", Instant.now());
    }
    if (workload.cloudAllowed() && workload.needsCloudCoverage()) {
      return new RouteDecision(
          "cloud", cloud, "explicit cloud-only coverage", Instant.now());
    }
    return new RouteDecision(
        "onprem", onPrem, "default synthetic route", Instant.now());
  }
}

public final class HybridDriverFactory {
  private final URI onPrem;
  private final URI cloud;

  public HybridDriverFactory(URI onPrem, URI cloud) {
    this.onPrem = onPrem;
    this.cloud = cloud;
  }

  public RoutedDriver open(Workload workload) throws Exception {
    RouteDecision route = RoutePolicy.decide(workload, onPrem, cloud);
    System.out.printf(
        "route test=%s pool=%s reason=%s decidedAt=%s%n",
        workload.testId(), route.pool(), route.reason(), route.decidedAt());

    ChromeOptions options = new ChromeOptions();
    options.setPlatformName(workload.platformName());
    options.addArguments("--headless=new");

    RemoteWebDriver driver =
        new RemoteWebDriver(route.endpoint().toURL(), options);
    Capabilities actual = driver.getCapabilities();

    System.out.printf(
        "session=%s pool=%s browser=%s version=%s platform=%s%n",
        driver.getSessionId(),
        route.pool(),
        actual.getBrowserName(),
        actual.getBrowserVersion(),
        actual.getPlatformName());

    return new RoutedDriver(driver, route);
  }

  public static void main(String[] args) throws Exception {
    URI onPrem = URI.create(System.getenv("ON_PREM_GRID_URL"));
    URI cloud = URI.create(System.getenv("CLOUD_WEBDRIVER_URL"));

    Workload workload = new Workload(
        "checkout-redaction-17",
        DataClass.RESTRICTED,
        "linux",
        false,
        false);

    HybridDriverFactory factory = new HybridDriverFactory(onPrem, cloud);
    try (RoutedDriver routed = factory.open(workload)) {
      routed.driver().get("https://www.selenium.dev/selenium/web/web-form.html");
      System.out.println("title=" + routed.driver().getTitle());
    }
  }
}

The factory logs before creating the browser. That matters when construction times out, because there may be no session ID. Once the session exists, it logs returned capabilities rather than assuming the request was honored. If the requested platform and returned platform disagree, fail the infrastructure check before running product assertions.

Production code should keep cloud credentials out of the endpoint string printed to logs. Read them from the platform's secret mechanism and redact URLs. Many providers require vendor-specific options. Put those in a provider adapter backed by the provider's official documentation, not in a generic example copied between services.

Do not catch every SessionNotCreatedException and try the other endpoint. A cloud timeout may occur after the provider created a browser but before the response arrived. A second attempt can leak capacity and run the same test twice. Retry only when the failure is classified as pre-creation, the alternate destination is allowed, and cleanup of the first attempt is understood.

Unit-test RoutePolicy without contacting either Grid. Cover every data class, cloudAllowed value, and special-coverage branch. Then add one integration test per destination to prove endpoint configuration, authentication, and returned capabilities. Keeping pure policy tests separate from costly browser tests makes failures much easier to interpret.

Use Relay only with explicit pool stereotypes

A single Router needs a capability that distinguishes two otherwise compatible Chrome pools. The following ConfigMap carries two Selenium TOML files. The local Node advertises tta:pool=onprem, and the Relay advertises tta:pool=cloud. Custom capabilities use a namespaced key rather than inventing an unprefixed WebDriver capability.

apiVersion: v1
kind: ConfigMap
metadata:
  name: selenium-hybrid-node-config
data:
  onprem.toml: |
    [node]
    detect-drivers = false

    [[node.driver-configuration]]
    display-name = "On-prem Chrome"
    max-sessions = 4
    stereotype = '{"browserName":"chrome","platformName":"linux","tta:pool":"onprem"}'

  relay.toml: |
    [node]
    detect-drivers = false

    [relay]
    url = "https://webdriver-gateway.example.test/wd/hub"
    status-endpoint = "/status"
    protocol-version = "HTTP/1.1"
    configs = [
      "5",
      '{"browserName":"chrome","platformName":"linux","tta:pool":"cloud"}'
    ]

Mount the appropriate file into each Node process and start the Node with Selenium's --config option. The gateway hostname is an example boundary, not a claim about any vendor. It should terminate provider authentication without exposing credentials in the ConfigMap and provide a status response compatible with the Relay health check.

A Relay request must carry the same pool capability:

import java.net.URI;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.RemoteWebDriver;

public final class RelayPoolProbe {
  public static void main(String[] args) throws Exception {
    URI grid = URI.create(System.getenv("GRID_URL"));
    String pool = args[0];

    if (!pool.equals("onprem") && !pool.equals("cloud")) {
      throw new IllegalArgumentException(
          "pool must be onprem or cloud");
    }

    ChromeOptions options = new ChromeOptions();
    options.setPlatformName("linux");
    options.setCapability("tta:pool", pool);
    options.addArguments("--headless=new");

    RemoteWebDriver driver =
        new RemoteWebDriver(grid.toURL(), options);
    try {
      System.out.println("SESSION_ID=" + driver.getSessionId());
      System.out.println("REQUESTED_POOL=" + pool);
      System.out.println("ACTUAL_CAPS=" + driver.getCapabilities());
      driver.get("https://www.selenium.dev/selenium/web/web-form.html");
    } finally {
      driver.quit();
    }
  }
}

Configure all relevant Nodes consistently, because Selenium's default slot matcher compares custom capabilities in one direction only and defaults to a match. It walks the extension capability names declared by the stereotype, keeps the ones the request also declares, compares just those, and returns true when that intersection is empty. Two consequences follow, and both run opposite to intuition. A local Node whose stereotype omits tta:pool still matches a request that carries tta:pool=onprem, because the stereotype contributes no name to compare. A request that omits tta:pool matches the on-prem Node and the cloud Relay equally, because the missing key is filtered out of the comparison before it can exclude anything. Omitting the capability on either side widens the destination set rather than narrowing it. After changing stereotypes, restart or re-register Nodes according to your deployment process and inspect the live Grid model. Editing a ConfigMap does not retroactively change an already registered Node.

Query GraphQL for live Nodes before sending traffic. The nodesInfo stereotypes show what the Distributor sees, which is more useful than the intended file in source control.

set -euo pipefail

GRID_URL="$1"

curl --fail-with-body --silent --show-error \
  -H 'Content-Type: application/json' \
  --data '{"query":"{ nodesInfo { nodes { id uri status stereotypes sessionCount maxSession } } }"}' \
  "$GRID_URL/graphql" | python3 -m json.tool

Expect separate Node identities or URIs for the local and Relay capacity and confirm the pool key appears in their stereotypes. If the ConfigMap says cloud but GraphQL does not, stop there. The running Grid has not loaded the route you intend.

After creating a probe session, query the exact ID. The documented session object includes nodeId, nodeUri, and capabilities.

#!/usr/bin/env python3
import json
import sys
import urllib.request

grid_url = sys.argv[1].rstrip("/")
session_id = sys.argv[2]
query = (
    '{ session(id: "' + session_id + '") '
    '{ id nodeId nodeUri capabilities } }'
)
request = urllib.request.Request(
    grid_url + "/graphql",
    data=json.dumps({"query": query}).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)
with urllib.request.urlopen(request, timeout=5) as response:
    payload = json.load(response)

if payload.get("errors"):
    raise SystemExit(json.dumps(payload["errors"]))
print(json.dumps(payload["data"]["session"], indent=2))

The nodeUri is placement evidence. Returned capabilities are browser evidence. Neither replaces the policy log. A session on the expected Relay Node can still receive the wrong browser version if provider matching differs, and correct capabilities do not prove a restricted test was allowed to leave the network.

Close the hole where a request omits the pool key

Namespaced stereotypes constrain placement only while every request carries the key. Because the matcher ignores extension capabilities the request does not declare, a session request with no tta:pool at all is compatible with the on-prem Node and the cloud Relay at the same time. The Distributor is then free to pick whichever compatible slot is available first, which in a busy period is usually the Relay. That is the opening scenario of this article: a restricted payment test reaching a public cloud browser while every ConfigMap in source control still reads correctly, because the request never carried the capability the routing rule depends on. No amount of Relay configuration closes that path by itself, and the failure is silent because the session is created successfully.

Selenium states the requirement in both directions: "Custom capabilities need to be set in the configuration in all Nodes. They also need to be included always in every session request." Treat both halves as mandatory rather than as advice. Every Node stereotype behind the shared Router must declare tta:pool, including Nodes that predate the policy, and every session request must declare it too.

Enforcement has to live in the client, because Grid will not reject the omission for you. Set the capability in the driver factory rather than in individual tests, refuse to construct a driver when the workload has no resolved pool value, and after creation assert that the returned capabilities and the GraphQL nodeUri both agree with the requested pool. Add one integration test that deliberately submits a pool-less request and asserts which Node received it. That test is worth more than any review of the ConfigMap, because it measures the matcher's real behavior instead of the intended configuration.

If a shared Router cannot guarantee that every client sets the key, the honest conclusion is that capability routing is not a data boundary for that environment. Fall back to separate Routers or to client-side endpoint selection, where an omitted capability cannot silently widen the destination set because the destination was chosen before the request was built. The trade is more entry points to configure, in exchange for a placement rule that survives a careless caller.

Prove the failure paths, not only the happy route

The first worked case is restricted data with no local capacity. Occupy every on-prem slot, then submit a restricted workload. Client-side routing should still select the on-prem URL. Relay routing should request tta:pool=onprem. The request may queue and eventually hit the configured session request timeout. Passing means no cloud session appears, not that the test finishes quickly.

Capture the route decision, queue payload, Node stereotypes, and cloud provider history for the interval. The absence of a cloud job is part of the proof. The operational fix could reserve secure slots, prioritize restricted jobs, schedule them away from peak traffic, or reduce their concurrency. Every option costs utilization or elapsed time.

The second case is cloud-only coverage. Request a browser or platform that the local fleet deliberately does not provide, mark the synthetic workload cloud-allowed, and require cloud coverage. Verify the decision selects cloud, the GraphQL nodeUri belongs to the Relay Node or the client log names the cloud endpoint, and returned capabilities meet the request. Then run a page assertion.

Do not accept “session created” as the only pass condition. A provider may normalize a platform string, choose a nearby browser version under its documented matching rules, or reject unsupported capabilities. Decide which fields require exact equality and which permit a reviewed range. Browser-version flexibility increases session success but weakens reproducibility.

The third case is a cloud outage during a burst. Make the gateway return an unavailable status or block it in a disposable environment. Cloud-eligible, cloud-required tests should fail with a provider-route error. Restricted tests should remain on premises and continue if local capacity is healthy. General synthetic tests may use on-prem only if the written policy allows that destination and the workload does not require cloud-only coverage.

Record how long failure detection takes. Relay health checks, Grid model updates, queue timeout, client HTTP timeout, and CI timeout can all be different. A request may wait after the provider is already known to be unhealthy. Shorter detection improves feedback but can mark a provider down during a brief network wobble. Tune with measured recovery behavior.

Add an asymmetric network test. The Router can reach the Relay Node, but the Relay cannot reach the provider gateway. The Grid front door stays healthy while relay sessions fail. Node status, Relay logs, and provider reachability from the Relay network namespace separate this from client-to-Router failure.

Finally, exercise an ambiguous new-session timeout. Delay the response after provider creation in a controlled stub or provider test environment. Confirm your client does not immediately create another session elsewhere. Search for the first session and clean it up. This is the failure most fallback implementations skip because the exception arrives before application code receives an ID.

Wrong placement has several convincing lookalikes.

A test can run in the correct browser pool and still reach the wrong application environment. Base URL selection often lives beside Grid selection, so logs say “cloud” and a test hits staging instead of the intended isolated environment. Record provider route and application target as separate fields. Never infer one from the other.

Returned capabilities can also look wrong because the test runner overwrote options after policy evaluation. Log normalized requested capabilities immediately before RemoteWebDriver construction. Compare that record with the GraphQL queue request and returned capabilities. If intent changes before the Router, fix the client configuration rather than Grid matching.

Queue delay is not proof of wrong placement. A request carrying tta:pool=cloud may wait because every relay slot is occupied. GraphQL shows the correct queued capability and matching stereotype. The route is right; capacity is unavailable. Adding local Nodes without the cloud stereotype should not affect it.

A misregistered Node is closer to the real problem. Its deployment label may say onprem while its live stereotype says cloud. GraphQL exposes what the Distributor uses. Treat the live registration as authoritative for diagnosis, then fix the configuration and recycle the Node. Renaming a Pod or dashboard series cannot change capability matching.

DNS can produce a subtler mismatch. A cloud gateway hostname might resolve to different regional endpoints from different networks. The client-side decision log shows the intended URL, yet traffic reaches a region with different data residency. Resolve and connect from the actual runner or Relay environment, inspect certificates and gateway logs, and make region selection explicit in network design.

Provider dashboards can lag. A job may appear minutes after the test fails, which makes a reviewer associate it with a retry. Join on a test correlation value only when the provider officially supports one, otherwise use session ID and tightly bounded timestamps. Do not invent a vendor capability called buildId or testName without checking that provider's documentation.

Migrate by making every old assumption visible

Inventory current entry points first. Search CI definitions, test libraries, command-line wrappers, and local developer scripts for Grid URLs. A new central factory has no effect if one suite constructs RemoteWebDriver directly against an old environment variable.

Create a route matrix from real suite requirements. List data class, application network, browser coverage, expected pool, permitted fallback, and owner. Review the surprising rows with security and product teams. Legacy tests often use production-like records without anyone formally classifying them.

Introduce decision-only mode. The existing endpoint remains unchanged, but the policy computes and logs what it would choose. Compare those decisions with current placement for several full test cycles. This catches missing metadata without moving traffic.

Next, enforce the most restrictive rules. Route restricted tests explicitly on premises and reject missing classification for suites that use sensitive fixtures. Canary a small synthetic cloud-only group. Verify cost, latency, artifacts, network access, and cleanup.

If adopting Relay, register it with zero production traffic first by using a capability no normal suite requests. Run the dedicated probe, inspect GraphQL, then enable one canary group. If using client-side routing, deploy the factory with cloudAllowed false by default and opt in suites individually.

Add policy conformance to test reports. A report should show requested classification, route decision, session ID, actual browser, and placement evidence. Keep secrets and raw data out. Review mismatches as infrastructure failures even when product assertions pass.

Rollback must preserve the hard rules. Disabling cloud should send eligible work to an allowed on-prem destination or fail it. It must not remove classification checks. Disabling on-prem for maintenance should fail restricted work rather than reinterpret it as synthetic.

Pay for explicit routing only where the distinction matters

Client-side selection adds a shared library, two endpoint configurations, provider adapters, and policy-version rollout. Relay adds a Node layer, custom stereotypes, a health contract, and another command hop. Both demand integration tests and incident evidence. Hybrid is not automatically simpler than running two clearly separate suites.

Cloud overflow reduces idle local capacity but introduces provider cost, network latency, artifact handling, and data review. Reserved on-prem capacity makes restricted jobs predictable but sits unused at quiet times. The right balance depends on wait-time objectives and the cost of a placement error, not only browser-minute price.

Do not build hybrid routing when every test is synthetic, every target is public, and one provider already supplies required coverage within budget. A second pool creates policy and operational work without a meaningful boundary.

Avoid a shared Relay when cloud and on-prem must be isolated for regulatory or blast-radius reasons. Separate Routers and credentials provide a clearer control boundary. A client policy can still choose between them for eligible workloads.

Do not use capability routing as a secret-management system. Capabilities travel with session requests and can appear in logs, GraphQL, and provider records. They may carry labels, never credentials or sensitive business data.

Do not fall back after an existing session fails mid-test. WebDriver sessions are stateful. Creating a browser elsewhere does not continue the same page, cookies, downloads, or application transaction. Restart the test from a known clean point and record the provider change as a new attempt.

The design earns its keep when placement has a reason a reviewer can state: protected data stays inside, specialized coverage goes outside, and ordinary synthetic work uses a controlled cost policy. If nobody can explain why a session chose a pool from its inputs and evidence, the system is not routing. It is guessing.